Beginning at 5:45am PDT our Traditional API and Virtual Terminal servers hit max CPU utilization. Our clients began to see intermediate timeout connections to our servers. Our load balancer's began to implement emergency traffic optimization shaping, but with the servers being overloaded they began to fail health checks. Our internal monitors began alerting warnings at 5:51am and critical errors at 5:56am. Our external monitors and automated procedures began notifying our OnCall staff starting at 5:50am from which PayTrace started taking action. At 6:28am the root symptom was isolated and we began implementing mitigation strategies without dropping current connections. At 6:35am CPU utilization began to stabilize and intermediate outage symptoms began to decline. At 6:40am all incoming and outbound traffic was flowing smoothly. Further optimizations were implemented yesterday afternoon to help prevent this situation from occurring again.

We sincerely apologize about the inconvenience this incident has caused. We hope the transparency of this incident and following optimizations/mitigation steps give you confidence in our commitment to providing optimal up-time of our service. We thank you for your patience and continued business.