Possible Infrastructure Issues
Incident Report for PayTrace
Postmortem

What Happened?

Follow our Saturday migration to our New Datacenter; Monday July 18th at 7AM PDT our Firewall Intrusion Prevention (IPS) System became bottlenecked with heavy legal traffic. Traffic was still flowing through our network successfully, Clients began to see heavy latency.

HEAVY LATENCY DURATION WINDOW - 3 HOURS

What Caused This to Happen?

The Firewall IPS hit a max threshold, which caused heavy latency leaving some clients with the symptom of an outage. The key was to first identify the root cause, which was isolated around 8AM PDT, then make sure the traffic coming through the Firewall IPS was legal traffic, then strategize around a solution to safely recycle many CPU processes to protect current active traffic, but resolve the bottleneck.

What are We Doing to Prevent This?

We have taken action to upgrade the Firewall IPS to handle extremely higher traffic, preventing heavy latency.

Our Apologies

We strive to maintain a stable and reliable service for our customers. We are sorry for any inconvenience this outage may have caused you or your customers. We have seen dramatic increases in performance throughout our platform with our New Datacenter and have removed single points of failures.

Posted Jul 18, 2016 - 22:30 PDT

Resolved
This incident has been resolved.
Posted Jul 18, 2016 - 11:14 PDT
Monitoring
We have resolved the latency throughput issue. We are seeing throughput stabilize. We will continue to monitor this.
Posted Jul 18, 2016 - 10:05 PDT
Identified
We have isolated the performance issues to be with our network edge. We are investigating and working on a solution. Traffic is still flowing in, but some request are seeing latency.
Posted Jul 18, 2016 - 08:18 PDT
Investigating
Our automated systems are having trouble reaching our servers and we're investigating the possible connectivity issues. More information will be made available soon.
Posted Jul 18, 2016 - 07:38 PDT
This incident affected: JSON API and Branded Site.