What Happened?
Follow our Saturday migration to our New Datacenter; Monday July 18th at 7AM PDT our Firewall Intrusion Prevention (IPS) System became bottlenecked with heavy legal traffic. Traffic was still flowing through our network successfully, Clients began to see heavy latency.
HEAVY LATENCY DURATION WINDOW - 3 HOURS
What Caused This to Happen?
The Firewall IPS hit a max threshold, which caused heavy latency leaving some clients with the symptom of an outage. The key was to first identify the root cause, which was isolated around 8AM PDT, then make sure the traffic coming through the Firewall IPS was legal traffic, then strategize around a solution to safely recycle many CPU processes to protect current active traffic, but resolve the bottleneck.
What are We Doing to Prevent This?
We have taken action to upgrade the Firewall IPS to handle extremely higher traffic, preventing heavy latency.
Our Apologies
We strive to maintain a stable and reliable service for our customers. We are sorry for any inconvenience this outage may have caused you or your customers. We have seen dramatic increases in performance throughout our platform with our New Datacenter and have removed single points of failures.