At 4:15AM Pacific our load balancer at our primary data center ceased responding. Once this occurred, all connectivity to PayTrace virtual terminal services including our API were unreachable. After working with our provider and diagnosing the problem our load balancer was back in operation at 5:28AM Pacific.
The failure was caused by the load balancer ceasing to respond and requiring a full reset. This problem was magnified by this being a single point of failure in our network.
PayTrace is beginning a major investment in upgrading all of our networking infrastructure and eliminating single points of failure like this. We were already in this process before this incident occurred. The timeline for these improvements will be occurring between July and September of this year.
We are also reviewing all of our currently operating equipment and identifying any potential problems that might occur before we complete our upgrade project.
We strive to maintain a stable and reliable service for our customers. We are sorry for any inconvenience this outage may have caused you or your customers and we are working to continue to upgrade our services and responsiveness for you.