Site Latency
Incident Report for PayTrace
Postmortem

At 10:22AM we experienced a large increase in volume that degraded site performance in particular for our web users. By 10:23AM we had technicians diagnosing the problem and working on potential mitigation resolutions. They intervened and increased capacity to the system to help minimize the impact of the volume. By 10:47AM we saw improvements in performance and by 11:30AM the incident had been resolved.

 

We have been and are continuing to work on scaling our systems for higher redundancy, increased alerting and monitoring, and enhanced scalability. We believe these efforts will reduce the risk of these issues. In the immediate future to help provide a consistent level of service we are investing in more capacity of our web stack to be ready to handle increased volume.

Thanks,

The PayTrace Team

Posted Nov 28, 2018 - 15:05 PST

Resolved
We have stopped the negative impact causing latency. We will be investigating the root cause and will provide a postmortem when available.
Posted Nov 27, 2018 - 11:39 PST
Investigating
We are currently investigating reports of latency when accessing PayTrace. We are working on identifying the issue and bringing our site back to full speed. Please check back with the site periodically as we resolve this this issue. Additionally, you may also check our status page as we will be providing real time updates.
Posted Nov 27, 2018 - 10:30 PST
This incident affected: JSON API, Virtual Terminal, and Traditional API & Secure Checkout.