On behalf of everyone at PayTrace, I sincerely apologize for PayTrace’s service interruption on Tuesday, June 25, 2019. PayTrace’s purpose is to make our customers happy. We know that delivering you reliable payment processing solutions 24 hours a day, 7 days a week, and 365 days a year is essential to your satisfaction. We understand that you depend on payment processing providers to ensure that your finances flow smoothly and business operates efficiently. We failed to meet your performance standards and fulfill our purpose yesterday.
PayTrace has taken great pride in our commitment to providing consistent uptime. We’ve been delivering on this promise of reliable service with confidence. The events yesterday, understandably create questions for our partners, merchants, and integrators. It’s still early in the process of gathering data from the incident. We’ve launched a full investigation and as we’ve regained access to our system, our team of technical engineers is reviewing reports, logs, and gathering data. Here’s what we know so far:
What happened: Our systems experienced a hard reboot, likely related to a power outage. Monitoring and alerting systems notified our engineers immediately, and we reached out to the data center. As the systems came online from the reboot, there were numerous issues in the hardware within the networking and virtualization layers. These errors blocked access to the system. Data center PCI security controls prevented us from accessing the infrastructure. Once we gained access to the environment, our engineers were able to bring our systems back online properly and restored access. We recovered all scheduled processes and transactions were able to process. Engineers continued testing and monitored the system closely throughout the night. All normally scheduled services ran successfully.
What’s next: Our engineers are reviewing logs to understand the exact sequence of events. We realize that power outages should not create a cascading incident. We are investigating the network configuration to identify the vulnerability that caused our service disruption. Even though emergency plans, redundancy, and fail-over procedures were in place, we have learned that our data center partner did not perform according to our Service Level Agreement. We recognize that PayTrace is accountable for the vendors that help deliver our service and are actively evaluating options that greatly improve our reliability. We will provide an additional update on this incident after our investigation is complete.
I want to assure you that all of our systems are operational, and we have not experienced any data loss. PayTrace is committed to making this right. You not only have my full attention; you also have the full support of all PayTracers.