First Data North Processing Platform experiencing issues

Incident Report for PayTrace

Postmortem

Root Cause Corrective Action from First Data

Platform - North Front-End

Regarding - Paytrace Gateway Authorization Service Interruption

Incident Record - IR12749251

Issue Description & Recovery From 1:52 p.m. CST until 4:44 p.m. CST on January 25, 2017, a subset of clients experienced an intermittent service interruption processing authorization transactions via Paytrace Gateway on the North platform. First Data support teams rerouted traffic to an alternate circuit and Paytrace Gateway restarted its link to restore service.

Business Impact Affected clients sporadically received time outs when processing transactions requiring the client to re-try the transaction or accept a different form of payment.

Root Cause The issue was due to a problem with a circuit in the Omaha Data Center for "Meet in the Middle VPN" users, such as Paytrace Gateway. After First Data support teams moved traffic to the alternate circuit in the Chandler Data Center to mitigate impact, Paytrace Gateway processing did not automatically resume. Paytrace Gateway restarted its connection to restore service.

Preventative Measures First Data support teams replaced a faulty card in the impacted circuit on January 26, 2017 to resolve the issue. In addition, they reviewed the process of rerouting traffic during single circuit issues to identify improvements.

We apologize for any inconvenience caused by this issue.

Posted Feb 27, 2017 - 11:30 PST

Resolved

First Data traffic continues to flow across alternate ISP Circuit Backbone.

Posted Jan 25, 2017 - 15:41 PST

Monitoring

First Data has migrated our networks to an alternate ISP Circuit Backbone.

Posted Jan 25, 2017 - 14:55 PST

Update

First Data has various backbone ISPs (Internet Service Providers) for client data to travel across. Verizon ISP Circuit Backbone has encountered damage, Verizon is working on getting the damage repaired. In the short term, First Data is trying to resolve the negative impact by migrating client networks to an alternate ISP Circuit Backbone.

PayTrace will give an update as soon as First Data migrates our network connections to an alternate ISP Circuit Backbone.

Posted Jan 25, 2017 - 14:39 PST

Update

First Data is still trying to implement a resolution on their First Data North processing platform. Thank you for your patience.

Posted Jan 25, 2017 - 14:01 PST

Identified

First Data has Identified the problematic equipment on their end within the First Data North platform. First Data is trying to implement a resolution.

Posted Jan 25, 2017 - 13:23 PST

Update

We are still investigating outage with First Data North (FDT).

Posted Jan 25, 2017 - 12:57 PST

Investigating

We are on the phones with First Data North, and are aware of the issue. We will update everyone on the situation when more information becomes available.

Posted Jan 25, 2017 - 12:32 PST

This incident affected: Transactional Processing (Traditional), JSON API, Virtual Terminal, Traditional API & Secure Checkout, and Processing Networks (First Data North (FDT)).