TSYS Transaction Processing Delays
Incident Report for PayTrace
Postmortem

Incident Date: August 18, 2017

Owner Area: TSYS 3rd Party Vendor

A TSYS Third Party vendor that provides Telecom services from Merchant Terminals to TSYS Sierra platform experienced and outage in which 12 out of 14 SSL gateways were not taking traffic. This outage was caused by a network trace being initiated on the gateways at 14:09 EDT. The data trace software inadvertently hung and caused socket disconnects across the 12 gateways causing them to become unresponsive. At 16:19EDT the gateways were successfully restarted, restoring service for SSL connectivity.

Posted Aug 30, 2017 - 13:21 PDT

Resolved
TSYS has officially stated the negative impacts to all their endpoints are now resolved.
PayTrace is waiting for official cause of the outage from TSYS.
Posted Aug 18, 2017 - 14:22 PDT
Update
TSYS is still working on getting all endpoints stable for all processing transactions (swipe, keyed, chip, etc...).
Intermittent outages are still occurring for processing transactions.
Posted Aug 18, 2017 - 13:34 PDT
Update
TSYS is still working on getting all endpoints stable for all processing transactions (swipe, keyed, chip, etc...).
Merchants using the TSYS platform are currently still experiencing intermittent responses to all transaction requests.
Posted Aug 18, 2017 - 12:49 PDT
Monitoring
We are in contact with TSYS and waiting for their resolution.
Posted Aug 18, 2017 - 12:13 PDT
Identified
We are currently investigating an issue connecting to the TSYS processing platform.
All TSYS processing endpoints are going offline and online, we noticed errors and attempting to failover traffic as endpoint become online.
Merchants using the TSYS platform are currently experience intermittent responses to transaction requests.
We are investigating with TSYS NOC.
Posted Aug 18, 2017 - 11:37 PDT
This incident affected: Transactional Processing (Traditional) and Processing Networks (TSYS (Vital)).