Possible degraded JSON API performance
Incident Report for PayTrace
Postmortem

Business Impact

  • Clients called in stating authorizations could not go through via JSON & Traditional API.
  • API requests suffered errors upon response request being malformed.

Root Cause

  • There was a minimal misstep when releasing database enhancements. We recognized the situation quickly when it began, isolated the cause and performed a rollback.

Preventative Measures

  • The problem was not caused by our systems or any infrastructure concerns, but as a result of oversight by people working on complicated systems. We identified a test coverage use case for this scenario, we have added this to our additional automated validation checks.
  • The next day we were able to release our database enhancements with non-disruptive results.

We apologize for any inconvenience caused by this issue.

Posted 8 months ago. Oct 08, 2018 - 15:22 PDT

Resolved
We have stopped the negative impact. We will be investigating root cause and will provide a postmortem when available.
Posted 8 months ago. Sep 24, 2018 - 16:31 PDT
Investigating
We are currently investigating an issue with our JSON API regarding possible degraded performance. Updates will be posted as they become available.
Posted 8 months ago. Sep 24, 2018 - 16:06 PDT
This incident affected: JSON API.