Possible degraded JSON API performance
Incident Report for PayTrace
Postmortem

Business Impact

  • Clients called in stating authorizations could not go through via JSON & Traditional API.
  • API requests suffered errors upon response request being malformed.

Root Cause

  • There was a minimal misstep when releasing database enhancements. We recognized the situation quickly when it began, isolated the cause and performed a rollback.

Preventative Measures

  • The problem was not caused by our systems or any infrastructure concerns, but as a result of oversight by people working on complicated systems. We identified a test coverage use case for this scenario, we have added this to our additional automated validation checks.
  • The next day we were able to release our database enhancements with non-disruptive results.

We apologize for any inconvenience caused by this issue.

Posted 9 months ago. Oct 08, 2018 - 15:22 PDT

Resolved
We have stopped the negative impact. We will be investigating root cause and will provide a postmortem when available.
Posted 10 months ago. Sep 24, 2018 - 16:31 PDT
Investigating
We are currently investigating an issue with our JSON API regarding possible degraded performance. Updates will be posted as they become available.
Posted 10 months ago. Sep 24, 2018 - 16:06 PDT
This incident affected: JSON API.