Degraded performance in Timeline
Incident Report for Gainsight
Postmortem

Issue:  Degraded performance in Timeline
An unusual amount of database traffic on a single database cluster temporarily caused an increase in errors and latency for customers using our Timeline feature.

Timeline on the 16th of September:
14:59 UTC - Synthetic monitoring alerts and customer reports signaled issues with accessing our Timeline feature so we opened an incident and started investigation.
15:05 UTC - By this time, we discovered and prevented the offending traffic from causing further impact.
15:15 UTC - After stability as confirmed, the Gainsight US Status Page was updated.
17:18 UTC - The incident status was closed after observing a period of stability.

Resolution:
Once identified, we blocked the offending statements and prevented related traffic.
We are in the process of adding timeout and resilience logic to prevent issues like this and expect to be complete within Q1 of 2022.
We take availability and performance seriously and will continue to make improvements for the best possible experience. Please email support@gainsight.com if there are further questions.

Posted Nov 16, 2021 - 17:25 UTC

Resolved
This incident has been resolved. RCA details will be posted once available.
Posted Sep 16, 2021 - 17:18 UTC
Update
While the performance is at a normal level, we continue to monitor the services for any further issues.
We will post further updates once we have more information.
Posted Sep 16, 2021 - 16:16 UTC
Monitoring
We have taken recovery action and the performance is now back to normal levels.
Posted Sep 16, 2021 - 15:15 UTC
Investigating
We are investigating performance issues related to Timeline and will update as we have more information.
Posted Sep 16, 2021 - 14:59 UTC
This incident affected: Gainsight CS - US1 Region (Gainsight CS US1 Application).