Elevated errors in NXT Authentication

Incident Report for Gainsight

Postmortem

On 2021-04-20, NXT customers not already logged in were unable to access their NXT instance between 15:27 and 19:43 UTC due to the third party Auth0 experiencing a major outage. This service disruption impacted our US NXT instance only.

Customers already logged in, as well as SFDC and EU customers were not affected.

After receiving alerts, our team verified that authentication was not possible for new NXT sessions. Any existing NXT sessions that were already authenticated were able to continue working as expected.

From the service provider side, the issue was caused from large collection scans resulting in database locks and resource contention.

Actions taken by the service provider to mitigate and prevent moving forward:

Implemented a fix for the database queries in question
Increased database capacity
Worked with the database software vendor for performance analysis
Implemented additional monitoring of system performance

We are exploring all possible options with the vendor to increase resiliency. We take availability and performance seriously and will continue to make improvements for the best possible experience.

Posted Apr 23, 2021 - 21:58 UTC

Resolved

As authentication services remain stable, we are closing this incident. We will post root cause information as we receive it.

Posted Apr 20, 2021 - 22:28 UTC

Monitoring

All users are now able to log in. We will continue to monitor with the service provider.

Posted Apr 20, 2021 - 19:55 UTC

Update

Authentication is still down for new NXT logins. Existing sessions that were already logged in or using the SFDC version are still not affected. Thank you for your patience as we work with the upstream service provider to resolve this issue.

Posted Apr 20, 2021 - 19:41 UTC

Update

Posted Apr 20, 2021 - 18:41 UTC

Update

Authentication is still down for new NXT logins. Existing sessions that are already logged in or using the SFDC version are still not affected. We are working with the upstream service provider.

Posted Apr 20, 2021 - 17:41 UTC

Update

Authentication is still down for new NXT logins. We are working with the upstream service provider.

Posted Apr 20, 2021 - 16:39 UTC

Investigating

We have identified an issue with our Upstream Service provider causing Authentication errors while logging into the Gainsight NXT application.

Users who are already logged in into NXT version or using the SFDC version of Gainsight are not affected.

We are closely monitoring and will post updates as soon as they are available.

Posted Apr 20, 2021 - 15:57 UTC

This incident affected: Gainsight CS - US1 Region (Gainsight CS US1 Application).