US1 - Connector Delays
Incident
Report for Gainsight
Postmortem
Incident:
Beginning around 12:00 UTC on the 13th of June, Engineers were alerted of elevated queue levels for Connector services in CS-US1.
Root Cause:
A leader node was found to have higher than usual disk activity which prevented optimal job execution for Connector services.
Recovery Action:
Engineers scaled the number of Connector instances to correct the issue temporarily. Additionally, Engineers skipped long-running and duplicate jobs to help recover.
Preventive Measures:
System configuration adjustments have been made to prevent these issues moving forward.
Resolved
This incident has been resolved. A subset of customers faced connector queue delays during this incident window. We will add RCA details as they become available.
Monitoring
A fix has been implemented and we are monitoring. The Connectors queue was blocked for analysis and troubleshooting during this incident. We have since unblocked, and any duplicate sync jobs were aborted with no data impact. Please expect delays while the queue clears.
Identified
The issue has been identified and a fix is being implemented.
Investigating
Beginning around 12:00 PM UTC today, we detected a delay in Connectors traffic and adjusted accordingly. As we still have queue delays, we are investigating further and will update as more information becomes available.
This incident affected: Gainsight CS - US1 Region (US1 Data Ingestion Queue).