A Transient Network fault resulted in a few of the running processes getting into a hung state. While the resiliency ensured the subsequent jobs to succeed, a few hung jobs remained in the queue for a longer time.
Recovery Action :
a) Restarting of services
b) Manual intervention to retrigger the hung jobs
c) Timeline entries are populated
While the applications have a high level of resiliency in place to handle network issues, we are working on further enhancements to catch even remote cases.
We apologize for the inconvenience caused. Please reach out to firstname.lastname@example.org
if you have questions.
All queues are back to expected levels. Email sent between 3:30 AM and 3:30 PM UTC may not currently show in Timeline for some customers. No data was lost but we will be deploying a fix to re-synchronize timeline entries.
We are still investigating Root Cause and will update this incident once the Timeline entries are populated and analysis is complete.