Incident Summary
Incident Start Date: March 25, 2024, 6:23 AM MST
Incident End Date: March 25, 2024, 6:39 AM MST
Duration: Approximately 16 minutes
Affected Service: Admin console, HTTP Dictations, Voice Shortcuts
Service Environment: https://healthcare.nvoq.com
Incident Description
On March 25, 2024, 6:28 AM MST, the nVoq operations team began receiving alerts that automated checks of the Healthcare environment were failing. To resolve the issue, the nVoq operations team replaced the front-end application servers for the Healthcare service. This incident is similar to the incident on Jan 23, 2024.
Incident Timeline
Root Cause Analysis
The root cause of the outage was identified as follows:
Root Cause: Preliminary investigation indicates an issue with third party token authentication processing. The token verification step was delayed for a number of requests which backed up the application servers.
Contributing Factors: When the authentication tokens could not be verified from the third party, our application servers did not release the resources used to validate the tokens.
Preventive Measures: We will be updating our application server code to better handle token processing when the third party token verification servers are unavailable.
Conclusion
At nVoq, we understand that our Voice service availability is critical to your daily workflow and would like to apologize for the outage affecting the Healthcare environment on March 25, 2024.
Our team will continue to monitor the situation closely and implement any necessary changes to ensure the reliability and stability of the Healthcare environment.