Incident Summary

Incident Start Date: January 23, 2024, 6:24 AM MST
Incident End Date: January 23, 2024, 6:38 AM MST
Duration: Approximately 14 minutes
Affected Service: Admin console, HTTP Dictations, Voice Shortcuts
Service Environment: https://healthcare.nvoq.com

‌

Incident Description

On January 23, 2024, 6:24 AM MST, the nVoq operations team began receiving alerts that automated checks of the Healthcare environment were failing. To resolve the issue, the nVoq operations team replaced the front-end application servers for the Healthcare service.

Incident Timeline

• January 23, 2024, 6:24 AM MST: The first alert was received that an automated check was failing on the Healthcare environment.
• January 23, 2024, 6:34 AM MST: The operations team began a refresh of the application server instances.
• January 23, 2024, 6:38 AM MST: The Healthcare environment returns to a normal state.

‌

Root Cause Analysis

The root cause of the outage was identified as follows:

Root Cause: Preliminary investigation indicates an issue with third party token authentication processing. The token verification step was delayed for a number of requests which backed up the application servers.

Contributing Factors: When the authentication tokens could not be verified from the third party, our application servers did not release the resources used to validate the tokens.

Preventive Measures: We will be updating our application server code to better handle token processing when the third party token verification servers are unavailable.

Conclusion

At nVoq, we understand that our Voice service availability is critical to your daily workflow and would like to apologize for the outage affecting the Healthcare environment on January 23, 2024.

Our team will continue to monitor the situation closely and implement any necessary changes to ensure the reliability and stability of the Healthcare environment.