Incident Summary

Incident Start Date: March 6, 2024, 8:10 AM MST
Incident End Date: March 6, 2024, 1:30 PM MST
Duration: Approximately 5 hours and 20 minutes
Affected Service: Admin console Review and Correct page
Service Environment: https://healthcare.nvoq.com

‌

Incident Description

On March 6, 2024 at approximately 8:10 AM MST, details for some dictations performed on the Healthcare system stopped appearing on the Review and Correct page. Dictations were transcribed and the text was returned to users, but the dictation details for the Review and Correct page were not being returned.

The transcription transaction records were being stored within the database, but not populated within the new tables now used by the review and correct calls. Although, there were some entries being stored properly in new database table used by Review and Correct.

At 8:30 AM MST, the nVoq operations checks reported that there were issues with the end-to-end dictation processing of multiple test dictations. The investigation started and it was determined that the behavior was similar to the incident on November 9, 2023.

At 8:45 AM MST, the nVoq operations team began to replace the affected application servers in the Healthcare system and dictation details began appearing on the Review and Correct page within 20 minutes. All servers were updated by 9:20 AM MST.

Details for the dictations performed from 8:10AM to 9:20AM were backfilled to the Review and Correct page between 1:00 PM and 1:30 PM MST that same day..

Incident Timeline

• March 6, 2024, 8:10 AM MST: Dictation details stop appearing on the Review and Correct page for the Healthcare system.
• March 6, 2024, 8:30 AM MDT: The operations team is notified via automated check that there were issues with the end-to-end dictation processing of multiple test dictations.
• March 6, 2024, 8:45 AM MDT: The application servers are replaced with new application server instances.
• March 6, 2024, 9:20 AM MDT: The Healthcare Review and Correct page is populating all dictation details again.
• March 6, 2024, 1:30 PM MDT: Missing dictation details for the period of 8:10AM to 9:20AM are backfilled to the Review and Correct page.

‌

Root Cause Analysis

The root cause of the outage was identified as follows:

Root Cause: An error occurred which resulted in the dictation details not being stored in a timely fashion for the R&C page.

Preventive Measures: Software changes have been made in the upcoming 19.0.0 release which should reduce the likelihood of this issue.
 

Conclusion

At nVoq, we understand that our Voice service availability is critical to your daily workflow and would like to apologize for the Review and Correct service degradation outage affecting the Healthcare environment on March 6, 2024.

We were able to identify the issue with automated monitoring and start the work to address the problem. Our team will continue to monitor the situation closely and implement the necessary changes to ensure the reliability and stability of the Healthcare environment.