Incident Summary

Incident Start Date: November 9, 2023, 10:00 AM MST
Incident End Date: November 9, 2023, 2:20 PM MST
Duration: Approximately 4 hours and 20 minutes
Affected Service: Admin console Review and Correct page
Service Environment: https://healthcare.nvoq.com

‌

Incident Description

On November 9, 2023 at approximately 10:00AM MST, details for dictations performed on the Healthcare system stopped appearing on the Review and Correct page. Dictations were transcribed and the text was returned to users, but the dictation details for the Review and Correct page were not being returned.

The transcription transaction records were being stored within the database, but not populated within the new tables now used by the review and correct calls. Although, there were some entries being stored properly in new database table used by Review and Correct.

At 2:00PM MST, the nVoq operations team began to replace the affected application servers in the Healthcare system and dictation details began appearing on the Review and Correct page within 20 minutes.

Details for the dictations performed from 10:00AM to 2:20PM were backfilled to the Review and Correct page later in the evening.

Incident Timeline

• November 9, 2023, 10:00 AM MST: Dictation details stop appearing on the Review and Correct page for the Healthcare system.
• November 9, 2023, 1:08 PM MDT: The operations team is notified that customers are reporting an issue with the Review and Correct page and begins remediation efforts.
• November 9, 2023, 2:00 PM MDT: The application servers are replaced with new application server instances.
• November 9, 2023, 2:20 PM MDT: The Healthcare Review and Correct page starts populating dictation details again.
• November 9, 2023, 9:14 PM MDT: Missing dictation details for the period of 10:00AM to 2:20PM are backfilled to the Review and Correct page.

‌

Root Cause Analysis

The root cause of the outage was identified as follows:

Root Cause: An error occurred which resulted in the dictation details not being stored in a timely fashion for the R&C page.

Preventive Measures: We are making changes to our internal monitoring to detect if this problem occurs in the future. Software changes have been made in the upcoming 19.0.0 release which should reduce the likelihood of this issue.
 

Conclusion

At nVoq, we understand that our Voice service availability is critical to your daily workflow and would like to apologize for the Review and Correct service degradation outage affecting the Healthcare environment on November 9, 2023.

Our team will continue to monitor the situation closely and implement the necessary changes to ensure the reliability and stability of the Healthcare environment.

If you have any questions or comments, please contact us at support@nvoq.com.