Websocket dictation experiencing elevated error rates
Incident Report for nVoq Public
Postmortem

On Jan 10, 2023, at 7AM MDT we began receiving reports that customers were getting intermittent errors when trying to perform dictations with the General Medicine topic on the Healthcare system.  The type of error message indicated there was not enough dictation capacity to process the requests.   Investigation into the amount of running dictation servers revealed that there was enough capacity to process the requests, but the load balancer was not distributing the incoming dictations correctly across all available dictation servers.  This resulted in a percentage of dictations requests being rejected while other dictations were processed properly.   A change was made at 9:20AM MDT to switch the load balancer distribution algorithm from “round robin” to “least outstanding requests”.  After this change was made, we observed a sharp drop in error levels and dictation performance returned to normal.

At nVoq, we understand that our Voice service availability is critical to your daily workflow and would like to apologize for this healthcare.nvoq.com service degradation. 

If you have any questions or comments, please contact us at support@nvoq.com.

Posted Jan 11, 2023 - 13:22 MST

Resolved
This incident has been resolved.
Posted Jan 10, 2023 - 13:34 MST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jan 10, 2023 - 09:43 MST
Identified
The issue has been identified and a fix is being implemented.
Posted Jan 10, 2023 - 09:37 MST
Investigating
We are experiencing elevated error rates for websocket dictations using the General Medicine topic on the Healthcare system. We are investigating the issue and actively working towards a solution.
Posted Jan 10, 2023 - 07:36 MST
This incident affected: healthcare.nvoq.com (Healthcare: Websocket Dictations).