Websocket dictation experiencing elevated error rates

Incident Report for nVoq Public

Postmortem

On Jan 10, 2023, at 7AM MDT we began receiving reports that customers were getting intermittent errors when trying to perform dictations with the General Medicine topic on the Healthcare system.  The type of error message indicated there was not enough dictation capacity to process the requests.   Investigation into the amount of running dictation servers revealed that there was enough capacity to process the requests, but the load balancer was not distributing the incoming dictations correctly across all available dictation servers.  This resulted in a percentage of dictations requests being rejected while other dictations were processed properly.   A change was made at 9:20AM MDT to switch the load balancer distribution algorithm from “round robin” to “least outstanding requests”.  After this change was made, we observed a sharp drop in error levels and dictation performance returned to normal.

At nVoq, we understand that our Voice service availability is critical to your daily workflow and would like to apologize for this healthcare.nvoq.com service degradation. 

If you have any questions or comments, please contact us at support@nvoq.com.

Posted Jan 11, 2023 - 13:22 MST

Resolved

This incident has been resolved.
Posted Jan 10, 2023 - 13:34 MST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jan 10, 2023 - 09:43 MST

Identified

The issue has been identified and a fix is being implemented.
Posted Jan 10, 2023 - 09:37 MST

Investigating

We are experiencing elevated error rates for websocket dictations using the General Medicine topic on the Healthcare system. We are investigating the issue and actively working towards a solution.
Posted Jan 10, 2023 - 07:36 MST
This incident affected: healthcare.nvoq.com (Healthcare: Websocket Dictations).