iOS Wireless Microphone issue on Healthcare and Test environments.
Incident Report for nVoq Public
Postmortem

Incident Summary

Incident Start Date: September 19, 2023, 6:35 PM MDT
Incident End Date: September 20, 2023, 1:45 PM MDT
Duration: Approximately 19 hours and 10 minutes
Affected Service: iOS Wireless Microphone Functionality
Service Environment: https://healthcare.nvoq.com

Incident Description

On September 19, 2023, at 6:35 PM MDT, a routine infrastructure update was conducted in the Healthcare environment. This update inadvertently caused a disruption in the iOS wireless microphone functionality. Specifically, the servers responsible for iOS wireless mic functionality were unintentionally removed from their load balancers during the update. This prevented the text of the diction audio sent by the iOS Wireless microphone from appearing in the Voice client window.

 

Incident Timeline

  • September 19, 2023, 6:35 PM MDT: The incident originated with the deployment of an infrastructure update, which led to the inadvertent removal of servers responsible for iOS wireless microphone functionality.
  • September 20, 2023, 9:25 AM MDT: The first customer-reported issue related to the iOS wireless microphone functionality disruption was received for the Healthcare environment.
  • September 20, 2023, 11:02 AM MDT: The infrastructure team was formally notified of the incident and began working on a fix.
  • September 20, 2023, 1:45 PM MDT: A fix for the issue was developed and tested on staging environments, then successfully deployed to the Healthcare environment, restoring normal iOS wireless microphone functionality.

Root Cause Analysis

The root cause of the outage was identified as follows:

Root Cause: During a routine infrastructure update, the servers responsible for iOS wireless microphone functionality were accidentally removed from their associated load balancers.

Contributing Factors: The incident occurred due to a combination of factors, including the complexity of automated testing of iOS hardware.

Preventive Measures: The infrastructure team is implementing additional alarms and monitoring systems to detect unusual and unlikely situations, such as load balancers with zero servers attached. This proactive approach will help identify and address potential issues before they impact service availability.

 

Conclusion

At nVoq, we understand that our Voice service availability is critical to your daily workflow and would like to apologize for the outage affecting the iOS wireless microphone functionality between September 19, 2023, and September 20, 2023.

Our team will continue to monitor the situation closely and implement the necessary changes to ensure the reliability and stability of the Healthcare environment.

If you have any questions or comments, please contact us at support@nvoq.com.

Posted Sep 28, 2023 - 18:33 MDT

Resolved
This incident has been resolved.
Posted Sep 20, 2023 - 18:31 MDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Sep 20, 2023 - 13:45 MDT
Update
A fix for this issue has been deployed to test.nvoq.com and will be deployed to healthcare.nvoq.com in the next 30 minutes.
Posted Sep 20, 2023 - 13:28 MDT
Identified
We have identified an issue that prevents the iOS Wireless Microphone app from communicating with the nVoq Voice client in the healthcare.nvoq.com and test.nvoq.com environments when the Wireless Microphone is configured to use websockets.

A fix for this issues is currently being worked on and we will update this incident when we have more information on the time this fix will be deployed.
Posted Sep 20, 2023 - 11:51 MDT
This incident affected: healthcare.nvoq.com (Healthcare: Websocket Dictations) and test.nvoq.com (Test: Websocket Dictations).