Incident Start Date: September 19, 2023, 6:35 PM MDT
Incident End Date: September 20, 2023, 1:45 PM MDT
Duration: Approximately 19 hours and 10 minutes
Affected Service: iOS Wireless Microphone Functionality
Service Environment: https://healthcare.nvoq.com
On September 19, 2023, at 6:35 PM MDT, a routine infrastructure update was conducted in the Healthcare environment. This update inadvertently caused a disruption in the iOS wireless microphone functionality. Specifically, the servers responsible for iOS wireless mic functionality were unintentionally removed from their load balancers during the update. This prevented the text of the diction audio sent by the iOS Wireless microphone from appearing in the Voice client window.
Root Cause Analysis
The root cause of the outage was identified as follows:
Root Cause: During a routine infrastructure update, the servers responsible for iOS wireless microphone functionality were accidentally removed from their associated load balancers.
Contributing Factors: The incident occurred due to a combination of factors, including the complexity of automated testing of iOS hardware.
Preventive Measures: The infrastructure team is implementing additional alarms and monitoring systems to detect unusual and unlikely situations, such as load balancers with zero servers attached. This proactive approach will help identify and address potential issues before they impact service availability.
At nVoq, we understand that our Voice service availability is critical to your daily workflow and would like to apologize for the outage affecting the iOS wireless microphone functionality between September 19, 2023, and September 20, 2023.
Our team will continue to monitor the situation closely and implement the necessary changes to ensure the reliability and stability of the Healthcare environment.
If you have any questions or comments, please contact us at firstname.lastname@example.org.