Network Connectivity Loss at Primary Datacenter
Incident Report for nVoq Public
Postmortem

On the evening of October 8, nVoq engineers performed network maintenance at the primary datacenter serving the SayIt platform at healthcare.nvoq.com and secure.nvoq.com. This maintenance was routine and was not expected to disrupt SayIt services. Due to a configuration error in one of the network edge devices, a switching loop was created that disrupted access to the SayIt platform at healthcare.nvoq.com and secure.nvoq.com from 8:11PM MDT to 8:17PM MDT.

We understand that SayIt is part of your daily critical workflow and would like to apologize for this service interruption. We have added new procedures to our network maintenance checklists to prevent the issue from reoccurring. If you have any questions or issues with your SayIt service, please contact us at support@nvoq.com.

Posted 5 days ago. Oct 10, 2019 - 01:06 MDT

Resolved
This incident has been resolved.
Posted 6 days ago. Oct 08, 2019 - 23:06 MDT
Monitoring
From 8:11PM MDT until 8:15PM MDT network connectivity was interrupted at the primary datacenter, and Sayit services were disrupted while automated systems sent traffic to the seconday datacenter.

By 8:18PM MDT all SayIt traffic was being processed at the primary datacenter again. We are monitoring the network connections and will update this incident when satisfied everything is resolved.
Posted 7 days ago. Oct 08, 2019 - 20:27 MDT
This incident affected: SayIt at healthcare.nvoq.com (Healthcare: SayIt Administration Portal, Healthcare: Voice Shortcuts, Healthcare: HTTPS Dictations, Healthcare: Websocket Dictations) and SayIt at secure.nvoq.com (Secure: SayIt Administration Portal, Secure: Voice Shortcuts, Secure: HTTPS Dictations, Secure: Websocket Dictations).