Network Connectivity Loss at Primary Datacenter

Incident Report for nVoq Public

Postmortem

On the evening of October 8, nVoq engineers performed network maintenance at the primary datacenter serving the SayIt platform at healthcare.nvoq.com and secure.nvoq.com. This maintenance was routine and was not expected to disrupt SayIt services. Due to a configuration error in one of the network edge devices, a switching loop was created that disrupted access to the SayIt platform at healthcare.nvoq.com and secure.nvoq.com from 8:11PM MDT to 8:17PM MDT.

We understand that SayIt is part of your daily critical workflow and would like to apologize for this service interruption. We have added new procedures to our network maintenance checklists to prevent the issue from reoccurring. If you have any questions or issues with your SayIt service, please contact us at support@nvoq.com.

Posted Oct 10, 2019 - 01:06 MDT

Resolved

This incident has been resolved.

Posted Oct 08, 2019 - 23:06 MDT

Monitoring

From 8:11PM MDT until 8:15PM MDT network connectivity was interrupted at the primary datacenter, and Sayit services were disrupted while automated systems sent traffic to the seconday datacenter.

By 8:18PM MDT all SayIt traffic was being processed at the primary datacenter again. We are monitoring the network connections and will update this incident when satisfied everything is resolved.

Posted Oct 08, 2019 - 20:27 MDT

This incident affected: healthcare.nvoq.com (Healthcare: nVoq Voice Administration Portal, Healthcare: Voice Shortcuts, Healthcare: HTTPS Dictations, Healthcare: Websocket Dictations).