[canada.nvoq.com] Completed dictations and shortcuts not showing on the Review & Correct page

Incident Report for nVoq Public

Postmortem

To display entries on the SayIt Administrator Review & Correct page, metadata from completed dictations and shortcuts is populated from a read-replica database server. This metadata is replicated from the primary database to the read-replica before being displayed on the Review & Correct page.

On Sunday, July 21, 2019 at 12:51PM MDT, the read-replica database instance for the canada.nvoq.com SayIt system became unavailable and replication of the dictation and shortcut metadata from the primary database server stopped. Consequently, the Review & Correct page stopped populating new entries for completed dictations and shortcuts.

At 02:25PM MDT, replication between the primary and the read-replica databases was re-established, and the missing Review & Correct entries began populating on the page.

This incident did not impact real-time dictation or shortcut activity and all dictations and shortcuts performed during the incident time frame are populated in the Review & Correct page.

Posted Jul 22, 2019 - 23:13 MDT

Resolved

The read replica database instance has fully recovered and the Canada SayIt Administration Portal Review & Correct page is correctly showing all completed dictations and shortcuts.

Posted Jul 21, 2019 - 15:31 MDT

Update

We are continuing to monitor for any further issues.

Posted Jul 21, 2019 - 14:43 MDT

Monitoring

We have implemented a fix, and the Canada SayIt Administration Portal is now populating new dictations and shortcuts on the Review & Correct page. All previous dictations and shortcuts performed since the incident started are now populated on the Review & Correct page as well. We are monitoring the Canada system closely and will update this incident in one hour.

Posted Jul 21, 2019 - 14:42 MDT

Identified

The read replica database instance that populates the Review & Correct page has become unavailable. Completed dictations and shortcuts are not showing up on the page. We are working on recovering the read replica database. Normal SayIt dictations and shortcuts are being processed.

Posted Jul 21, 2019 - 12:55 MDT

This incident affected: canada.nvoq.com (Canada: nVoq Voice Administration Portal).