Identified major outage

Incident Report for Universign

Postmortem

On Tuesday 27th, at 17:09, we withstood a software crash of our main database.

As a result, all our services were down for 3 minutes until the backup database took over.

These three minutes were followed by a 7 minute period during which performance was degraded. No data were lost during this incident.

Unfortunately, the crash triggered the desynchronisation of several database components.

Therefore, we had to perform an urgent maintenance in order to resynchronize all data.

This lead to an other downtime of one minute between 19:10 to 19:11, followed by 13 minutes of slightly degraded performance.

Sincerely,

Posted Nov 28, 2018 - 15:03 CET

Resolved

The maintenance operations were successfully completed.

The incident is closed.

We are preparing a global report about this incident.

We deeply apologize for the inconvenience.

Posted Nov 27, 2018 - 19:43 CET

Update

The incident is fixed. However a database was affected, and we need an urgent maintenance window to make sure no data becomes at risk. We will perform maintenance at 7 pm for a few minutes. The service could briefly go down at that time.

Posted Nov 27, 2018 - 18:18 CET

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Nov 27, 2018 - 17:22 CET

Identified

We are facing a major outage.

The problem is identified and we are working on a fix right now.

Posted Nov 27, 2018 - 17:19 CET

This incident affected: Signature management API, Timestamp API, Seal API, and Web Application.