We apologize for the inconvenience caused by the incident on the 18/10/2023 between 3:05 PM and 5:08 PM.

‌

Timeline:

18/10/2023 3:05 PM: Latency alarm on WS service
18/10/2023 3:09 PM: Intermittent alarms on unavailability of WS service

‌

‌Resolution of the incident:

18/10/2023 3:09-4:15 PM: Backoffice stopped
18/10/2023 3:09-4:25 PM: Reboot of overloaded servers
18/10/2023 3:20 PM: Server taken out and back into the pool
18/10/2023 4:25 PM: Server taken out and back into the pool
18/10/2023 4:31 PM: IP address blocked‌

‌

End of incident:
18/10/2023 5:08 PM: All services are operational
19/10/2023 12:27 PM: IP address unblocked
‌

Identified root cause:

We are investigating high contention at the database level which caused increased latency of queries, going so far as to make the service unavailable.
‌

Preventive actions:

We are investigating high contention at the database level which caused increased latency of queries, going so far as to make the service unavailable.

We are investigating changing our database management service.

We are investigating integrating a query mitigation service.