29 May 2026

Summary

A routine database update was applied during a deployment. Because of the way the change was written, the system had to rebuild one of its largest and most heavily used tables rather than apply a quick, in-place change. While that rebuild was in progress it temporarily held up other operations, causing a build-up of database connections. As almost every request relies on this part of the database, Astalty became slow or unavailable for approximately 8 minutes, until the rebuild completed and service recovered on its own.

Incident Timeline

Root cause analysis

Root cause

A database update was applied that, because of how the change was written, could not be applied instantly. Instead, the system had to rebuild an entire large table. This particular table is central to Astalty and is used by almost every request, so while it was being rebuilt other operations were held up. Database connections built up faster than they could be cleared, and the application became slow or unavailable until the rebuild finished.

Contributing factors

• Database updates are applied automatically as part of our deployment process, so this change reached production during normal traffic without a separate gate for this type of operation.
• Our existing code review checks and safeguards did not cover this particular type of change, so it wasn't flagged before deployment. This kind of issue is also difficult to reproduce in testing.
• We did not yet have an automated check that predicts whether a database update will apply quickly or require a slow rebuild before it reaches production.

Technical analysis

Some database changes can be applied instantly, while others require the system to rebuild the whole table before they take effect. This change fell into the second category, but that wasn't obvious when it was written, and it isn't the kind of thing our standard testing catches.

Impact assessment

Customer impact

Customers were unable to access Astalty, or experienced significant slowness, for approximately 8 minutes while the update completed. Because the system could still read existing information during this time, many pages continued to load, so most customers experienced degraded performance rather than a complete outage for the full window.

System impact

While the update was in progress, other operations on the affected part of the database were held up. Database connections built up and the application servers were unable to process requests normally until the update completed.

Resolution steps

Resolution summary

The operation was self-correcting and non-destructive. Existing information stayed intact and consistent throughout — there was no data loss or corruption, and no restore or rollback was required. Once the update finished, everything cleared and service recovered automatically.

Detailed steps

• The incident was reported at 19:38 and the cause was identified by 19:42 — within about four minutes — by the team who know the system well.
• Because the affected table was a manageable size, the update completed quickly, so the total incident lasted around 7–8 minutes and resolved without manual intervention.

Lessons learned

What went well

• Fast detection and diagnosis — the team identified the cause within about four minutes of the incident being reported.
• The operation was self-correcting and non-destructive — no data loss, no restore required, and automatic recovery once it finished.
• It was a partial rather than total outage — existing information could still be read throughout, and the manageable size of the affected data kept the impact small.

Areas for improvement

• This type of database change has a subtle failure mode that isn't caught by our standard testing.
• Because updates apply automatically during deployment, other similar changes to large, heavily used parts of the database could cause the same kind of issue.
• We did not have a documented process for responding to this specific situation quickly.

Actions taken

• Added a review guideline so this type of database change cannot be introduced again.