29 June 2026

Summary

A routine database update was applied during a deployment. Because of the type of change, the system had to rebuild one of its largest and most heavily used tables rather than apply a quick change. While the rebuild was in progress it held up other operations and caused database connections to build up. As this part of the database is central to almost everything in Astalty, the application became unavailable to all customers for approximately 24 minutes (10:02am to 10:26am).

Incident Timeline

Root cause analysis

Root cause

A database update was applied to one of our largest and most heavily used tables. Because of the type of change, the system could not apply it instantly and instead had to rebuild the entire table, which held up other operations while it was in progress. As this part of the database is used by almost everything in Astalty, this caused a full outage. A similar change was included for a second table in the same update.

Contributing factors

• An automated safety check did flag the change as one that could be slow, but the warning was advisory only and did not stop the change from being merged or deployed.
• The change was reviewed and not identified as risky. In part this was because no customer-facing functionality relied on it yet, but this kind of update carries the same deployment risk as a functional change.
• We had guidelines covering some risky database changes, but not this specific type.
• The update was deployed during business hours, when the system is under heavier load.

Technical analysis

Some database changes can be applied instantly, while others require the system to rebuild the entire table before they take effect — and while that rebuild is happening, other operations are held up. This change required a full rebuild of a very large, heavily used table, which is why the impact was significant.

Impact assessment

Customer impact

All customers experienced a complete outage for approximately 24 minutes (from 10:02am to 10:26am). During this window, customers could not log in, load pages, or perform any operations.

Resolution steps

Resolution summary

The platform recovered after the database was restarted, which cleared the held-up operations. The change that caused the issue was then reversed. We also began improving our automated safety checks so that warnings about this kind of high-impact change are harder to miss in future.

Detailed steps

• Incident detected (10:02am): Astalty was unavailable; customers could not log in or use the app.
• Cause identified: a database update was rebuilding a large, heavily used table and holding up other operations.
• Service restored (10:26am): the database was restarted, clearing the blockage.
• Change reversed: the update that caused the issue was subsequently rolled back to prevent it happening again.

Lessons learned

What went well

• Our automated safety checks did identify the risky change before it was merged.
• The team responded quickly and contained the outage to about 24 minutes.
• A previous incident had already led to new safeguards; this incident extends that learning further.

Areas for improvement

• Make high-impact warnings unmissable. Warnings about slow changes to our largest, most heavily used tables should be impossible to overlook and should require explicit sign-off before deploying.
• Deploy policy for critical changes. Higher-risk changes to the most heavily used parts of the database should be deployed outside business hours.
• Simplify where we can. Review whether some database rules can be simplified to reduce the risk of this kind of change.

Actions taken

• All blocking operations on our database are now automatically flagged and blocked from being merged until approved by two engineering leads.
• We are working on a longer-term solution — improvements to our internal processes, automations and automated checks to prevent this from happening again.