29 July 2026

Summary

Astalty's database ran out of available storage, which meant it could no longer process write operations and the application became unavailable. Storage autoscaling was enabled, but the instance had already reached its configured maximum storage limit and could not grow any further. All customers were unable to access Astalty via the web or the native app for approximately 7 minutes. No customer data was lost.

Incident Timeline

Root cause analysis

Root cause

Our AWS RDS database instance ran out of available storage. Storage autoscaling was enabled but the instance had reached its configured maximum storage limit and could not grow any further.

Once storage was fully consumed, the database was unable to process write operations, which caused the application to become unavailable.

Contributing factors

• Autoscaling was configured, but a hard maximum storage limit was applied, meaning it would not scale past that point.
• Excessive logging was occurring, which caused unexpected database storage growth.
• Insufficient alarms were configured to notify us about low database storage.

Technical analysis

RDS storage autoscaling increases allocated storage automatically when free space runs low, but only up to a configured maximum storage threshold. It does not scale beyond that value under any circumstances.

Sustained storage growth — accelerated by a higher-than-expected volume of logging — carried the instance up to its configured maximum storage threshold. At that point autoscaling had no remaining headroom to add capacity.

Free storage continued to fall until it reached zero. With storage exhausted, the database could no longer accept writes, and application servers began throwing exceptions, leading to a full outage.

Because no low-storage alarm was configured with sufficient headroom, the condition was not surfaced in advance and was only detected once the outage itself occurred.

Impact assessment

Customer impact

• All customers were unable to access Astalty via the web or the native app during the incident window.
• Duration of impact: approximately 7 minutes.
• No customer data was lost.

System impact

• Database storage was fully exhausted, preventing the database from processing write operations.
• Application servers became unresponsive due to the database being unable to accept writes.

Business analysis

• Temporary disruption to all active customers.

Resolution steps

Resolution summary

The issue was resolved by manually reconfiguring the RDS instance's storage to add capacity beyond the previous limit, which restored normal database operation and application access.

Detailed steps

• Identified that the RDS instance had exhausted its available storage and reached its configured maximum storage threshold by inspecting AWS.
• Manually reconfigured the instance's storage to provide additional capacity above the previous limit.
• Monitored the instance as capacity was applied and the database resumed accepting writes.

Verification

• Customer access to the application was restored.
• Application exceptions generated during the outage stopped.
• Internal verification confirmed no data loss occurred.

Lessons learned

What went well

• The team responded immediately and took decisive action to restore service.
• Our outage notifications alerted the right people that Astalty was down, enabling a fast response.
• The issue was identified and resolved quickly (within approximately 7 minutes).
• No data loss occurred.

Areas for improvement

• Additional alarms are needed to notify us of low database storage with sufficient headroom to act before storage is exhausted.
• Excessive logging needs to be reduced to prevent unexpected, runaway database storage growth.
• The maximum storage threshold and headroom on the instance should be reviewed so autoscaling has room to respond to future growth.

Key takeaways

• Database storage must have both proactive alerting and sufficient autoscaling headroom so capacity limits are surfaced before they cause an outage.
• Logging volume must be controlled so it does not become an unmanaged driver of storage growth.

Follow-ups