Summary

On 14 October 2026, between 09:00 and 09:06 IST (03:30 to 03:36 UTC), the Testsigma web application in the US region was unavailable for approximately six minutes. A routine cleanup job caused one database server to restart, and the application servers restarted along with it. Service recovered on its own, and no customer data was lost or changed.

Impact

Only US-region accounts were affected; the EU and India regions ran normally throughout.

• What users saw: errors or timeouts in the web app and API for about six minutes.
• Data: nothing was lost or changed.
• Running tests: continued without interruption, and their results were saved once the application was back.

Timeline

All times are on 14 October 2026 in IST (UTC+5:30).

• 09:00 — A scheduled cleanup job started removing an old test run; one US database server ran out of working memory and restarted.
• 09:01 — Application health checks failed while the database was unavailable, so all US application servers restarted together.
• 09:04 — The database server finished restarting and was back online.
• 09:06 — All application servers were serving traffic again; service fully restored.

Root cause

The database server restarted because a single cleanup operation needed more working memory than its configured limit allowed. Our DevOps team reviewed the crash and confirmed this limit was too low for how our data is structured.

What triggered it. Deleting an old test run also removes everything linked to it: results, settings, configuration and any re-runs, each with linked records of their own. Working through this chain in one go exceeded the database's memory limit for a single operation, which caused it to restart.

Why the whole US region was affected. The application's health check depends on the database. While the database restarted, every application server failed that check and was restarted at the same time, turning one database restart into a short region-wide outage.

What made it more likely. A defect in the cleanup job caused it to retry failed work many times, so these large deletes ran far more often than intended.

Resolution

Service recovered within six minutes once the database and application servers restarted automatically. The fixes below are planned and not yet applied; together they address all three causes: the low memory limit, the cleanup job's retries, and the health check that restarts every server at once.

Short-term measures.

1. Raise the database memory default parameter limit on all US database servers to a level well above what our largest cleanup operations need. Because the change requires a database restart, the DevOps team will roll it out carefully: validate it first on staging databases that mirror production; apply it on a weekend during the lowest-traffic US hours, with the window announced to customers in advance; restart one database server at a time (about 4–5 minutes each), moving on only after the previous one is confirmed healthy; and monitor throughout, with the previous setting ready to restore if needed.
2. Fix the cleanup job so it no longer retries failed work repeatedly, and spread its work across the day.

Long-term measures.

1. Make the health check resilient so a database issue no longer restarts every application server at once.
2. Delete data in smaller batches so no single cleanup operation works through a long chain of linked records.
3. Apply the same memory-limit change in the EU and India regions, using the same safe-window process.