Backend-Issues
We're currently seeing a higher amount of timeouts and 5xx errors.
The team is investigating further.
Upgrading to major-outage, it affects a lot of TSS.
The team is giving it's best.
A first root-cause is analyzed and being tackled.
Creation of new TSS has been blocked for now.
since 15:18 (16:18 UTC+1) - we're back to >100 positive transactions per seconds. We're further increasing capacities an bringing the system back to operational state.
15:34 (16:34 UTC+1) - we're back to 250+ tx/s
Back to 350+ tx/s
In case you're seeing E_SMAERS_GATEWAY_CAPACITIES_DEPLETED, this is due to new TSS being temporarily disabled while the incident is ongoing.
The effect of the incident is treated as RESOLVED 🎉.
We're now going over to a manual monitoring and cleanup.
Status set to RESOLVED
FYI - Back to 1k+ tx/s - operational an stable