PXL Vision - Incident - Performance Degradation – Details zu Vorfällen

Incident Summary: Performance Degradation – March 5–9, 2026

Between March 5 and March 9, 2026, customers using PXL Cloud and PXL Ident experienced intermittent slowness and increased latency, particularly affecting flows involving large video uploads. We sincerely apologize for the disruption this caused.

What happened

The degradation was traced to a dataplane update rolled out by our datacenter provider to their Network Load Balancer infrastructure. Following the update, requests exhibited significantly higher latency on the ingress path. Our internal monitoring detected the issue within minutes, and our team began investigating immediately.

How we resolved it

After several days of close collaboration with our datacenter provider's support team — including extensive diagnostic testing and log analysis — the decision was made to fully rollback the infrastructure changes. Following the rollback, completed on the evening of March 9, services returned to normal and were confirmed stable.

What we're doing going forward

We will continue to monitor our infrastructure closely during any planned maintenance windows from third-party providers. We are also working on improving our alerting and diagnostic tooling to reduce response time in similar situations in the future.

Thank you for your patience throughout this incident.

The performance degradation incident has been resolved. We will provide a post-mortem as soon as we collect the necessary data.

We implemented a fix with our third-party provider and are currently monitoring the results. The situation looks stable right now.

We are continuing to work on a fix for this incident. We were working with a third-party to identify the issue. We hope to update you shortly.