PXL Vision - Cronologia degli avvisi

We are pleased to confirm that the incident affecting PXL Sign is now fully resolved.

The PXL Sign services have been restored to normal operation and full performance.

Resolution Summary: We identified a technical issue that temporarily impacted our manual review process. An immediate workaround is in place, and all manual review services are now fully operational. A permanent fix will be applied shortly.

Key Details:

Thank you for your patience and understanding during this disruption. Please contact our support team if you experience any further issues.

You can view the final status on our Status Dashboard: https://status.pxl-vision.com/.

Sincerely,

The PXL Vision Team

We are pleased to confirm that the incident affecting all services is now fully resolved.

All services have been restored to normal operation and full performance. Customers should no longer be experiencing any issues.

Resolution Summary: Our service provider fixed the issues from their side.

Right now, all services are available and fully functional.

We are now conducting a thorough post-mortem analysis to prevent recurrence.

Key Details:

Thank you for your patience and understanding during this disruption. Please contact our support team if you experience any further issues.

You can view the final status on our Status Dashboard: https://status.pxl-vision.com/.

Sincerely,

The PXL Vision Team

We are pleased to confirm that the incident affecting all services is now fully resolved.

We want to inform you that the service provider has applied a fix. The apps are becoming operational, and we are monitoring the situation. We will continue to provide updates as the situation develops.

Key Details:

We will provide our next update as soon as possible.

Thank you for your continued patience.

You can view the final status on our Status Dashboard: https://status.pxl-vision.com/.

Sincerely,

The PXL Vision Team

This is an update regarding the ongoing incident affecting all services.

Current Status: The root cause of the service performance impact remains unknown at this time. Our service provider is undergoing an investigation with high priority. Services are not completely down, but suffer a performance degradation.
We are closely monitoring the situation and will provide an update with an estimated time of resolution or once service is restored.

Key Details:

Thank you for your continued patience.

You can monitor the status of this incident on our Status Dashboard: https://status.pxl-vision.com/.

Sincerely,

The PXL Vision Team

Summary

On 16 September 2026, between 07:29 and 07:37 UTC, PXL Vision services were unavailable, resulting in failed API requests and interrupted verification sessions. The disruption lasted approximately 8 minutes. Full recovery was verified at 07:54 UTC and the incident was closed at 07:55 UTC.

Earlier the same morning, from 04:05 UTC, a related condition caused a small number of requests to fail intermittently.

The root cause was a change to our internal network configuration, applied by our own engineering team, and the subsequent attempt to roll that change back.

What Happened

Beginning at 04:05 UTC, following a change to strengthen encryption of traffic between our internal services, some of our services began unexpectedly losing their established database connections. This produced intermittent request failures through the morning. To resolve this and enable further investigation, our team decided to roll the encryption change back.

The rollback proved considerably more disruptive than the original change. Reverting the configuration caused services across our production cluster to briefly lose network readiness, making the platform unavailable. Our automated monitoring alerted within two minutes, the team identified the cause immediately, and the previous configuration was re-applied at 07:37 UTC.

Application health was confirmed shortly afterwards and our QA team verified full recovery at 07:54 UTC.

Impact

• Duration: ~8 minutes of unavailability (07:29 UTC – 07:37 UTC, 16 September 2026); recovery verified 07:54 UTC

• Affected services: All PXL Vision production services

• Severity: P1

• Preceding degradation: Between 04:05 and 07:10 UTC, a small number of requests failed intermittently due to the database connection condition described under Root Cause. Affected requests returned errors and could be retried successfully.

• Nature of impact: Service unavailability and failed API requests. Verification sessions in progress did not complete and must be re-run. No loss or corruption of existing data.

Root Cause

Two issues were involved: an unexplained condition that prompted our team to act, and a limitation in how we acted. Only the second is understood.

The outage was caused by a known limitation in a third-party open-source networking component. Removing services from its encrypted layer is not a zero-downtime operation, though applying it is. We had reason to believe the reversal was safe: the documentation describes the operation as non-disruptive, and we had performed it successfully before. The maintainers' acknowledgement that it is not reliable sat in a closed issue thread, which we found only while investigating this incident.

The condition that prompted the reversal, internal services losing their established database connections, is not yet explained. It cleared once the affected services were restarted and has not recurred. We have added monitoring to detect it immediately should it return.

What We Learned

• Repeated success does not establish safety when a failure mode is non-deterministic. This operation had previously completed in production without incident and passed in our lower environments. Neither outcome was evidence of safety: the underlying failure is probabilistic. We had been reading successful attempts as a safety record. They were not one.

• Known limitations are not always documented where we would find them. The documentation described this operation as non-disruptive; the contradicting evidence sat in a closed issue thread we only found while investigating. Where a component's guarantees are load-bearing for us, we now check its issue history and test criteria, not the documentation alone.

• Clearing a symptom is not resolving a cause. Restarting the affected services stopped the connection errors. It did not explain them. The mechanism remains unexplained, so we have put detection in place rather than assuming it is resolved.

Actions Taken & Next Steps

This incident originated with our own change, and we take full responsibility for it. The following actions have been or are being taken:

• Affected services restarted: All remaining stale internal connections were cleared.

• Automated detection for the error signature: Production alerting now fires immediately on the specific database connection error involved in this incident.

• Ongoing production monitoring: We are actively monitoring production for any recurrence of this behaviour. It has not reappeared since services restarted.

Closing Note

We sincerely apologize for the disruption this incident caused to you and your workflows. This outage resulted from our own change and our own decision to reverse it, and we take full ownership of both.

The work behind the change was intended to strengthen the security of the platform. That does not make the interruption to your service any more acceptable, and we are applying the lessons above so that improvements of this kind do not come at your expense.

If you have questions about this incident, or if a verification flow in your integration was affected and you require specifics, our support team is ready to help.

— The PXL Vision Team

We are pleased to confirm that the incident affecting Our services is now fully resolved.

All services have been restored to normal operation and full performance. Customers should no longer be experiencing issues while using the application.

Resolution Summary: Our engineering team identified the issue, applied a corrected configuration, and confirmed that the service has returned to normal operation.

We are now conducting a thorough post-mortem analysis to prevent recurrence.

Key Details:

Thank you for your patience and understanding during this disruption. Please contact our support team if you experience any further issues.

You can view the final status on our Status Dashboard: https://status.pxl-vision.com/.

Sincerely,

The PXL Vision Team