Uh-oh, We Might Be Down…

Postmortem - Roger EU, 13 August 2026

What happened

On Thursday 13 August 2026, Roger's EU service (eu.roger.myq.cloud) was disrupted for approximately 72 minutes, including around 23 minutes of full unavailability.

The message broker that Roger uses to coordinate work between its components became overloaded and lost the internal consensus it needs to operate. Roger's API, device endpoints and web interface depend on that broker, so they became slow and then unavailable.

Customer impact

Roger was unreachable or unreliable for EU customers during the window above.
Print jobs submitted during the incident may have failed to save and would have needed to be resubmitted. This is the main user-visible data impact.
Devices and desktop clients disconnected and reconnected automatically once the service was restored. No customer action was required to recover.
No stored customer data was lost or exposed, and no data restore was required. The exception is the print jobs noted above, which were not saved at the time of submission and needed resubmitting.

Why it happened

A configuration change made to the message broker on 6 August had an unintended side effect. Roger automatically removes queues that have been unused for seven days, and normally these removals are spread evenly across each day. The change inadvertently reset all of those seven-day timers to the same moment.

Seven days later - at 11:18 on 13 August, during our busiest period - over 2000 queues expired simultaneously instead of gradually. Clearing them all at once overwhelmed the broker's internal coordination, which then cascaded:

The broker slowed sharply under the bulk cleanup.
The cleanup triggered a large internal reorganisation across the remaining queues, which is what pushed the system past its limits.
One broker node dropped out; clients reconnecting to the remaining nodes multiplied the load on them, and they failed in turn.
Restoring the cluster took two attempts, extending the outage.
The underlying trigger was a single change with a delayed effect that was not visible at the time it was made.

What we are doing about it

The broker cluster was restored and has been running normally since 13 August.
Broker capacity limits were increased the same day.

Preventing recurrence

Queue expiry will be staggered so that a single change can never re-synchronise a large group of timers again, and such operations will not be performed during peak hours.
We are reviewing the broker setting that caused the bulk cleanup to escalate into a cluster-wide reorganisation.

Reducing recovery time

Writing and rehearsing a documented rapid-recovery procedure, so restoration follows a known path.
Retuning health checks that were restarting broker nodes while they were legitimately recovering, which prolonged the outage.

Improving detection and resilience

Adding monitoring for bulk queue-removal events, which had no alerting at all.
Changing how Roger interacts with the broker so that broker slowness cannot cause database contention - the mechanism behind the failed print jobs.
Adding limits to client reconnection behaviour so that the loss of one broker node cannot overload the remaining ones.

Status page

We have identified that our external monitoring did not correctly update our public status page during this incident. This is being corrected so that future incidents are reflected publicly without delay.

Apology & Commitment to Improvement

We deeply regret the impact this incident had on our users and teams. Ensuring resilient and highly available messaging infrastructure remains a top priority.

We are strengthening monitoring, governance, and recovery automation to further reduce the likelihood and impact of similar events in the future.

Owner: DevOps Team