Availability of the Control Path

Introducing a broker between administrators and systems creates a dependency with an availability requirement higher than almost anything else you run.

What an outage means

Administrators cannot work, including on the incident that may have caused the outage.

Change freezes in practice, whether or not one is declared.

Workarounds are created under pressure, by competent people, and they persist.

Confidence in the programme drops sharply and does not recover quickly.

One serious outage during a major incident can end a deployment, which is a political outcome as much as a technical one.

The availability target

Higher than the systems it protects, because it is a dependency of all of them.

Including out of hours, which is when incidents happen.

Including during a site failure, which is when administrative access matters most.

Stated explicitly and designed for, rather than inherited from whatever the platform defaults to.

Designing for it

Clustering or active-active deployment, across failure domains.

Tested failover, not assumed. Quarterly at minimum.

Independent dependencies. The broker should not depend on a system whose recovery requires the broker.

Local caching where the product supports it, so a network partition does not remove access entirely.

Monitored as a tier zero system, with alerting to people who can act.

The circular dependency

The design error that causes the worst outcomes.

The vault authenticates against the directory.

The directory's administrative credentials are in the vault.

The directory fails.

Now nobody can obtain the credentials to fix it.

Break the circle deliberately: local break-glass accounts on the directory, offline credential storage, or an independent authentication path for recovery.

Draw the dependency graph and look for cycles. Most organisations have at least one and have never checked.

Degraded mode

Define what administrators do when the broker is unavailable.

Break-glass accounts on critical systems, documented and tested.

A stated time limit on degraded operation before escalation.

Recording still required where possible — target-side logs shipped centrally continue to work when the broker does not.

Rotation afterwards of everything used.

A review of what was done during the period.

Without a defined degraded mode, the mode is whatever the person on call invents, and it will not be recorded.

Change management for the broker

Treat changes to it as high risk, because they are.

Test in a non-production environment that actually resembles production.

Change windows outside peak administrative activity.

A rollback plan tested rather than assumed.

Never change it during a change freeze for something else, which concentrates risk.

The measure

Availability, measured from the administrator's perspective rather than from the component's uptime.

Time to failover, measured in tests.

Number of break-glass uses attributable to broker unavailability, which is the honest measure of whether it is reliable enough.

Administrator-reported friction, which predicts workarounds better than any uptime figure.

Testing failover properly

An untested failover is a hypothesis, and the failure mode is discovering that during an incident.

Fail over deliberately, in production, on a schedule.

Have an administrator perform real work during the failed-over state.

Measure how long the transition took and what was disrupted.

Test the failback, which is frequently less reliable than the failover.

Test with a dependency removed — the directory, the network path — rather than only with a clean component failure.

Quarterly, and after any change to the broker or its dependencies.

Measuring friction, not uptime

Component uptime and administrator experience diverge, and only the second predicts bypassing.

Time from intent to session established, measured end to end.

Failed attempts and their causes.

Break-glass uses attributable to broker problems.

Support tickets mentioning the broker.

Ask administrators directly, quarterly, whether the path is workable.

A system with high uptime and high friction will be routed around, and the uptime figure will show nothing wrong right up until the coverage measurement does.