When the Vault Itself Fails
The system holding every administrative credential is a single point of failure by design. Planning for its loss, and testing the plan.
Procedure
Concentrating credentials creates a dependency that did not previously exist. That is an acceptable trade, provided the failure case is planned.
The failure modes
Unavailable but intact: an outage, a network partition, a certificate expiry. Administrators cannot work; nothing is lost.
Corrupted: the data is damaged and restoration is required.
Lost: the vault and its backups are gone.
Compromised: an attacker has the contents, which is the worst case and requires rotating everything.
Each needs a different response, and organisations typically plan for none of them.
Unavailable
The most likely case and the one that produces bad workarounds.
High availability at the vault, with a tested failover.
Break-glass accounts on the critical systems, so work can continue.
A documented degraded mode stating what administrators should do, so that the answer is not invented under pressure.
A time limit on degraded operation, after which the incident escalates.
Corrupted or lost
Backups, encrypted, with keys stored separately and with a documented recovery order.
Backups tested by restoring, to a separate environment, on a schedule. A backup never restored is a hypothesis.
A recovery key or master credential stored offline with split custody.
A documented rebuild procedure that does not require access to the vault.
Where recovery fails, the fallback is rotating every credential in the estate, which is why the emergency rotation drill matters.
Compromised
Assume every credential the vault held is known.
Rotate everything, in tier order, starting with tier zero.
This is the drill, and an organisation that has never rehearsed it will take weeks.
Preserve evidence before rebuilding, which conflicts with speed and needs a decision made in advance.
Assume the audit log is untrustworthy if the attacker held vault administration, which is why logs should be shipped to a destination the vault administrators cannot alter.
What to test, and how often
Failover: quarterly.
Restore from backup: annually, into a separate environment, verifying that credentials actually work.
Break-glass retrieval and use: annually.
Emergency rotation of a tier one account: annually, timed.
Full degraded-mode operation: annually, as a tabletop at minimum.
The dependency check
List what the vault depends on: directory, network, certificates, database, storage, the identity provider.
For each, ask whether the vault is needed to fix it.
Circular dependencies are the failure. A vault that authenticates against a directory whose administrative credentials are in the vault cannot recover from a directory failure.
Break the circle deliberately, with local accounts, offline credentials or an independent authentication path for the recovery case.
Documenting it
One page, printed, held physically, because the digital copy may be inaccessible.
Who to call, what to retrieve, in what order, and what authorisation is needed.
Reviewed annually and after any architecture change.
Held in more than one location, including outside the primary site.
The dependency graph
Draw it once, and look for cycles.
List what the vault depends on: directory, network, certificates, database, storage, identity provider, DNS.
For each, ask whether the vault is needed to fix it.
Any yes is a cycle, and it will bite during exactly the incident where it matters.
Break each one deliberately: local accounts, offline credentials, or an independent authentication path reserved for recovery.
Redraw after any architecture change, because new dependencies arrive with new components and nobody checks.
The printed page
The recovery documentation must be readable when the systems are not.
One page, printed, held physically, in more than one location including outside the primary site.
Who to call, with numbers that do not depend on the corporate directory.
What to retrieve, from where, with whose authorisation.
The recovery order.
Reviewed annually and after any change to the architecture or the people named.
Tested by having someone unfamiliar follow it, which reveals the assumed knowledge that the author did not write down.