Disaster Recovery Drills: Choosing Scenarios Worth Testing

Disaster Recovery Drills: Choosing Scenarios Worth Testing

Every organisation with a voice platform has a continuity plan, and most have tested it in a way that proves very little. The test consisted of restoring the configuration file to a second system and confirming the file opened. That confirms the backup is readable. It does not confirm that calls reach the surviving site, that the surviving site's trunk still has the numbers, that the surviving site's queue configuration matches, that staff there know how to answer a call that has moved, or that anyone noticed the calls were not arriving. A recovery test is only meaningful when its failure criteria would surprise you. Otherwise it is a scheduled reassurance exercise.

Test the Business, Not Just the Backup

The distinction between a technical restore test and a recovery test is the distinction between a system working and a business continuing. A business recovery test has an objective expressed in business terms: calls reach a named destination within a stated time, calls are answered by staff who know what has happened, and no more than a stated proportion of attempts fail. Those criteria are harder to write than technical ones and considerably more useful.

Write the success criteria before the exercise, and have them reviewed by someone who will be accountable for the outcome in a real event. Criteria written by the person running the drill tend to be written so that the drill passes.

Scenario Portfolio

A single scenario proves very little. Different scenarios exercise different parts of the plan, and some scenarios reveal failures that others hide entirely. A sensible annual cycle covers several distinct failure classes rather than repeating the same one at greater frequency.

Running the Drill Safely

A drill that disrupts live calling has its own cost, and a drill conducted without safeguards can become the incident it was meant to test. Every exercise should therefore begin with safety preconditions. Confirm that a working path exists before simulating its loss. Identify in advance which actions are permitted during the exercise and which are forbidden, because in practice the most damaging drill outcome is an engineer improvising a recovery step during a simulated failure.

• Announce the exercise window in advance to the teams who will notice it — Avoids unnecessary escalations and lets staff prepare
• Confirm a genuinely working fallback path before simulating the loss of the primary one — Prevents the exercise from becoming an unplanned outage
• Nominate a drill controller who owns the scenario and a separate observer who only records — Prevents the person testing from deciding what counts as a pass
• State forbidden actions explicitly, including untested configuration changes — Reduces the risk of a simulated fault becoming a real one
• Pre-agree abort criteria and the time at which the exercise stops — Keeps the exercise inside a window the business can absorb
• Prepare a communication template for participants who are not in the drill loop — Stops the drill leaking as an apparent outage to real users
• Capture timings throughout, not just outcomes — Produces the evidence that makes the exercise useful later

Recording the Exercise Honestly

The report from a drill is the only durable output, and its value depends entirely on candour. A report that describes the plan as working as intended is not a useful artefact. The report should describe what was expected, what actually happened, where the elapsed time went, which assumptions turned out to be wrong, and which steps could only be completed by someone who already knew the system well.

Two categories of finding deserve particular attention. The first is elapsed time: how long between deciding to recover and the first call being answered. That number is the one that matters in a real event, and it is usually worse than the plan assumes. The second is undocumented dependency: an undocumented external system, an undocumented person, or an undocumented licence that turned out to be required.

Testing Degraded Operation

Most plans describe recovery to full service and say little about what happens in between. In practice, a voice platform is often usable in a reduced form well before it is fully restored, and defining that reduced form in advance saves valuable time during the incident. A reduced form might mean inbound calls to one location only, no recording, no voicemail, or a single shared answer point rather than the full ring group structure.

Defined degraded modes have two benefits. They give responders a defensible choice to make under pressure instead of improvising, and they give the business a realistic expectation about what will happen in the first hour. A plan that only describes the end state implicitly tells everyone that nothing works until everything works, which is rarely true and never helpful.

Frequency and Triggers

Annual drills are a common recommendation and a weak one. Frequency should follow how much can change: a site that reconfigures frequently needs more frequent testing than one that does not, and an organisation with high staff turnover needs more frequent testing than one with a stable team. In addition to the calendar, define event-based triggers that require an unplanned test regardless of schedule.

Redstone Approach

Redstone can support the planning and execution of voice continuity exercises. Our engineers can help you define realistic scenarios for your environment, review and improve existing continuity documentation, assist with failover and restore testing on Redstone REX IP-PBX and RGW gateway deployments, and provide technical observation during an exercise so that findings are recorded accurately. We can also advise on degraded-mode configurations that suit your call patterns. Continuity planning is a business process as much as a technical one, and we scope our involvement to the parts where technical input is genuinely useful.