backup_restore_testing v1.0.0

name: backup-restore-testing description: Build a real backup and restore practice: 3-2-1 backup design, restore drills that prove recovery works, integrity verification, and RTO/RPO targets in plain language. Use when setting up backups, auditing an existing backup setup, or preparing for a restore test.

Backup and Restore Testing

Purpose

A backup you have never restored is a hope with a timestamp. This skill turns "we have backups" into a tested recovery capability: how to structure backups, how to verify they are intact, and how to run drills that prove you can get back online inside your RTO with no more loss than your RPO allows.

Workflow

1. Define RTO and RPO first, in plain language

• RPO (Recovery Point Objective): the maximum data loss you can accept, measured in time. An RPO of one hour means you must have a recoverable copy no older than one hour before the incident. RPO decides backup frequency: itadon backup-strategy guidance and NIST SP 800-34 both frame it this way.
• RTO (Recovery Time Objective): the maximum acceptable time to bring the system back after the disruption starts. RTO decides your recovery architecture (hot standby vs cold restore).
• Set one pair per service tier. A database that processes orders deserves tighter numbers than a static blog. Write them down and get agreement before touching any tooling.

2. Structure the backups with 3-2-1, hardened

• 3 copies: the production data plus two backups.
• 2 different media: for example local disk plus cloud object storage, or NAS plus tape. This protects against a whole class of failure hitting one kind of media at once.
• 1 copy offsite: geographically separated, so fire, theft, or a regional outage cannot take everything.
• Origin: the 3-2-1 rule was popularized by photographer Peter Krogh in his 2005 book "The DAM Book: Digital Asset Management for Photographers"; CISA referenced the approach in a 2012 US-CERT publication.
• For ransomware risk, add the extensions the industry settled on:
• +1: one immutable or air-gapped copy (IDC's 2021 3-2-1-1 recommendation). Immutability means write-once: attackers who own the network still cannot alter or delete that copy.
• +0: zero recovery errors. Every copy gets verified as restorable, not just written. The QNAP solution page documents this as the 3-2-1-1-0 extension.
• What counts as offsite: a second cloud region or a separate provider, not a different folder on the same server. A realtime sync folder (Drive, Dropbox) is not a backup: ransomware and accidental deletes propagate through sync.

3. Make backups application-consistent

• Capture the whole recovery unit: database dump or snapshot, object storage contents, config files, secrets/key material recovery instructions, and the app version the data belongs to.
• For databases, enable point-in-time recovery: WAL archiving (PostgreSQL), binary logs (MySQL), or oplog (MongoDB), with an unbroken log chain from the last full backup.
• Never store the backup encryption keys or credentials next to the backups themselves. Keep an offline list of which variables exist, not their values.

4. Verify integrity at write time and before every restore

• Record a SHA-256 manifest (checksum per file) the moment the backup is created. bin/verify_backup.py does this: verify_backup.py create /backup/dir > manifest.json then verify_backup.py check /backup/dir manifest.json.
• NIST SP-1339 calls for storing file hashes of backup content and verifying integrity before restoration begins, using cryptographic hashing where possible.
• Checksums catch corruption, but they do not prove restorability. Only a restore test does.

5. Run the restore drill

• Pick the cadence by RTO tier: critical systems drill quarterly at minimum; everything else at least twice a year.
• Restore into an isolated environment, never by overwriting production. The drill is a rehearsal, not a gamble.
• Time the whole drill with a stopwatch: from "we decide to restore" to "service is back and verified". Compare against the RTO. If the measured time exceeds the target, the RTO is a wish, not a target.
• During the drill, run the application smoke tests, not just a file listing. A database that mounts but cannot serve queries is not recovered.
• Fill in references/drill-runbook.md before the drill: scope, steps, who owns each step, success criteria. Fill it out after: what actually happened, timings, friction found.

6. Keep a written backup policy

• Use assets/backup-policy-template.md. It forces the decisions that matter: RTO/RPO per service, what is backed up, schedule, retention, where copies live, encryption/key ownership, restore destination, last successful drill date.
• Review the policy annually and after any real incident or drill that exposed a gap.

Output Contract

Every engagement with this skill produces:

1. RTO/RPO statement for each in-scope service (one sentence each, agreed, not assumed).
2. Backup topology showing the 3-2-1 layout: which copies exist, on what media, where.
3. Integrity evidence: a current SHA-256 manifest or equivalent verification record.
4. Drill record: filled-in runbook from the latest drill with measured restore time vs RTO.
5. Gap list: anything that failed or is unverified, with an owner and a date to fix it.

Operating Rules

1. A "backup succeeded" message is not verification. Verification is a checksum match plus a successful test restore.
2. Never run a first-time restore against production. First restores happen in isolation.
3. Do not restore from an unverified backup point. Pick the newest point within the RPO that has passing integrity evidence.
4. After an incident that involved an attacker, assume backups may be contaminated. Verify integrity and check logs for backup access before restoring.
5. Drills use scrubbed or synthetic data when the backup holds production data; do not rehearse on live customer data in a shared environment.
6. If the drill fails, that is a success of the drill. Record it, fix the cause, and drill again.
7. Every factual claim in written output must be verified or framed as an assumption. No invented statistics about failure rates.
8. Sync is not backup. Say so explicitly wherever a client confuses the two.