A customer admin bulk-deletes 12,000 records. A mis-scoped backfill overwrites one tenant's pricing table. An integration writes garbage for six hours. You have backups, so recovery should be routine. In a shared multi-tenant database it usually is not, because the tools that restore data work on a whole database, and the damage belongs to one tenant.
Per-tenant restore means returning one tenant's data to a known earlier state while every other tenant keeps working on current data. Native point-in-time recovery does not do this for you. This article proposes a design that does: recover into an isolated scratch instance, compute a reviewable plan, apply it with conditional writes, and verify the result before anyone calls it done.
The design is a proposal. It is not a product or a standard, and it is not a method Developers.dev claims to run in any named client system. It is written for a tech lead or DevOps lead who needs a defensible answer before the first incident rather than during it. Where the design touches personal data, deletion obligations, or side effects such as emails and webhooks, it says who must own the decision.
- Primary reader: Tech Lead or DevOps Lead running a shared-schema or shared-database SaaS.
- Content type: medium authority guide with a decision table and go/no-go controls.
- Buyer stage: failure recovery, meaning you are preparing for or living through a partial data-loss incident.
Key Takeaways
- Native recovery is cluster-shaped, tenant damage is row-shaped. PostgreSQL's continuous archiving supports restoring an entire cluster, not a subset. Amazon RDS point-in-time restore creates a new instance and leaves the source untouched. Both give you a historical copy, not a surgical undo.
- Recover into scratch, then apply a plan. Restore to an isolated instance, extract one tenant's slice, diff it against live data, and freeze the diff as a hashed, approved plan.
- Never overwrite blindly. Use target-side conditional writes (a row-version predicate inside the UPDATE). Rows that changed after the incident become conflicts for a human, not silent overwrites.
- Ambiguous results are not success. If a batch outcome is unknown, read the authoritative step marker before retrying. Mark the operation completed only after verification.
- Restore can resurrect erased data and fire side effects. Re-apply your erasure ledger to the plan, suppress outbound events, and re-sync derived stores. Identifiers in the plan are themselves sensitive.
- Use all-required go/no-go controls. A restore design is ready only when every control passes. A summed score cannot certify it.
Why One Tenant's Data Loss Is Not a Database Restore Problem
Key takeaway: Backups protect the database. Tenants are a logical slice of it. The mismatch between those two granularities is the entire problem.
In a shared-schema design, every table carries a tenant_id, and thousands of tenants write to the same physical tables. That is efficient, and our guide to multi-tenant database isolation patterns and scaling trade-offs covers why teams choose it. The cost shows up on the worst day. The backup was taken of the whole database, so the only restore the platform offers is of the whole database.
The vendor documentation says so directly. The PostgreSQL documentation on continuous archiving and point-in-time recovery states that this method can only support restoration of an entire database cluster, not a subset. The Amazon RDS point-in-time restore guide describes restoring to a specified time by creating a new DB instance without modifying the source instance. Managed Aurora uses a different API for the same idea. Check your own engine's documentation.
Put those facts together. Rolling the production cluster back to 14:04 would erase every other tenant's legitimate writes since 14:04. Restoring to a new instance is safe, but it hands you a second database, not a repaired tenant. Everything between "I have a historical copy" and "this tenant's data is correct in production" is application engineering that your database vendor does not provide.
What native recovery gives you versus what tenant restore needs
| Concern | Native point-in-time recovery | Per-tenant restore needs |
|---|---|---|
| Granularity | Whole cluster or instance | One tenant, often a subset of tables and a time window |
| Target | A recovered copy (new instance or recovered cluster) | The live production database, while it takes traffic |
| Conflicts with newer writes | Not applicable, the copy is a snapshot in time | Explicit conflict detection and resolution |
| Deleted or erased data | Restores whatever existed at the recovery point | Must exclude data that was lawfully erased afterward |
| Side effects | None, nothing is replayed to the outside world | Suppress or deliberately control emails, webhooks, billing, and search indexing |
| Evidence | Restore logs from the platform | Approved plan, per-row outcomes, verification report |
One more detail from the PostgreSQL documentation matters operationally. Recovery targets can be a timestamp, a named restore point, an LSN, or a transaction ID. Per the PostgreSQL write-ahead log configuration reference, if a recovery target is configured but archive recovery ends before the target is reached, the server shuts down with a fatal error. Your recovery job must treat that as a terminal failure. It must never fall back to "latest available state" and present that as the requested point.
Also note that a mistake and its commit often share the same second. Choose a recovery point slightly before the incident, or use a named restore point created before the risky operation. A cheap habit that pays for itself: have bulk imports, backfills, and migrations call pg_create_restore_point() (or your platform's equivalent marker) before they start. Our guide to database replication for high availability and disaster recovery covers the cluster-level side of recovery. This article covers the tenant-level gap that remains.
Options Compared: Four Ways to Recover One Tenant
Key takeaway: No option is free. Full rollback is fast to start and destroys other tenants' data. Selective restore is the safest general answer but needs the most engineering. Application-level history is the cheapest, yet it only covers what you thought to record.
Before you build anything, decide which of four recovery shapes you are actually choosing between. This table is the decision artifact for the architecture choice. The qualitative ratings assume a shared-schema PostgreSQL-style system and should be re-scored against your own data model.
| Option | Fits when | Cost | Risk | Speed to recover | Scales to many tenants | Hidden failure mode |
|---|---|---|---|---|---|---|
| A. Full cluster rollback | Platform-wide corruption, single-tenant database | Low build cost, high business cost | Very high for shared databases | Fast to start, slow to reconcile | No | Erases every other tenant's newer writes; usually forces a platform-wide outage and apology |
| B. Scratch recovery plus selective logical restore (proposed) | Shared schema, incident confined to one tenant | Medium to high build cost, low per-incident cost once built | Medium, controlled by plan review and conditional writes | Hours, dominated by recovery and review | Yes, same pipeline for any tenant | Dependency graph gaps leave orphaned or inconsistent rows; derived stores drift |
| C. Application-level history (soft delete, audit or event log, temporal tables) | Common undo cases you can predict | Ongoing storage and complexity cost | Low for covered cases | Minutes | Yes | Only covers modeled tables and actions; history itself may violate deletion obligations if retained forever |
| D. Database or schema per tenant | Few, high-value tenants, strict isolation contracts | High operational cost | Low blast radius | Fast per tenant | Strains at large tenant counts | Migration and backup operations multiply by tenant count; fleet-wide schema changes get slower |
Most mature systems end up combining C and B. History tables and soft deletes handle the frequent, predictable "oops" cases in minutes. The scratch-restore pipeline is the break-glass path for the unpredictable ones. Option B is the focus here because it is the one teams most often improvise at 2 a.m.
How teams usually approach it, and why that approach is fragile
The improvised version looks like this. Someone restores a backup to a spare instance, runs pg_dump filtered by tenant, hand-edits the output, and replays it with INSERT ... ON CONFLICT DO UPDATE. It works in a demo. It fails in production for structural reasons rather than human ones: the script has no concept of what changed since the incident, no record of what it did, no way to resume, and no owner for the personal data sitting on the spare instance. The rest of this article replaces that script with a pipeline whose every step has a defined exit condition.
A Proposed Design: The Tenant Restore Pipeline
Key takeaway: Treat tenant restore as a durable, multi-stage operation with an approval gate between "what we think should change" and "what we change."
The pipeline below is a proposed design. Names, stages, and states are illustrative and should be adapted to your schema. The principle behind it is that a restore is a consequential mutation of production data. It deserves the same rigor as a payment: durable identity, an approved immutable payload, atomic claiming, conditional writes, and verified completion.
The eight stages
| # | Stage | What happens | Exit condition | If it fails |
|---|---|---|---|---|
| 1 | Declare | Create an operation record: tenant, incident window, table scope, requested recovery point, requester | Operation ID exists, scoped to that tenant | Reject malformed or cross-tenant requests |
| 2 | Recover to scratch | Restore to an isolated instance at the recovery point; never the production instance | Instance reaches the requested point and passes a read check | Terminal failure if the point is outside retention or the target is unreachable. Do not substitute a different point silently |
| 3 | Extract the slice | Read the tenant's rows in foreign-key dependency order, using a tenant-scoped read role | Row counts and checksums recorded per table | Re-run extraction, it is read-only |
| 4 | Filter | Remove data subjects in the erasure ledger and anything under rules your data owner flags | Filter report reviewed by the privacy owner | Block the operation until the owner decides |
| 5 | Diff and plan | Compare the slice with live data. Emit steps (insert, update, delete) each carrying the expected live row version | Plan canonicalized and hashed | Regenerate. A new plan means a new hash and new approval |
| 6 | Approve | Human reviewer approves the exact plan hash with the diff summary | Approval bound to the plan hash | Expire stale approvals |
| 7 | Apply | Claim the operation, apply steps in batches with conditional writes, record step outcomes atomically | Every step has a recorded outcome | Resume safely, surface conflicts, never overwrite |
| 8 | Verify and close | Compare post-apply state with the plan, re-sync derived stores, then close | Verification report matches expectations | Stay open as "outcome unknown" or "completed with conflicts" |
An illustrative scenario
The numbers below are hypothetical and exist only to show how the stages interact. A tenant admin deletes 12,000 invoice rows at 14:05 and notices at 16:20. In the meantime the tenant has legitimately created 300 new invoices and edited 40 old ones, and 25 of the deleted invoices belong to a customer whose erasure request was completed at 15:00.
- Recovery point: 14:04, created as a new instance. Production keeps serving every tenant.
- Extraction finds 12,000 deleted rows plus the 40 rows the tenant later edited.
- The filter removes the 25 erased rows and records that in the report. The privacy owner confirms.
- The diff produces about 11,975 inserts and 40 candidate updates. The 40 updates are flagged because live versions are newer than the snapshot. They default to keep live unless a human says otherwise.
- The reviewer approves the plan hash. Apply runs, verification passes, search and cache are re-synced, and the operation closes.
The 300 new invoices were never touched. That is the point of building a plan instead of replaying a dump. The design preserves post-incident work by construction rather than by luck.
Execution Safety: Operation Identity, Conditional Writes, and Unknown Outcomes
Key takeaway: The apply stage is where restores go wrong. Make it resumable, tenant-scoped, and honest about what it does not know.
Durable identity bound to an approved payload
Give every restore a durable operation identity scoped to the tenant: (tenant_id, operation_id). Store the SHA-256 of the canonicalized plan when it is approved, and make that hash immutable (a trigger or write-once column constraint). If someone re-submits the same operation ID with a different plan, because they tweaked a filter to "just include one more table," reject it. A changed payload is a new operation with a new approval, not a retry. A legitimately later restore for the same tenant, say a second incident next month, gets a fresh operation ID and its own plan. This is the same tenant-scoped idempotency principle covered in our production-ready idempotency framework.
Atomic claiming and fencing
Two workers (or an impatient engineer plus a worker) must never apply the same operation. Claim work with a single conditional statement, and bump a fencing epoch on every claim so a stale worker can detect that it has lost ownership. The mechanics match those in our write-up on worker leases and fencing tokens for tenant-safe job queues. The following schema and statements are illustrative PostgreSQL-style SQL, not an official API contract.
CREATE TABLE tenant_restore_operation ( tenant_id uuid NOT NULL, operation_id uuid NOT NULL, plan_sha256 bytea NOT NULL, -- immutable once approved state text NOT NULL, -- planned, approved, applying, verifying, -- completed, completed_with_conflicts, -- failed_terminal, outcome_unknown fencing_epoch bigint NOT NULL DEFAULT 0, claimed_by text, lease_expires_at timestamptz, PRIMARY KEY (tenant_id, operation_id) ); CREATE TABLE tenant_restore_step ( tenant_id uuid NOT NULL, operation_id uuid NOT NULL, step_key text NOT NULL, -- table:primary_key:action outcome text NOT NULL, -- applied, conflict, skipped_erased PRIMARY KEY (tenant_id, operation_id, step_key) ); -- Atomic claim: succeeds only for the approved hash, only if unclaimed or lease expired UPDATE tenant_restore_operation SET claimed_by = $worker, fencing_epoch = fencing_epoch + 1, lease_expires_at = now() + interval '5 minutes', state = 'applying' WHERE tenant_id = $tenant AND operation_id = $op AND plan_sha256 = $approved_hash AND state IN ('approved', 'applying') AND (lease_expires_at IS NULL OR lease_expires_at < now()) RETURNING fencing_epoch;If the claim returns zero rows, the worker does nothing. It does not infer why. The cause could be a hash mismatch, a live lease, or a closed operation, and each needs a different human response.
Target-side conditional mutation, not read-compare-then-write
The most common mistake is to read the live row, compare it in application code, then write. Another transaction can change the row between the compare and the write. That is not compare-and-swap. The condition has to live inside the write itself, evaluated by the database.
-- One step, one transaction. Illustrative. BEGIN; -- Confirm this worker still owns the operation (fencing check, inside the transaction) SELECT 1 FROM tenant_restore_operation WHERE tenant_id = $tenant AND operation_id = $op AND fencing_epoch = $epoch AND lease_expires_at > now() FOR SHARE; -- Conditional write: only changes the row if it is still at the version the plan saw UPDATE invoice SET status = $status, amount_cents = $amount, row_version = row_version + 1 WHERE tenant_id = $tenant AND id = $id AND row_version = $expected_version; -- Application checks rowcount: 1 means applied, 0 means conflict (live row changed) INSERT INTO tenant_restore_step (tenant_id, operation_id, step_key, outcome) VALUES ($tenant, $op, $step_key, $outcome) ON CONFLICT DO NOTHING; COMMIT;Three details matter. First, the step marker is written in the same transaction as the mutation, so the marker is an authoritative record of whether the mutation committed. Second, every statement carries tenant_id, which is defense in depth on top of row-level security. Third, tables without a row-version column cannot use this pattern. Add one through an expand-contract change (see zero-downtime schema changes and backfills in multi-tenant SaaS). Otherwise, put the tenant behind a write fence for the duration of apply and disclose that serialization limit in the runbook. A fence trades availability for tenant-level safety, and the tenant must be told.
Unknown results, terminal failures, and verified completion
A connection can drop during COMMIT. The worker then does not know whether the step applied. The rule is simple: an unknown result means completion could not be confirmed. It is neither success nor verified failure. Do not blindly retry. Reconcile first.
def apply_step(step): # Illustrative pseudo-code, not an official client API marker = read_step_marker(step.tenant_id, step.operation_id, step.key) if marker: return marker.outcome # already decided, whatever it was try: return run_step_transaction(step) # conditional write + marker, one transaction except CommitOutcomeUnknown: # Bound the old session first (statement and idle-in-transaction timeouts), # then read the marker from a fresh session. The marker commits atomically # with the mutation, so it is the authoritative answer. wait_until_old_session_is_gone(step) marker = read_step_marker(step.tenant_id, step.operation_id, step.key) if marker: return marker.outcome mark_operation_state(step, 'outcome_unknown') raise NeedsReconciliation(step.key)Completion needs its own proof. Mark the operation completed only when (a) every plan step has a recorded outcome, (b) a verification query shows live rows match the plan's expectations for every applied step, and (c) referential integrity checks pass for the tenant. If any step is a conflict, the final state is completed_with_conflicts, which is deliberately not the same as completed. A failed_terminal state is for conditions retrying cannot fix, such as a recovery point outside retention or a rejected plan hash. These principles are the restore-specific version of the pattern described in reconciling before retrying ambiguous timeouts.
Privacy, Side Effects, and Derived Data After a Restore
Key takeaway: A restore reintroduces the past. Some of that past was deleted on purpose, and some of it will wake up systems that should stay quiet.
Erasure and the scratch instance
A recovered backup contains whatever existed at the recovery point. If a data subject's erasure was completed after that point, a naive restore resurrects them. The plan stage must therefore consult an erasure ledger: a durable, append-only record of completed erasures and holds that survives restores. Our guide to verified erasure workflows in distributed systems describes how to build one. Steps that touch a subject in the ledger become skipped_erased and are reported, not applied.
Three ownership questions need named answers before an incident, not during one:
- Who owns the scratch instance's lifecycle? It holds a full historical copy of production data. Define who may create it, who may read it, how long it lives, and who confirms it was destroyed.
- Who owns the decision when the ledger and the tenant disagree? The customer may want the data back. The law or contract may say otherwise. That is a decision for the privacy or legal owner, not the on-call engineer.
- How do backups themselves relate to erasure obligations? Obligations for backup copies differ by jurisdiction and contract. This article cannot settle that for your organization. Take it to your counsel and your data protection lead.
Identifiers are sensitive too
Plans, diffs, and logs are full of primary keys, email addresses, and external IDs. Those identifiers can identify a person on their own. Treat the plan as personal data: encrypt it at rest, restrict access to the reviewer and the worker role, set a retention limit, and keep row contents out of ordinary application logs. Log hashes and counts. Show reviewers the diff in a controlled tool. This also keeps the audit trail usable without turning it into a second copy of the production dataset.
Restrict the restore role
Run extraction and apply under a role whose reach is limited to the target tenant. PostgreSQL's row security policies can enforce this, but check the details for your version. Table owners and certain privileged roles can bypass policies unless you take explicit steps. Keep the explicit tenant_id predicate in every statement as a second layer.
Side effects and derived stores
Restored rows were written by the database, not by your application, so outbound behavior will not fire unless you wire it in. That sounds safe. The opposite failure is real too: an application-level apply path can trigger outbox events, webhooks, and notifications for every row it touches. Decide this deliberately, per system:
| System | Question to answer before apply | Typical safe default |
|---|---|---|
| Outbox or event stream | Should restored rows emit events? | Suppress normal events. Emit one explicit "tenant restored" event with the operation ID |
| Outbound webhooks | Will tenants receive thousands of "created" calls? | Suppress, then tell the tenant what changed |
| Email and notifications | Could a restored status trigger reminders or dunning? | Pause the tenant's notification jobs during apply |
| Billing and metering | Do restored rows double-count usage? | Exclude restored rows from metering or reconcile after |
| Search index and cache | Are they stale or missing the rows? | Targeted re-index and cache invalidation for the tenant |
| Warehouse and analytics | Do downstream models need a correction window? | Backfill the tenant's affected range, annotate the incident |
| Object storage references | Do restored rows point to files that were since deleted? | Verify existence, flag broken references instead of hiding them |
If you operate bulk write paths already, our resumable, tenant-safe bulk import blueprint shares several of these concerns, including write suppression and resumability.
Why This Fails in the Real World: Common Failure Patterns
Key takeaway: Capable teams fail here because restores are rare, owned by no one, and rehearsed never. These are system gaps, not individual mistakes.
The scenarios below are composites of failure shapes common to this class of system. They are illustrative and do not describe a specific client.
Failure pattern 1: The restore overwrites a day of legitimate work
The tenant noticed the problem late and kept working in the meantime. The restore script uses upsert semantics, so it cheerfully replaces newer edits with older data. Nobody notices until the tenant reports that yesterday's work vanished, which is a second incident created by the fix for the first. The governance gap is that the process has no conflict model. It assumes the tenant's data was frozen at the incident time. Conditional writes and a diff that defaults to keep live close this gap.
Failure pattern 2: A timeout, a retry, and a quietly changed script
A batch times out. The on-call engineer, who is tired and reasonable, reruns the script with a slightly widened filter to catch what "must have been missed." The first run had actually committed. Now some rows are applied twice under two different intents, and there is no record of which run did what. The process gap is that identity is attached to a ticket, not to an immutable plan. Binding the operation to a plan hash makes the widened rerun a new, separately approved operation. Step markers make the timeout reconcilable.
Failure pattern 3: Erased data comes back
A recovery point from before an erasure is used to rebuild the tenant slice, and nobody filters against completed erasures. The data subject returns to production. Intelligent teams miss this because deletion and recovery are owned by different people who never review each other's runbooks. The erasure ledger must be a hard input to the plan stage, with the privacy owner as a named approver.
Failure pattern 4: The restore wakes the outside world
The apply path goes through the normal application services to keep validations. Every restored invoice fires a webhook and a reminder email. The tenant's customers receive a burst of messages about invoices they already paid. The gap is that restore was designed as a data operation, but the application treats every write as a business event. Side-effect suppression has to be an explicit mode, tested before it is needed.
Failure pattern 5: The backup is fine, the window is not
Detection takes three days, but the recovery window covers two. Platform documentation is explicit that point-in-time restore works only within the configured retention period (see the Amazon RDS guide above). The backup technology worked exactly as designed. The mismatch was between how fast tenants discover data problems and how long you keep recoverable history. Measure detection latency, not just backup success, and set retention with that distribution in mind.
Pattern summary: signals and fixes
| Pattern | Early warning signal | Structural fix |
|---|---|---|
| Overwrites newer work | Restore scripts use unconditional upserts | Row-version conditional writes, conflict state, diff defaults to keep live |
| Retry with changed scope | Runbook says "rerun with the corrected filter" | Plan hash binds approval, changed plan means new operation |
| Resurrected erased data | No erasure ledger input to restore tooling | Ledger-driven filter, privacy owner approval |
| Unwanted side effects | Restore runs through normal write paths | Explicit restore mode with suppressed events |
| Recovery window expired | Retention set by cost, not by detection latency | Track time-to-detect, align retention, add named restore points before risky jobs |
A Lower-Risk Rollout: Go/No-Go Controls, Persona Guidance, and a Recency Note
Key takeaway: Build read-only first, rehearse before you trust it, and treat readiness as a set of controls that must all pass.
Mandatory go/no-go controls
These controls are all required. Do not turn them into a score. A design that passes nine of ten is not 90 percent ready. It is not ready. A passing result also does not certify your system for production or for any compliance regime. It records that your team has evidence for each control.
- Tenant-scoped operation identity: every restore has a durable
(tenant_id, operation_id), and no operation can reference another tenant's data. - Immutable approved plan: the plan hash is stored at approval, cannot be changed, and key reuse with a different hash is rejected.
- Atomic claim with fencing: concurrent workers cannot both apply the same operation, and a stale worker's writes are refused.
- Target-side conditional writes: every update or delete carries its condition inside the statement. Any table without a version mechanism is handled by a disclosed write fence.
- Unknown-outcome handling: a documented, tested path reads the authoritative step marker before any retry.
- Terminal failure paths: out-of-retention recovery points, rejected plans, and unreachable targets end the operation as
failed_terminal, never as a silent substitute. - Erasure ledger enforced: plan generation consumes the ledger, and a named privacy owner approves the filter report.
- Scratch instance and plan data governed: access, encryption, retention, and destruction have named owners.
- Side effects controlled: restore mode suppresses outbound events, and derived stores have a re-sync procedure.
- Rehearsed end to end: a restore of a synthetic or consented test tenant has run through every stage in a non-production environment, including a forced mid-apply crash.
- Verified completion: the operation can close only after post-apply verification, and conflicts produce an explicit non-complete state.
Recommended rollout sequence
- Add safety nets first. Create named restore points before bulk jobs and align retention with how long tenants really take to notice problems.
- Ship the read-only half. Build recovery to scratch, extraction, and diff with no apply capability. Even this alone turns a multi-day scramble into a reviewable report.
- Add apply for a narrow table set. Start with leaf tables that have clean version columns. Expand along the dependency graph.
- Run game days. Kill the worker mid-batch, drop the connection during commit, change live rows after the plan, and confirm the pipeline lands in the right states.
- Document the customer-facing story. Decide what you tell the tenant about conflicts, skipped rows, and timelines.
General contingency-planning guidance, such as NIST SP 800-34 on contingency planning for information systems, emphasizes planning, testing, and exercising recovery. It is written for federal systems and is a general reference. It does not describe tenant-level restore, and you should map it to your own obligations rather than assume it applies as written.
Guidance by persona
| Persona | Decide first | Watch for |
|---|---|---|
| Tech Lead | Plan format, row-version strategy, dependency graph per table | Tables with no version column, cascading deletes, triggers that fire on restored rows |
| DevOps Lead | Scratch instance provisioning, retention alignment, worker isolation | Recovery duration at your data size, quota limits, leftover scratch instances |
| Engineering Manager | Ownership, rehearsal cadence, on-call authority to approve plans | Restore knowledge living in one person's head |
| CTO evaluating engineering depth | Which tenant-level RPO and time-to-restore you will promise contractually | Promises made before a rehearsal has produced evidence |
Could your team restore one tenant tonight without touching the others?
If the honest answer involves a wiki page and a hopeful script, it is worth a structured review before the next incident.
Talk to Developers.dev engineers about recovery design, rehearsal, and DevOps support for your multi-tenant platform.
Contact Us2027 Update, framed to stay valid
Two shifts are worth anchoring here without tying the advice to a date. First, managed databases and backup services continue to expose recovery as create a new copy, as the Amazon RDS documentation does, which keeps the application-level gap described above in place. Second, teams are increasingly letting automation, including AI agents, propose or even run operational fixes. Whatever executes a restore, human or automated, should hit the same controls: an approved plan hash, conditional writes, and verification. For agent-driven operations specifically, see our design for approval-gated AI agent actions with tenant-scoped execution. The fundamentals (granularity mismatch, conflict handling, erasure, and side effects) do not change with tooling. Re-check your database vendor's current recovery documentation whenever you upgrade engines or change platforms.
If you want help standing up the infrastructure side of this, such as scratch environments, observability, and runbooks, our DevOps services team works with engineering leads on exactly this kind of operational design.
Conclusion: Decide Before the Incident, Rehearse Before You Trust It
Per-tenant restore is not a backup feature you switch on. It is an application capability you design. The native tools give you an accurate copy of the past, and your job is to move the right parts of it into a live system safely. Five actions to take, in order:
- Measure the gap. Record how long tenants take to detect data problems and compare it to your recoverable history. Adjust retention, and start creating named restore points before bulk jobs.
- Name the owners. Assign owners for the scratch instance lifecycle, the erasure ledger, and restore approval. Write down who decides when the tenant's wishes and your obligations conflict.
- Build the read-only half first. Recovery to scratch, tenant slice extraction, and a diff report deliver value before any apply logic exists.
- Add conditional apply with step markers. Bind approval to an immutable plan hash, claim work atomically, use target-side conditional writes, and treat unknown outcomes as unconfirmed.
- Rehearse, then require every control. Run a mid-apply crash drill on a test tenant, and treat the go/no-go list as all-required.
About the authors and review: This article was prepared by Developers.dev engineering contributors and reviewed by the Developers.dev Expert Team. Developers.dev is a software development and staff augmentation company founded in 2007, with 1,000+ in-house IT professionals and a stated portfolio of 3,000+ projects, delivering to clients mainly in the USA, EMEA, and Australia. The company states that it holds CMMI Level 5, ISO 27001, and SOC 2 credentials. Those are company-level statements, not a certification of the design described here or of your own compliance status. The design is a proposal to adapt and test in your environment, and legal and privacy obligations remain specific to you.
Frequently Asked Questions
Can I restore a single tenant with native PostgreSQL point-in-time recovery?
Not directly. The PostgreSQL documentation states that continuous archiving supports restoring an entire database cluster, not a subset. The practical approach is to recover to a separate scratch instance at your chosen point, extract the tenant's rows, and apply them to production through a reviewed, conditional process.
Does Amazon RDS point-in-time restore overwrite my production database?
According to the Amazon RDS documentation, restoring to a point in time creates a new DB instance and does not modify the source instance. That makes it a good fit for the scratch-instance stage. You can only restore within your backup retention period, and Aurora uses a different API, so confirm the details for your engine.
Why not just upsert the restored rows over the live ones?
Upserts overwrite anything the tenant legitimately changed after the incident. A safer approach compares the recovered slice with live data, records the expected row version for each step, and applies changes only when the live row is still at that version. Rows that changed since become conflicts for review.
How do I handle a timeout while applying a restore batch?
Do not retry blindly. If each step's marker is written in the same transaction as the mutation, you can read that marker after bounding the old session with timeouts. A present marker tells you the outcome. If it is absent after the old session is gone, the step did not commit, and the system can proceed. If you still cannot establish the result, mark the operation as outcome unknown and escalate. Unknown is not success and not verified failure.
Could a restore bring back data that was deleted for privacy reasons?
Yes, if the recovery point predates the erasure. Maintain an erasure ledger that survives restores, filter plans against it, and have a named privacy owner review the filter report. Whether and how long backups may retain erased data depends on your jurisdiction and contracts, so consult your counsel.
Are plan files and diffs sensitive?
Yes. Primary keys, emails, and external IDs can identify people even without other fields. Encrypt plans, restrict access, apply a retention limit, and keep row contents out of general logs.
Is database-per-tenant the simpler answer?
It simplifies per-tenant recovery and increases blast-radius isolation, but it multiplies operational work such as migrations, monitoring, and backups by tenant count. For a large number of small tenants, shared schemas with a selective restore pipeline are often more practical. The right choice depends on tenant size, contracts, and team capacity.
How often should we rehearse tenant restore?
Rehearse after any significant change to the schema, the backup platform, or the restore tooling, and on a regular cadence your team can sustain. Include a forced worker crash and a changed live row to test conflict handling. The right frequency depends on your risk profile and release cadence.
Your next data incident will not wait for a perfect runbook.
Recovering one tenant safely takes plan hashing, conditional writes, privacy ownership, and rehearsal. It also takes engineers who have run this kind of recovery before.
Work with Developers.dev's cloud, DevOps, and SRE engineers to design and rehearse tenant-level recovery for your SaaS platform.
Contact UsPlatform Engineering Services
This guide is designed for engineering leaders who want to plan a practical implementation. Use the related Developers.dev path to compare delivery options, implementation fit, risk, and practical next steps.
Reviewed by the Experts team
This guide is reviewed for clarity, technical and operational relevance, service alignment, and a useful next step. Verified by our SEO team for clear search presentation.
Reviewed by the Experts team. Verified by our SEO team. Validate legal, security, data, budget, and operational requirements with the relevant stakeholders before rollout.

