Worker Leases and Fencing Tokens: A Tenant-Safe Design to Stop Duplicate Side Effects in Job Queues

A lease decides which worker should be doing a job. A fencing token decides whose write the system will accept. Teams that rely on the lease alone eventually ship a duplicate refund, a double-sent notification, or a clobbered record. This usually happens after a worker stalls, loses its lease, and then wakes up and carries on as if nothing changed.

This article is for tech leads, senior developers, and solution architects who run background workers, queue consumers, or agent runners that cause consequential side effects. It covers why lease expiry cannot be trusted by itself, what a more defensible claim protocol looks like, and where the fence has to be enforced for it to mean anything. It also covers what to do when the outcome of an attempt is genuinely unknown.

The design below, which we call Lease-Plus-Fence, is a proposed design assembled from established distributed-systems ideas. It is not a certified method, and it is not a product. All code is illustrative. It uses PostgreSQL syntax and must be adapted and tested against your own schema, isolation level, and failure modes before you rely on it.

1. Primary reader: a tech lead or solution architect who owns the worker tier.
2. Buyer-journey stage: execution and delivery. The queue exists, and the question is how to make concurrent workers safe.
3. Related reading on this site: idempotency in distributed systems, ambiguous timeouts and reconcile-before-retry, and dead letter queue redrive design.

Key Takeaways (BLUF)

1. Leases give liveness, fences give safety. A lease lets another worker take over a stalled job. Only a check at the resource being changed can stop the stalled worker from doing damage when it wakes up.
2. Never trust the worker's own clock or its belief that it still holds the lease. Pauses and network delays can happen between the last check and the write. The check must be part of the write itself.
3. Claim work with one atomic statement that increments a per-operation fence, sets a lease from the database clock, and returns both. Completion must be conditional on that same fence.
4. Bind each operation to a tenant-scoped identity and an immutable payload hash. Reject a reused key with a different payload. A legitimate later operation gets a new identity from business intent, not from a timer.
5. An unknown outcome is not success and not failure. After an ambiguous attempt, reconcile against the authoritative system before any retry. If that is impossible, park the job as unknown.
6. Use all-required go/no-go controls, not a score. A summed score can certify a design that is missing the one control that matters.

Why "Only One Worker Runs This Job" Is Not a Guarantee You Can Buy

In short: any mechanism that hands out work for a bounded time will sometimes have two workers who both believe the job is theirs. You can make that rare. You cannot make it impossible without help from the resource being modified.

Martin Kleppmann's widely cited analysis, How to do distributed locking, separates two reasons for a lock. An efficiency lock avoids wasting work, and an occasional duplicate is acceptable. A correctness lock protects state, and a duplicate means corruption, lost data, or an irreversible action. Most worker-tier incidents come from using an efficiency-grade mechanism for a correctness-grade job.

The same analysis shows why a lease is not enough. A lease has a timeout, which is the right design because a crashed worker must not hold a job forever. But a worker can be paused after it checked its lease and before it writes. The pause might be a garbage collection stop, a page fault, CPU starvation, or a write request delayed in the network. When it resumes, the lease has expired and a second worker may already be running. The first worker's check on its own lease cannot close that gap, because the pause can fall exactly between the check and the write.

Cause of a stale worker	What the worker believes	What is actually true
Long runtime pause or CPU starvation	"I checked, my lease is valid."	Lease expired during the pause; another worker claimed the job.
Network delay on the outbound write	"I sent the write in time."	The write reaches the target after a newer attempt has already acted.
Broker redelivery before completion	"This is my only copy of the message."	Visibility timeout lapsed; the message was delivered again.
Slow downstream dependency	"The call is still in flight."	The call may complete, but the lease and any retry have moved on.
Heartbeat thread alive, work thread stuck	"I am still renewing, so I am fine."	The lease is held but no progress is being made (a liveness failure, not a safety one).

Standard message brokers describe this behavior openly. For example, the AWS documentation on Amazon SQS visibility timeout explains that a received message is hidden for a period, and that if it is not deleted before the period ends it can be received again. Check the current documentation for the queue type you use, and design consumers on the assumption that a message can be processed more than once.

Implication for you: stop asking "how do I guarantee a single worker?" Ask instead, "what happens at the resource if a second worker, or a stale first worker, writes?" Safety must live there.

How Most Teams Approach It, and Where Each Approach Stops Being Safe

In short: common mechanisms are good at one property. Few provide both mutual exclusion and a way for the target to reject stale writers.

Approach	What it does well	Gap for consequential work
Cache-based lock with TTL (set-if-absent)	Cheap, fast, fine for avoiding duplicate effort	No monotonic token. A paused holder can still write after expiry. Kleppmann also argues that multi-node variants rely on timing assumptions that real systems violate.
Broker visibility timeout only	Simple redelivery after a crash	Redelivery is the very event that creates the second worker. Nothing stops the first from finishing.
Row claim with FOR UPDATE SKIP LOCKED	Lets many workers pull from a queue-like table without blocking each other	The row lock is held only for the transaction. Once the claim commits, it is just a status flag. The PostgreSQL documentation itself notes that skipping locked rows gives an inconsistent view of the data and is suitable for queue-like consumption, not general-purpose reads.
Session or advisory locks	Strong exclusion while the session lives	Tied to connection state. Connection pools, failovers, and long external calls complicate it, and the lock is invisible to the external target.
Leader election for a singleton worker	Reduces concurrency	Leaders can be deposed while mid-action. The same stale-leader problem applies without a token at the target.

The pattern behind every row is the same. The exclusion mechanism and the thing being protected are different systems, and only one of them knows the truth. A fencing token fixes this by making the protected resource take part. In Kleppmann's formulation, each lock acquisition gets a number that always increases, the client sends it with every write, and the storage service rejects any write carrying a lower number than one it has already seen.

This requires the target to take an active role. If your target is a database you own, that is straightforward. If it is a third-party API that knows nothing about your tokens, the fence cannot be enforced there, and you need a different control (covered in the fourth section).

For retry-safe request handling at the API boundary, see our durable webhook receiver design. That article covers inbound delivery. This one covers what happens after the work is accepted and several workers compete to run it.

Lease-Plus-Fence: A Proposed Design for Atomic Claim, Heartbeat, and Conditional Completion

In short: store each operation as one durable row keyed by tenant and operation identity. Claim it with one atomic statement that bumps a fence and sets a lease. Make every later state change conditional on the fence you were given.

Design rules

1. Durable identity: the key is (tenant_id, operation_id). The operation_id comes from the caller's business intent, such as the refund request they created, not from a timestamp or a retry counter.
2. Immutable approved payload: store a hash of the approved payload at enqueue time. If the same key arrives with a different hash, reject it. Never silently overwrite.
3. Atomic claim: a single statement selects a candidate with FOR UPDATE SKIP LOCKED, increments fence, and sets the lease from the database clock.
4. Conditional everything: heartbeat, completion, and failure all include fence = :my_fence AND state = 'claimed'. A row count of zero means "you no longer own this". The worker must stop and must not retry the write.
5. Lease is liveness, fence is safety: losing a lease only lets someone else try. The fence decides whose write counts.
6. Unknown is a state: if the attempts are exhausted and the last outcome could not be confirmed, the job moves to unknown and waits for reconciliation. It does not move to failed or succeeded.

Illustrative schema

CREATE TABLE work_item (   tenant_id        text        NOT NULL,   operation_id     text        NOT NULL,   payload_hash     bytea       NOT NULL,   -- hash of the approved, immutable payload   payload_ref      text        NOT NULL,   -- pointer to payload; keep PII out of this table   state            text        NOT NULL DEFAULT 'queued',   fence            bigint      NOT NULL DEFAULT 0,   lease_owner      text,   lease_expires_at timestamptz,   attempts         int         NOT NULL DEFAULT 0,   max_attempts     int         NOT NULL DEFAULT 5,   outcome_code     text,   updated_at       timestamptz NOT NULL DEFAULT now(),   PRIMARY KEY (tenant_id, operation_id),   CHECK (state IN ('queued','claimed','succeeded',                    'failed_terminal','unknown','quarantined')) );

Enqueue: bind identity to payload

-- 1. Atomic insert. The primary key decides the race, not application code. INSERT INTO work_item (tenant_id, operation_id, payload_hash, payload_ref) VALUES (:tenant_id, :operation_id, :payload_hash, :payload_ref) ON CONFLICT (tenant_id, operation_id) DO NOTHING RETURNING operation_id;  -- 2. If no row came back, the key already exists. Compare the stored hash. --    The stored hash is immutable, so this read-compare cannot race with a write. SELECT payload_hash, state FROM work_item WHERE tenant_id = :tenant_id AND operation_id = :operation_id; -- equal hash   -> replay: return the existing state, schedule nothing new -- different    -> reject with a conflict error; do not mutate anything

The insert itself is the inbox record and the work schedule at the same time. There is no separate "remember the key" step that could be lost between two systems. If you must also publish an event, write it in the same transaction, using the approach in our outbox pattern guide.

Claim: one atomic statement

WITH candidate AS (   SELECT tenant_id, operation_id   FROM work_item   WHERE attempts < max_attempts     AND (state = 'queued'          OR (state = 'claimed' AND lease_expires_at < now()))   ORDER BY updated_at   FOR UPDATE SKIP LOCKED   LIMIT 1 ) UPDATE work_item w SET state            = 'claimed',     fence            = w.fence + 1,     lease_owner      = :worker_id,     lease_expires_at = now() + interval '60 seconds',     attempts         = w.attempts + 1,     updated_at       = now() FROM candidate c WHERE w.tenant_id = c.tenant_id   AND w.operation_id = c.operation_id RETURNING w.tenant_id, w.operation_id, w.fence, w.attempts, w.payload_ref;

Two details matter. First, now() is the database clock, so no worker's clock decides when a lease ends. Second, a claim with attempts > 1 means a previous attempt may already have produced a side effect. That worker must reconcile before acting, not just repeat the call.

Heartbeat, complete, and park as unknown

-- Heartbeat: extend only if I still own this exact claim. UPDATE work_item SET lease_expires_at = now() + interval '60 seconds', updated_at = now() WHERE tenant_id = :tenant_id AND operation_id = :operation_id   AND fence = :fence AND state = 'claimed'; -- rows = 0 -> ownership lost: stop work immediately.  -- Complete: conditional on the same fence. UPDATE work_item SET state = 'succeeded', outcome_code = :code,     lease_owner = NULL, lease_expires_at = NULL, updated_at = now() WHERE tenant_id = :tenant_id AND operation_id = :operation_id   AND fence = :fence AND state = 'claimed'; -- rows = 0 -> a newer claim exists; do not retry this write.  -- Sweeper: exhausted attempts with an expired lease are UNKNOWN, not failed. UPDATE work_item SET state = 'unknown', updated_at = now() WHERE state = 'claimed' AND lease_expires_at < now()   AND attempts >= max_attempts;

The completion statement is a true conditional mutation executed by the system that owns the state. That is different from reading the row, comparing in application code, and then writing. Between the read and the write another worker can claim the row, so a read-compare-write sequence is not compare-and-swap. If you ever cannot express the condition inside the write itself, document that limitation and add serialization (for example, a row lock held across the check and the write) rather than hoping the window is small.

Where the Fence Must Be Enforced: Four Target Types and What Each Allows

In short: the fence is only as strong as the weakest system that performs the side effect. Classify every target before you promise anything about duplicates.

Target type	How to enforce	What you can honestly claim
A. Same relational database as the work table	Within one transaction, lock the work row with the fence and state conditions (SELECT ... FOR UPDATE), perform the business write, then update the work row. Commit once.	A stale worker cannot commit its business write. This is the strongest case. See PostgreSQL's explicit locking documentation for row-lock behavior.
B. Data store with conditional writes	Store the fence or a version on the target record and use the store's conditional write. For example, DynamoDB supports condition expressions that make a write succeed only if an attribute meets your condition.	Writes with a lower fence are rejected by the store. Keep the condition in the write request, not in a prior read.
C. External API with a documented idempotency key and a way to look up the result	Send a tenant-scoped key derived from (tenant_id, operation_id). On ambiguity, look up the result by that key before retrying.	Duplicate attempts are deduplicated as the provider defines it. Verify key retention window, payload-mismatch behavior, and scope in the provider's documentation. This is dedupe, not fencing.
D. External system with no idempotency and no lookup	Cannot be fenced. Limit to a single attempt, record the intent before the call, and send ambiguous results to unknown for human or out-of-band reconciliation.	At most one attempt. You cannot claim exactly-once. Consider an approval step for high-impact actions.

Pattern for target type A, illustrative

BEGIN; -- Fails to return a row if I lost the claim; a concurrent reclaim skips this locked row. SELECT 1 FROM work_item WHERE tenant_id = :tenant_id AND operation_id = :operation_id   AND fence = :fence AND state = 'claimed' FOR UPDATE; -- If no row: ROLLBACK and stop. Otherwise do the business write: UPDATE invoice SET status = 'refunded' WHERE tenant_id = :tenant_id AND invoice_id = :invoice_id   AND status = 'paid'; -- Then mark the work row complete (conditional on the same fence), and: COMMIT;

A WHERE status = 'paid' guard on the business write is a second, independent safety net. It means even a fencing bug cannot refund an already refunded invoice. Defense in depth is cheap here.

If you use a coordination service

If you generate fences from a consensus system instead of a database counter, the principle is the same. The etcd project's lock contrib README discusses version-number validation, where the shared resource validates requests from clients. The token is only useful if the resource checks it. A token that is generated but never validated is decoration.

The ambiguous-outcome rule

For consequential mutations, a timeout means unknown. It does not mean failed. Query the authoritative system by operation identity first. Only if it confirms the effect did not happen should a new attempt proceed, under a new fence. Our reconcile-before-retry design walks through that sequence.

The Integration Contract: Acknowledgment, Terminal Failure, Uncertain Input, and Privacy Ownership

In short: a safe claim protocol is useless if the surrounding contract lets messages vanish, retries run forever, or sensitive identifiers leak into logs. Define each outcome and its owner explicitly.

Verified acknowledgment

Acknowledge the broker message (delete it, commit the offset, or return a success response to a sender) only after the work row is durably committed, because that row is both the inbox record and the schedule. If the commit fails, do not acknowledge, and let redelivery happen. If you must acknowledge before the work completes, which is normal, the work row is the source of truth for what remains. The broker is only a wake-up signal. Never rely on an in-memory queue as the record.

Outcome table

State	Meaning	Automatic retry?	Who owns the next step
succeeded	Effect confirmed, completion recorded under the winning fence	No	Nobody. Replays return this result.
failed_terminal	A confirmed failure that retrying cannot fix (validation rejected, permission denied, business rule violated)	No	Service owner. Needs a new operation if the business still wants it.
unknown	Completion could not be confirmed. Not success, and not verified failure.	Only after reconciliation	On-call plus the owner of the authoritative system
quarantined	Uncertain input: unknown schema version, unparseable payload, or hash mismatch with the approved payload	No	Producer team. Fix the producer or approve a new payload.
queued or claimed	In progress or waiting	Yes, bounded by max_attempts	The worker tier

Terminal failure and redrive

Every retry path needs a ceiling and a destination. Bounded attempts, then failed_terminal or unknown, never an infinite loop. Redriving parked items is itself a consequential action. It must preserve the original operation identity and must re-run reconciliation for anything in unknown. The dead letter queue redrive playbook covers the operational side.

Uncertain input

If the worker cannot interpret an input with confidence, it should not guess. Unknown event types, unsupported versions, and fields that fail validation go to quarantined. A worker that "best-effort" interprets an ambiguous payload for a financial or data-deleting action is making a policy decision that nobody approved.

Tenant-safe identity

1. Include tenant_id in every key, every query predicate, and every lock or cache key. A bare operation_id shared across tenants invites cross-tenant collisions and information leaks.
2. Enforce tenant scoping in the database as well as in application code, for example with row-level security or per-tenant schemas, depending on your isolation model. Our multi-tenant database isolation guide compares the options.
3. Distinguish legitimate later operations from replays. Two refunds for one order are two operations with two identities. Two deliveries of one refund request are one operation. The caller's intent object decides, not the arrival time.
4. Do not reuse a key with a different payload hash, even if the first attempt failed. Create a new operation instead, so the audit trail stays honest.

Privacy ownership

Identifiers can themselves be sensitive. A tenant ID, user ID, or an operation ID derived from an email address can reveal who did what. Treat them accordingly.

1. Keep payloads by reference. The work table should hold pointers and hashes, not raw personal data.
2. Assign an owner for retention and deletion of work rows and payloads, and make sure erasure workflows reach them. See verified erasure workflow design.
3. Hash or tokenize identifiers in logs and metrics. Avoid putting raw payloads or high-cardinality personal identifiers in labels.
4. Scope any compliance statement carefully. Using a lease protocol does not make a system compliant with any privacy law. Your legal obligations depend on your data, your jurisdictions, and your contracts, and your counsel should confirm them.

Why This Fails in the Real World: Three Illustrative Failure Patterns

In short: intelligent teams still ship unsafe worker tiers because each component looks correct in isolation. The failures live in the seams and in the governance around them. The scenarios below are composite patterns, offered as illustrations, not reports about a specific customer.

Failure pattern 1: the lock was right, the write was unchecked

Scenario (illustrative): a team protects a payout job with a TTL lock in a cache. Under load, a downstream call slows, the lock expires, and a second worker takes over. The first worker's call completes and it writes its result. Both workers pay.

Why smart teams miss it: the code reviewed fine, because it did acquire a lock. The assumption that the lock holder is always the only writer was never written down as a requirement, so no test attacked it. Ownership was split. One team owned the lock library, another owned the payout API, and nobody owned the invariant that spans them.

Systemic fix: put the fence check in the write. Add a guard on the business row (status transitions), and name an owner for the end-to-end invariant.

Failure pattern 2: the heartbeat that lies

Scenario (illustrative): a worker renews its lease from a background thread while the main thread is blocked on a dead connection. The lease never expires, so no other worker takes over, and the job stalls silently. Operators then "fix" it by shortening the lease, which brings back the duplicates from pattern 1.

Why smart teams miss it: safety and liveness were tuned with the same knob. A lease duration is a liveness parameter. If it is also your only safety mechanism, every operational change trades one failure for the other.

Systemic fix: tie heartbeats to progress (renew only when the work loop advances), alert on claimed-but-not-progressing items, and let the fence carry safety so the lease can be tuned freely.

Failure pattern 3: the retry that was really a new request

Scenario (illustrative): an API client generates a fresh idempotency key on every retry, or two tenants happen to reuse the same sequential IDs. In the first case, duplicates sail past dedupe. In the second, one tenant's job is treated as another tenant's replay.

Why smart teams miss it: the dedupe logic was verified with a single tenant and a single client library. Key generation was left to each caller, with no contract and no mismatch rejection.

Systemic fix: define identity as (tenant_id, operation_id) from business intent. Store the payload hash and reject mismatches. Add contract tests that run with several tenants and colliding IDs.

Fault-injection tests worth having

1. Pause a worker after it claims and before it writes, past lease expiry. Confirm its write is rejected and the second worker's succeeds.
2. Kill the worker after the external call but before the completion write. Confirm the next claim reconciles instead of repeating.
3. Send the same key twice with the same payload, then with a different payload. Confirm replay versus conflict.
4. Run two tenants with identical operation IDs. Confirm no cross-effects.
5. Fail the database over mid-claim. Confirm there is no double-claim and no lost row.

Go/No-Go Controls, Trade-offs, and a Lower-Risk Rollout (With a 2026 Update)

In short: treat the controls below as all-required gates for any worker that causes consequential side effects. If any single gate fails, the answer is no-go for that job class. Do not add up points. A design that scores well but lacks a fence at the target is still unsafe.

Mandatory go/no-go controls (every one must be true)

#	Control	Evidence to show
1	Operation identity is durable and tenant-scoped	Primary key includes tenant; contract test with colliding IDs
2	Identity is bound to an immutable approved payload hash; changed-payload reuse is rejected	Conflict test passes
3	Claim is atomic across concurrent workers	Concurrency test with many workers; no double-claim
4	Every state change is a target-side conditional mutation on the fence	Code review: no read-compare-then-write on ownership
5	The system performing the side effect enforces the fence, or the job is classed as type C or D with its stated limits	Target classification recorded per job type
6	Ambiguous outcomes route to reconciliation or unknown, never to blind retry or to success	Fault-injection test after external call
7	Terminal failure and quarantine exist, with bounded attempts and a named owner	Runbook and alert routes exist
8	Acknowledgment happens only after the durable work record commits	Crash test between receipt and commit
9	Identifiers and payloads follow a data-minimization, retention, and deletion plan with an owner	Data-flow review sign-off

Passing these does not certify a system as production ready. It means you have removed known, avoidable ways to cause duplicate side effects. You still need load testing, security review, and operational readiness checks appropriate to your context.

Trade-offs to accept with eyes open

Choice	Benefit	Cost or risk
Database-backed claims	Atomic, auditable, easy to reason about	Write load and row contention on the work table. Very high throughput may require sharding or a different store.
Short leases	Fast recovery from crashed workers	More spurious takeovers, so you rely harder on the fence and on reconciliation
Long leases	Fewer duplicate attempts	Slow recovery and larger stall windows
Conservative unknown state	Avoids wrong automatic decisions	Needs people or tooling to resolve. Some items wait.
Type C and D targets	Can integrate with any external system	No fencing. You are limited to dedupe or single attempt, and must say so to stakeholders.

A lower-risk rollout

1. Inventory job types by side-effect class: reversible, idempotent, irreversible. Start with the irreversible ones.
2. Classify each target as A, B, C, or D from the fourth section, and write down what you can truthfully claim.
3. Shadow first: run the claim table alongside the existing mechanism and compare which jobs each would have double-run. Do not switch the system of record until the comparison is clean.
4. Cut over one job type and keep the old path behind a flag until the fault-injection tests pass in a production-like environment.
5. Operationalize: dashboards for claimed-but-stalled, unknown age, quarantine rate, and takeover rate.

If your worker tier runs AI agent actions that call tools with real-world effects, the same controls apply, with an extra approval gate for high-impact calls. See approval-gated AI agent actions.

2026 update, framed for the long term

As of 2026, more teams are putting long-running, retry-prone work behind queues: LLM pipelines, agent tool calls, and data backfills. These jobs are slower and more variable than classic request handlers, which makes lease expiry during healthy work more likely, and the cost of a duplicated action higher. The underlying principle has not changed since it was described in the distributed-systems literature: time-based ownership cannot be the last line of defense. Tools, brokers, and cloud services will keep changing. The invariants will not. Identity, immutable intent, atomic claim, a check at the resource, and an honest unknown state.

Not sure which of your jobs can safely double-run?

Most worker tiers have a few jobs where a duplicate is harmless and a few where it is not. Finding the difference before an incident is cheaper than after.

Talk to Developers.dev engineers about a worker-tier safety review for your queue, scheduler, or agent runtime.

Contact Us

Conclusion: What to Do on Monday

The decision in front of a tech lead is not whether to adopt a particular library. It is whether each consequential job can be shown to be safe when two workers believe they own it. Five concrete actions:

1. List your consequential job types and classify each target as A, B, C, or D. Write the strongest honest claim you can make about duplicates for each one.
2. Find every place where ownership is decided by a read followed by a write, and replace it with a single conditional mutation on a fence, or document the serialization you rely on.
3. Re-key operations as (tenant_id, operation_id) bound to an immutable payload hash, and add the changed-payload and colliding-tenant tests.
4. Add the unknown and quarantined states and assign an owner to each. Route every ambiguous timeout through reconciliation.
5. Run the pause-after-claim test in a staging environment. If a stale worker can still change the target, that job is no-go until it cannot.

About the authors and review. This article was prepared by the Developers.dev engineering content team. Developers.dev is an offshore software development and staff augmentation company founded in 2007, with in-house engineers who build and operate cloud, data, and AI systems for clients in the USA, EMEA, and Australia. The company states that it holds CMMI Level 5, SOC 2, and ISO 27001 credentials. These are company-level statements and do not certify the design described here. Reviewed by Developers.dev Expert Team.

References used: Martin Kleppmann, How to do distributed locking; PostgreSQL SELECT documentation (locking clause); Amazon SQS visibility timeout; DynamoDB condition expressions. Always confirm current product behavior in the vendor's own documentation.

Frequently Asked Questions

What is the difference between a lease and a fencing token?

A lease is a time-limited claim that lets the system recover when a worker crashes. A fencing token is a number that increases each time a claim is granted and is sent with every write, so the target can reject a write from an older claim. The lease provides liveness. The fence provides safety. You need both for consequential work.

Is FOR UPDATE SKIP LOCKED enough to prevent duplicate processing?

No. It is good for letting many consumers pull from a queue-like table without blocking each other. The PostgreSQL documentation notes that skipping locked rows gives an inconsistent view of the data and is not suited to general-purpose reads. The row lock lasts only until the transaction ends. After that, you need a durable claim with a fence and conditional completion.

1. Use it for the claim step.
2. Do not treat it as ownership for the length of a long external call.

Can I use a Redis-style TTL lock for correctness-critical jobs?

Not on its own. Such locks can be reasonable as an efficiency optimization, where an occasional duplicate is acceptable. For correctness, the analysis by Martin Kleppmann recommends a system that provides monotonic fencing tokens and a target that enforces them. If you keep a TTL lock, add a fence check at the resource, or a status guard on the business record.

What if the external API I call does not support fencing?

Then you cannot honestly claim fencing for that target. Check whether it offers an idempotency key and a lookup by key. If it does, send a tenant-scoped key and reconcile before retrying, and verify the provider's retention and mismatch behavior. If it offers neither, limit the job to one attempt, record the intent first, and park ambiguous results as unknown.

How should I handle a timeout after calling an external system?

Treat it as unknown. Do not retry blindly and do not mark success. Query the authoritative system using the operation identity. If it confirms the effect happened, record success. If it confirms it did not, retry under a new fence. If you cannot confirm either, move the job to unknown for reconciliation.

How do I tell a legitimate second operation from a duplicate?

Derive the operation identity from business intent, such as the specific refund request, not from time or a retry counter. The same identity with the same payload hash is a replay. The same identity with a different payload hash is a conflict and must be rejected. A new intent, even for the same order, gets a new identity.

Does this design make my system compliant with privacy regulations?

No. It supports good data handling by keeping payloads by reference and by giving deletion an owner, but compliance depends on your data, jurisdictions, and contracts. Treat identifiers as potentially sensitive, minimize what you log, and have your legal counsel confirm your specific obligations.

Is your queue one stalled worker away from a duplicate payout?

Leases, fences, reconciliation, and tenant-safe identity are the kind of details that separate a demo from a system you can trust at 3 a.m.

Bring in Developers.dev engineers, including dedicated backend developers and DevOps specialists, to review your worker tier or build it with you.

Contact Us

Related service

Platform Engineering Services

This guide is designed for engineering leaders who want to reduce delivery, security, and operational risk. Use the related Developers.dev path to compare delivery options, implementation fit, risk, and practical next steps.

Read the primary guideRequest a free consultation

Editorial review

Reviewed by the Experts team

This guide is reviewed for clarity, technical and operational relevance, service alignment, and a useful next step. Verified by our SEO team for clear search presentation.

Reviewed byDevelopers.dev Experts Team

Reviewed2026-10-30

FocusPlatform Engineering Services

SEO verified byDevelopers.dev SEO Team

SEO verified2026-10-30

Reviewed by the Experts team. Verified by our SEO team. Validate legal, security, data, budget, and operational requirements with the relevant stakeholders before rollout.