Incremental Sync from Third-Party APIs: A Checkpoint, Tombstone, and Reconciliation Design

Incremental sync from a third-party API is safe only if three things hold. The cursor advances only after the data it covers is durably applied. Deletes and invalid tokens are first-class states, not edge cases. A scheduled reconciliation pass catches whatever the change feed missed. Most sync bugs are not fetch bugs. They are commit-ordering bugs, and they stay silent until a customer notices a missing record weeks later.

This article is written for solution architects and tech leads who own a pull-based integration, for example a CRM, calendar, directory, or billing sync that runs per tenant. Webhook receivers are a different problem, and we cover them in our guide to a durable webhook receiver. Here the system pulls changes on a schedule using a sync token, delta link, or updated-since filter.

The design below is a proposed design, not a standard and not a method any vendor mandates. It combines patterns documented by providers with general distributed-systems practice. Code is illustrative: it uses PostgreSQL-style SQL and pseudocode, not an official SDK contract, so adapt it to your provider's documented behavior.

1. Primary persona: Solution Architect or Tech Lead.
2. Stage: execution and delivery of a production integration.
3. Outcome: a set of mandatory go/no-go controls you can test before you ship.

Key Takeaways

1. Advance the cursor last. Commit the checkpoint in the same transaction as the data it covers. If the two can diverge, you have a silent-gap bug waiting to happen.
2. Fence every worker. A stale worker must not be able to commit a checkpoint. Use a lease epoch checked by a conditional update on the database side.
3. Treat invalid tokens and deletes as states. Google's Calendar guide documents a 410 response that requires a full sync. Microsoft Graph's delta query represents removals with an @removed annotation. Build both paths before launch.
4. Unknown is not success. If you cannot confirm that a page or a write was applied, read the authoritative state before retrying anything consequential.
5. Reconcile on a schedule. Change feeds can replay, delay, or omit. A periodic comparison is the only control that detects drift you did not anticipate.
6. Own the privacy surface. Tenant IDs, source IDs, and cursors can all identify people or reveal data. Assign a named owner for retention and erasure of staged payloads.
7. Use go/no-go controls, not a score. Every control in the table below must pass. A high average does not offset a failed fencing test.

Why Incremental Sync Fails Quietly: Context and the Usual Approach

Key takeaway: The usual implementation is a loop that fetches a page, writes records, and saves a cursor. It works in demos and breaks when a worker crashes between the second and third steps, because the order of those steps was never designed.

Why the problem exists

A full read of a third-party system is expensive and often rate-limited. Providers therefore offer a way to ask only for what changed since last time. Microsoft's documentation for Microsoft Graph delta query describes this as discovering created, updated, or deleted entities without a full read, and notes that it reduces the chance of throttling. Google's guide to synchronizing Calendar resources efficiently describes the same two-stage idea: an initial full sync that yields a sync token, then repeated incremental syncs that present the previous token and store the new one.

That convenience hides a contract. The provider promises a stream of changes relative to a token. You promise to apply those changes and remember the token. The failure window sits between those two promises.

How most teams approach it, and why it fails

1. Write a scheduled job per connection.
2. Fetch pages until the provider returns a final token.
3. Upsert records as each page arrives.
4. Save the final token at the end.

This looks reasonable. It fails in at least four ways:

1. Crash mid-run: the token is not saved, so the next run replays pages. If your writes are not idempotent, you duplicate side effects.
2. Token saved before writes finish: an optimization someone adds to reduce replays. A crash now skips changes permanently.
3. Two runs overlap: a slow run and its successor both operate on one connection. The slower one can overwrite newer data or save an older token.
4. Deletes ignored: the job upserts what it sees and never removes anything, so deleted source records live forever in your system.

Change-feed shapes and their hidden failure modes

Before designing anything, classify what the provider gives you. The first two rows are documented vendor behavior from the sources cited above. The remaining rows are generic patterns, so verify them against your provider.

Feed shape	Example	What it gives you	Hidden failure modes
Opaque sync token	Google Calendar syncToken	Server-computed position; deleted entries included in results	Server can invalidate the token (HTTP 410), forcing a full sync; query parameters must stay identical across requests
Delta link	Microsoft Graph @odata.deltaLink	Changes since last round; removed items carry @removed	Replays are possible; the same entity can appear several times; propagation delay is possible; do not assume ordering
Updated-since timestamp	Generic updated_after filter	Simple to implement	Timestamp ties, clock skew, late-committing transactions at the provider, and no deletes unless the provider soft-deletes
Offset or page-number paging over a mutable set	Generic ?page=7	Easy to reason about	Inserts and deletes during paging cause skipped or duplicated rows
Webhook only	Push events	Low latency	Missed deliveries and no history; needs a pull-based backstop
Full snapshot and diff	Nightly export	Simplest correctness story	Cost, rate limits, and delayed detection

One consequence follows. For every feed shape, ask what it does with deletes and what it does when the position is lost. The answers decide your architecture more than the fetch loop does.

A Proposed Design: The Sync Contract and Its Four States

Key takeaway: Model each connection as a small state machine with a durable, tenant-scoped identity. Every page you process ends in exactly one of four outcomes, and each outcome has a defined next step.

The sync contract (proposed)

We call the set of rules below a sync contract. This is a proposed design for your team to adopt or adapt. It is not an established proprietary method, and nothing in it is a provider requirement.

1. Identity: every stream is keyed by (tenant_id, connection_id, resource_type). Nothing is ever keyed by provider ID alone, because two tenants can legitimately connect to the same upstream account or reuse identifiers.
2. Single writer: at most one lease holder advances the checkpoint, enforced by the database and not by the scheduler.
3. Commit ordering: records and checkpoint change in one atomic transaction.
4. Explicit outcomes: applied, retryable, terminal (quarantined), or unknown.
5. Reconcile: a separate path compares your state to the provider's on a schedule.

The four page outcomes

Outcome	Meaning	Checkpoint	Next step
Applied	Transaction committed and confirmed by a read of committed_cursor	Advanced	Fetch next page
Retryable	Provider returned a throttle or transient error; nothing was applied	Unchanged	Back off using the provider's retry guidance, then repeat the same request
Terminal	Record or response cannot be processed (schema violation, permanent authorization failure)	Unchanged for the stream, or the record is quarantined by policy	Quarantine with a reason code, alert an owner, stop or skip by explicit policy
Unknown	Completion could not be confirmed, for example the connection dropped after COMMIT was sent	Undetermined until read	Read the authoritative checkpoint, then decide. Never assume success or failure

The word unknown matters. An unknown result means completion could not be confirmed. It is not success, and it is not a verified failure. Treat the two cases differently, because retrying an unknown can double-apply and skipping an unknown can lose data.

Data flow

[Provider change feed]      | request(cursor_in)      v [Fetcher: holds lease epoch N]      | page + cursor_out      v [Staging: raw page, content hash, tenant-scoped]      | validate (schema, tenant binding)      v [Apply transaction]   1. fence: UPDATE sync_state WHERE lease_epoch = N AND committed_cursor = cursor_in   2. upsert records guarded by source version   3. write outbox rows for downstream consumers   4. COMMIT (cursor advances here, and only here)      |      v [Outbox relay] --> downstream consumers (acknowledge per consumer)

If you already publish downstream events, step 3 is the outbox pattern. The point is that the checkpoint, the data, and the intent to notify consumers become visible together or not at all.

Checkpoint Commit Protocol: Lease, Fence, Apply, Advance

Key takeaway: A read-then-write check in application code is not a compare-and-swap. Put the check inside the conditional UPDATE so the database decides who wins.

Step 1: claim the stream atomically

Scheduler-level locks are advisory. A worker can be paused by garbage collection or a network partition, wake up after its lease expired, and keep going. Claim work with one conditional statement and use the returned epoch as a fencing token. We explain the broader failure mode in worker leases and fencing tokens.

-- Illustrative PostgreSQL-style SQL, not an official API contract UPDATE sync_state    SET lease_epoch      = lease_epoch + 1,        lease_owner      = $worker_id,        lease_expires_at = now() + interval '2 minutes'  WHERE tenant_id = $tenant    AND connection_id = $conn    AND resource_type = $type    AND (lease_owner IS NULL OR lease_expires_at < now()) RETURNING lease_epoch, committed_cursor; -- 0 rows returned: another worker holds the stream. Skip this cycle.

Step 2: fetch with the committed cursor, read-only

Fetching is a read, so a retry does not create a side effect. It still consumes your rate-limit budget, so honor the provider's documented retry guidance. Keep query parameters identical across a sync session. Both providers cited above say this matters: Google states each list request should use the same parameters, and Microsoft states that state tokens encode the original query parameters.

Step 3: apply and advance in one transaction

-- Illustrative. Run the fence statement first and check rows affected in app code. BEGIN;  UPDATE sync_state    SET committed_cursor = $cursor_out,        last_commit_at   = now()  WHERE tenant_id = $tenant AND connection_id = $conn AND resource_type = $type    AND lease_epoch = $my_epoch    AND committed_cursor IS NOT DISTINCT FROM $cursor_in; -- rows affected = 0  => ROLLBACK and stop; this worker is stale or the cursor moved.  INSERT INTO records (tenant_id, connection_id, source_id, source_version, payload) VALUES ($tenant, $conn, $source_id, $source_version, $payload) ON CONFLICT (tenant_id, connection_id, source_id) DO UPDATE SET payload = EXCLUDED.payload,               source_version = EXCLUDED.source_version  WHERE records.source_version < EXCLUDED.source_version;  INSERT INTO outbox (tenant_id, event_type, dedupe_key, body) VALUES ($tenant, 'record.changed', $dedupe_key, $body) ON CONFLICT (tenant_id, dedupe_key) DO NOTHING;  COMMIT;

Three details deserve attention:

1. The fence is the conditional UPDATE. The stale worker's statement matches zero rows, and the application aborts. This is target-side conditional mutation. PostgreSQL documents that INSERT ... ON CONFLICT DO UPDATE guarantees an atomic insert-or-update outcome under concurrency, and it supports a WHERE condition on the update, which is how the version guard works here.
2. The version guard needs a real source of ordering. If the provider supplies a monotonic version or ETag, use it. If it only supplies an updated_at timestamp, ties are possible. In that case compare a content hash as well and disclose the residual limitation: two distinct changes within one timestamp tick may be applied in either order.
3. Dedupe in the outbox is tenant-scoped. The key is (tenant_id, dedupe_key), never dedupe_key alone. A global key lets one tenant's event suppress another's. See the idempotency framework for the underlying rules.

Step 4: handle an unknown commit

If the database connection fails after you sent COMMIT, you do not know whether the transaction applied. Do not retry the page blindly and do not mark it failed. Open a new connection and read committed_cursor. If it equals cursor_out, the page was applied. If it equals cursor_in, it was not, and you can repeat the page. If it equals neither, another writer interfered, which is a sev-level event: stop the stream and alert.

def resolve_unknown_commit(tenant, conn, cursor_in, cursor_out):     state = read_state_fresh_connection(tenant, conn)   # authoritative read     if state.committed_cursor == cursor_out:         return 'applied'     if state.committed_cursor == cursor_in:         return 'not_applied'      # safe to repeat the same page     return 'diverged'             # stop stream, page an owner

Terminal failure handling

Decide up front what a permanently unprocessable record does. There are two defensible policies. Block the stream and alert, which preserves ordering and is safer for data where order matters. Or quarantine the single record with a reason code and continue, which protects freshness for the other records. Pick one per resource type and write it down. Quarantined items need an owner, a retention period, and a redrive path, which is the territory of dead letter queue redrive design.

Deletes, Invalid Tokens, and Replays: Tombstones and Generational Resync

Key takeaway: A deletion is data. If your pipeline cannot represent one, downstream systems keep stale records indefinitely. If your pipeline treats an invalid token as a reason to wipe, a provider hiccup can erase data downstream consumers already acted on.

Represent deletes explicitly

Google's synchronization guide says incremental results always contain deleted entries so clients can remove them from storage. Microsoft Learn says removed instances are represented by an id and an @removed object, whose reason can be changed (deleted but restorable) or deleted (not restorable). It also says restored items appear as newly created instances. Your schema should therefore distinguish at least these states:

Source signal	Your record state	Downstream event	Notes
Created or updated	Active, version advanced	record.changed	Guard with version or hash
Removed, restorable	Tombstoned (soft)	record.removed (reason: restorable)	Keep identifier so a later restore reconciles cleanly
Removed, permanent	Tombstoned, payload purged per policy	record.removed (reason: permanent)	Retain only the minimum identifier needed to ignore replays
Replay of an already-applied change	No change	None (deduped)	Version guard rejects it
Delete for an unknown ID	Optional tombstone	None	Can happen after a resync or out-of-order delivery; do not error

Replays are an expected condition, not a bug. Microsoft Learn states that applications must be prepared for them and that a single entity can appear multiple times in a response. It also warns not to assume a specific order from a delta query. An apply step that is idempotent and version-guarded absorbs all of this.

When the token is invalid: do not wipe in place

Google's guide says the server may invalidate a sync token, for example through expiration or ACL changes, and respond with HTTP 410. Its documented guidance is to clear the client's store and run a new full sync. For a local cache that is fine. For a system of record whose changes feed other services, an in-place wipe is dangerous: consumers see a flood of deletions followed by a flood of creations, and anything triggered by a deletion fires wrongly.

A safer proposed pattern is a generational resync:

1. Mark the stream resync_required. Stop incremental processing. Do not clear anything.
2. Start a new generation (generation = g + 1) and run the full read into a staging area keyed by generation.
3. Only when the full read finishes and the provider returns its final token, diff generation g + 1 against the live records.
4. Apply creates, updates, and genuine deletions through the same version-guarded, outbox-backed transaction path. Missing-from-snapshot records become tombstones with reason resync_absent, not silent deletes.
5. Swap the checkpoint to the new token and retire generation g.

Be careful with step 4. A snapshot can be incomplete if you stopped on a terminal error or a partial page. Only treat absence as deletion after the full read is confirmed complete.

Illustrative cost of a resync

The numbers below are assumptions to show why this matters. They are not measurements.

Assumption	Value
Records per tenant	2,000,000
Page size	500
Pages per full read	4,000
Tenants forced to resync together after a provider-side change	200
Total requests	800,000
Shared request budget across tenants	50 requests per second
Elapsed fetch time, ignoring apply time and throttling	about 4.4 hours

The lesson is operational. Give resyncs their own queue and budget, with per-tenant fairness, so one tenant's backfill does not starve everyone's incremental syncs.

Reconciliation, Write-Back, and Privacy Ownership

Key takeaway: Reconciliation is the control that finds drift you did not predict. Write-back needs a durable operation identity bound to an immutable payload. Privacy needs a named owner, because tokens, identifiers, and staged payloads can all expose personal data.

Scheduled reconciliation

Even a correct pipeline drifts: provider bugs, delayed propagation, human edits in the provider's admin console, or your own past defects. Reconciliation compares a cheap summary of your state to the provider's and escalates only on mismatch.

Technique	What it detects	Cost	Limitation
Count comparison per partition	Gross loss or duplication	Low	Offsetting errors cancel out
ID-set comparison (sampled or ranged)	Missing and orphan records	Medium	Needs a provider list endpoint that supports ranges
Hash comparison of canonical records	Content drift	Medium to high	Requires a stable canonical form on both sides
Rolling full verification	Everything above	High	Spread across days to respect rate limits

Reconciliation findings should go through the normal apply path. Do not patch the table directly, or you bypass version guards, tombstones, and outbox events.

Write-back and echo suppression

If your system also writes to the provider, your own change returns through the change feed. Without a guard you can treat your write as a fresh user edit and write it again, creating a loop. Record the operation ID and payload hash on each outbound write. When a matching change arrives, mark it as an echo and skip re-propagation.

For consequential outbound mutations, follow the discipline from our article on ambiguous timeouts in consequential API calls. In summary:

1. Durable identity: create an operation row keyed by (tenant_id, connection_id, operation_id) before calling the provider.
2. Immutable payload hash: store the hash of the approved payload on that row. If the same operation ID arrives with a different hash, reject it. Do not silently overwrite.
3. Atomic claim: a worker takes the operation with a conditional update from pending to in_flight, so two workers cannot send it concurrently.
4. Reconcile before retry: after a timeout, read the provider's state for that record (or look up by your idempotency key if the provider supports one). Retry only when the read proves the change did not land. If the read cannot answer, the status is unknown and goes to a human or a slower reconciliation job.
5. Legitimate later edits get new operation IDs. A second, deliberate change to the same record is a different operation, not a duplicate.

Be honest about provider-side concurrency. If the provider supports conditional updates (for example If-Match with an ETag), use them. That is the only true compare-and-swap you can get across a network boundary. If it does not, a read-compare-write sequence is not atomic. Serialize writes per record in your own system, and document that a concurrent edit made directly in the provider between your read and your write can still be overwritten.

Privacy ownership

Sync pipelines copy data across a trust boundary and then keep copies in logs, staging tables, outbox payloads, and dead-letter stores. Identifiers are not harmless: tenant IDs, user emails, and provider object IDs can identify people or reveal customer relationships. Address these explicitly:

1. Cursors are sensitive configuration. Microsoft Learn says state tokens encode the original query parameters, and both providers call tokens opaque. Store them with the same protection as the data, and keep them out of logs, traces, and metric labels.
2. Redact URLs in logs. A nextLink or deltaLink is a full URL carrying a token.
3. Keep identifiers out of high-cardinality metric labels. Use an internal surrogate tenant key if you need per-tenant dashboards, and restrict who can map it back.
4. Name an owner for retention of staged raw pages, quarantine items, and outbox history. Set a default expiry for each.
5. Make erasure traceable. A deletion request has to reach records, staging, outbox payloads, quarantine, and downstream stores, with verification at each. See our design for verified erasure in distributed systems.

Which laws and contracts apply depends on your tenants, their data, and your data processing agreements. This article is engineering guidance, not legal advice. Have counsel confirm your obligations.

Why This Fails in the Real World

Key takeaway: Capable teams still ship these bugs because each component looks correct in isolation. The failures live in the seams: ordering, ownership, and governance. The scenarios below are illustrative composites, not reports about specific customers.

Scenario 1: the optimistic checkpoint

A team saves the cursor after fetching each page to shorten replays after crashes. Apply work runs in an asynchronous queue. During a deploy, workers are restarted and the queue drains partially. The cursor is ahead of the data. No error appears, because every component reported success. Weeks later, a customer asks why three accounts are missing.

Why intelligent teams miss it: the fetcher and the applier have different owners, and no one owns the invariant that a checkpoint implies applied data. Control: single-transaction commit and the fencing test in the go/no-go table.

Scenario 2: the helpful wipe

A provider invalidates tokens for a group of tenants. The client library follows documented guidance and clears the local store before a full sync. Downstream billing consumes record-removed events and begins closing subscriptions. The records return an hour later, but the downstream actions already ran.

Why intelligent teams miss it: the documentation is correct for a cache, and the team applied it to a system of record. Control: generational resync with reasoned tombstones, plus a downstream hold on destructive actions from resync_absent events.

Scenario 3: the timestamp tie

A provider exposes only an updated_after filter. The job stores the maximum timestamp it saw and asks for strictly later records. Two records share the same timestamp at a page boundary, and one is never returned. Separately, a long provider-side transaction commits late with an older timestamp, which the next query also skips.

Why intelligent teams miss it: tests use small datasets with unique timestamps. Control: overlap the window (re-read a safety margin and rely on version-guarded idempotent apply), and run reconciliation to catch the rest.

Scenario 4: the shared pool

One worker pool serves all tenants. A large tenant starts a backfill and consumes the request budget. Small tenants fall behind for hours. In a code path that builds cache keys from provider IDs alone, two tenants that connected the same upstream account see each other's cached records.

Why intelligent teams miss it: multi-tenancy was treated as a database concern, not as a property of every key, queue, and cache. Control: tenant-scoped keys everywhere and per-tenant fairness limits. Our guide to multi-tenant database isolation patterns covers the storage side.

Failure-to-control map

Failure pattern	Root gap	Mandatory control
Silent gap after crash	Checkpoint and data not atomic	Single-transaction commit; unknown-commit resolution
Overwritten by stale worker	No fencing	Lease epoch checked in a conditional UPDATE
Destructive downstream actions after resync	Wipe-in-place	Generational resync; reasoned tombstones
Missed changes under timestamp feeds	Ties and late commits	Overlap window; version guard; reconciliation
Cross-tenant bleed	Keys lacking tenant scope	Tenant-scoped keys and dedupe; isolation test
Sensitive data in logs	No privacy owner	Token redaction; retention owner; erasure trace

Go/No-Go Controls, Recommendations by Persona, and a 2027 Update

Key takeaway: Every control below is mandatory. There is no score, and no total that certifies a sync as production-ready. One failed control is a no-go until it is fixed or an owner accepts the risk in writing.

Go/no-go table (all required)

#	Control	Pass condition	Evidence to produce
1	Atomic checkpoint	Cursor and data commit in one transaction; kill the worker between steps and no gap appears	Fault-injection test log
2	Fencing	A worker holding an old epoch cannot advance the cursor	Test with two workers and a forced lease expiry
3	Unknown-commit resolution	After a simulated lost COMMIT acknowledgment, the system reads authoritative state before acting	Test and runbook
4	Idempotent, version-guarded apply	Replaying any page ten times yields identical state and one set of downstream events	Replay test
5	Tombstones	Removed items (restorable and permanent) propagate; restored items reappear	Fixture covering each provider signal you use
6	Invalid-token path	Simulated invalid token triggers generational resync with no in-place wipe and no spurious destructive events	End-to-end test
7	Terminal failure policy	Block or quarantine is chosen per resource; quarantined items have an owner and retention	Written policy, alert rule
8	Tenant-scoped keys	Idempotency, dedupe, cache, and queue keys include tenant and connection; cross-tenant test passes	Isolation test
9	Write-back safety	Operation identity bound to payload hash; changed-payload reuse rejected; ambiguous timeout reconciled before retry; echoes suppressed	Test cases per rule
10	Reconciliation	Scheduled job exists, uses the normal apply path, and alerts on mismatch	Job config and a seeded-drift test
11	Fairness	Resync and backfill run on separate budgeted queues; a large tenant cannot starve others	Load test
12	Privacy ownership	Tokens and URLs are redacted; retention and erasure owners are named; erasure reaches staging, outbox, and quarantine	Log audit and erasure drill record

Recommendation by persona

1. Solution architect: insist on controls 1, 2, and 6 before any detailed design review. They are the hardest to retrofit.
2. Tech lead: build the fault-injection harness first. Controls 1 through 4 become cheap to prove once the harness exists.
3. Engineering manager: assign owners for controls 7, 10, and 12. These fail through neglect, not through code.
4. CTO evaluating a build-versus-buy decision: ask any vendor or partner to show evidence for each of the twelve rows. Vendor statements about reliability are not evidence for your tenants' data. Check them against the vendor's primary documentation.

2027 Update: what to re-verify, whatever the year

Provider behavior changes. Before each major release of your integration, re-read the provider's current synchronization documentation. Specifically confirm how long tokens remain valid, which query parameters are allowed on incremental calls, how deletes are signaled, whether ordering or replay guarantees have changed, and the current rate-limit rules. The behaviors cited here come from Google's Calendar synchronization guide and Microsoft's Graph delta query overview as they read when we reviewed them. Treat them as examples of the pattern, and treat your provider's live documentation as the contract. The twelve controls above are provider-neutral and should stay valid when any individual API changes.

Is your sync pipeline provably safe, or just untested?

A cursor that advances too early loses data silently. Find the gap before a customer does.

Talk to Developers.dev engineers about an integration design review, including fencing, tombstones, and reconciliation.

Contact Us

If you need hands-on capacity, see our API integration services or the staff augmentation model for adding senior backend engineers to an existing team.

Conclusion: Five Actions Before You Ship a Sync

Incremental sync is a contract between two systems, and the dangerous part is the moment between applying data and remembering where you stopped. Use these actions to close that gap.

1. Classify your provider's change feed. Write down how it signals deletes, what happens when a token or position is lost, and whether replays and reordering are possible. Base your design on the provider's live documentation.
2. Move the checkpoint into the apply transaction. Add a lease epoch and a conditional UPDATE as the fence. Then run a kill-the-worker test to prove no gap appears.
3. Replace wipe-in-place with generational resync. Represent deletions as reasoned tombstones, and hold destructive downstream actions that originate from a resync.
4. Schedule reconciliation and name the owners. Assign one owner each for quarantine, reconciliation alerts, and privacy retention and erasure.
5. Run the twelve go/no-go controls as a gate. Do not average them. If one fails, record who accepted the risk, or fix it.

About the authors and review

Developers.dev is an offshore software development and staff augmentation company, founded in 2007, with 1000+ in-house engineers delivering integration, cloud, data, and AI work for clients across the USA, EMEA, and Australia. The company lists CMMI Level 5, SOC 2, and ISO 27001 among its accreditations. These describe the company's own delivery processes. They do not make any particular integration, or your use of this design, compliant with any standard or regulation. This article was reviewed by the Developers.dev Expert Team.

Frequently Asked Questions

What is incremental sync, and how is it different from polling the whole API?

Incremental sync asks the provider only for changes since a stored position, such as a sync token, delta link, or timestamp. Full polling rereads everything each time. Providers like Google Calendar and Microsoft Graph document incremental mechanisms mainly to save bandwidth and reduce throttling. The trade-off is that you must manage the stored position correctly.

Why should the cursor be saved in the same transaction as the data?

If the cursor moves first and the data write fails, the skipped changes are never requested again. If the data is written first and the cursor save fails, you only replay pages, which an idempotent apply step absorbs. A single transaction removes the ordering question and makes the checkpoint a reliable statement that the covered data is applied.

What should I do when the provider says my sync token is invalid?

Stop incremental processing and mark the stream as requiring a resync. Google's guide documents an HTTP 410 response for this case and recommends a full sync. If your data feeds other systems, run the full read into a new generation, diff it against live records, and apply changes through the normal path. Avoid wiping live data first.

Can I use an updated-since timestamp as my cursor?

Yes, with caveats. Timestamp ties, clock skew, and late-committing transactions at the provider can cause missed records, and many such filters do not report deletes. Re-read an overlapping window, use a version or content-hash guard so replays are harmless, and run scheduled reconciliation to catch what the overlap misses.

Is a read-compare-then-write check enough to prevent overwrites?

No. Between your read and your write another actor can change the record. Use a conditional mutation evaluated by the target, such as a database WHERE clause on the lease epoch and cursor, or a provider conditional update like If-Match when supported. If the provider offers none, serialize writes per record on your side and disclose the residual race.

How do I handle a timeout when writing back to the provider?

Do not retry blindly. Treat the result as unknown, read the provider's authoritative state for that record, and retry only if the read proves the change did not apply. Bind the operation ID to an immutable payload hash and reject reuse with a different payload. Give legitimate later edits a new operation ID.

Do sync tokens and identifiers count as sensitive data?

They can. Microsoft Learn notes that state tokens encode the original query parameters, and identifiers such as emails or object IDs can identify people. Keep tokens out of logs and metric labels, protect them like the data they point to, and assign an owner for retention and erasure. Your specific legal obligations depend on your contracts and jurisdictions, so confirm them with counsel.

Shipping an integration your customers will trust with their data?

Silent data gaps cost more to debug than to prevent. Pressure-test your design against twelve mandatory controls.

Work with Developers.dev engineers on tenant-safe sync, reconciliation, and verified erasure design.

Contact Us

Related service

API Integration Services

This guide is designed for engineering leaders who want to plan a practical implementation. Use the related Developers.dev path to compare delivery options, implementation fit, risk, and practical next steps.

Read the primary guideRequest a free consultation

Editorial review

Reviewed by the Experts team

This guide is reviewed for clarity, technical and operational relevance, service alignment, and a useful next step. Verified by our SEO team for clear search presentation.

Reviewed byDevelopers.dev Experts Team

Reviewed2026-11-02

FocusAPI Integration Services

SEO verified byDevelopers.dev SEO Team

SEO verified2026-11-02

Reviewed by the Experts team. Verified by our SEO team. Validate legal, security, data, budget, and operational requirements with the relevant stakeholders before rollout.