Idempotent Retry Patterns for Batch Deposits #

Any retry policy on top of a network is, whether you designed it that way or not, an at-least-once delivery system: a request can succeed on the server and still time out before the client sees the response, so the client retries and the operation runs twice. For a batch that deposits datasets or mints identifiers, “twice” is not a performance detail — it is two records where there should be one, or two DOIs for the same object, an irreversible pollution of a persistent-identifier namespace. This page is the safety layer that makes the retry loops elsewhere in the Async Batch Processing overview safe to run: given that a deposit will sometimes be attempted more than once, how do you guarantee it takes effect exactly once? It assumes you already retry failed deposits and are comfortable with Python 3.10+; the goal is to make every write idempotent so a replay is provably a no-op.

The core move is to stop thinking of a deposit as “create a record” and start thinking of it as “ensure a record exists for this key.” An idempotency key is a stable, deterministic name for the intended effect of an operation, derived from the content rather than assigned by the server. Derive it from a hash of the payload — the bytes of the dataset plus the canonical metadata — and the same input always produces the same key no matter how many times the batch runs. Pair that key with a conditional write (If-None-Match: * on an HTTP PUT, or an INSERT ... ON CONFLICT DO NOTHING in a database) and the server itself refuses the second write, collapsing at-least-once delivery into exactly-once effect. A small dedup store records which keys have already produced which result, so a replay can return the original outcome without touching the repository at all. This is the same content-addressing discipline the FAIR Principle Breakdown relies on for the findability and reusability guarantees a research archive must uphold.

The Failure-to-Mechanism Reference Table #

Not every failure should be retried, and not every retry needs the same idempotency mechanism. A 429 is retryable and safe to replay because the operation never happened; a network timeout after a POST is retryable but dangerous, because the write may already have landed — that ambiguous case is exactly where idempotency keys earn their keep. A 412 Precondition Failed from a conditional write is not a failure at all: it is the server confirming the record already exists, which is a success from the caller’s point of view. The table below maps each failure class to whether it is retryable and which mechanism makes that retry safe.

The two rows that separate a correct system from a broken one are the ambiguous ones: a timeout or 5xx after the request was sent. Only a deterministic key plus a conditional write turns those into safe replays; without them, every such retry is a coin flip on whether you just created a duplicate.

Production Implementation #

The implementation has two parts: a dedup store keyed by content hash that records the result of each completed deposit, and a guarded deposit function that consults the store, issues a conditional write, and treats a precondition failure as a successful de-duplication. The idempotency key is a SHA-256 over the canonicalized payload, so the key is stable across processes and machines — a retry from a different worker computes the identical key. Because the same guarantee underpins how identifiers are minted, this pattern is the prerequisite for the DOI Minting with DataCite workflows, where a double-mint is unrecoverable.

from __future__ import annotations

import hashlib
import json
import logging
import sqlite3
from dataclasses import dataclass
from typing import Protocol

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("fair.idempotency")


def idempotency_key(payload: dict[str, object]) -> str:
    """Derive a stable key from canonicalized content. Same input -> same key."""
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")
    return hashlib.sha256(canonical).hexdigest()


@dataclass(frozen=True)
class DepositResult:
    """The outcome of a deposit, recorded so a replay can return it verbatim."""

    key: str
    identifier: str
    status: str  # "created" or "existing"


class IdempotencyStore:
    """A durable key -> result map. Backed by SQLite for crash-safe dedup."""

    def __init__(self, db_path: str = "idempotency.db") -> None:
        self._conn = sqlite3.connect(db_path, isolation_level=None)  # autocommit
        self._conn.execute("PRAGMA journal_mode=WAL")
        self._conn.execute(
            "CREATE TABLE IF NOT EXISTS deposits ("
            "  key TEXT PRIMARY KEY,"
            "  identifier TEXT NOT NULL,"
            "  status TEXT NOT NULL)"
        )

    def get(self, key: str) -> DepositResult | None:
        row = self._conn.execute(
            "SELECT key, identifier, status FROM deposits WHERE key = ?", (key,)
        ).fetchone()
        return DepositResult(*row) if row else None

    def put(self, result: DepositResult) -> bool:
        """Record a result. Returns False if the key already existed (a race)."""
        cur = self._conn.execute(
            "INSERT OR IGNORE INTO deposits (key, identifier, status) VALUES (?, ?, ?)",
            (result.key, result.identifier, result.status),
        )
        return cur.rowcount == 1


class DepositClient(Protocol):
    """The minimal contract a repository client must satisfy for guarded deposits."""

    def conditional_put(self, key: str, payload: dict[str, object]) -> tuple[int, str]:
        """PUT with If-None-Match: *. Returns (status_code, identifier)."""
        ...


class NonRetryableError(RuntimeError):
    """A terminal error (validation failure) that must never be replayed."""


def guarded_deposit(
    client: DepositClient, store: IdempotencyStore, payload: dict[str, object]
) -> DepositResult:
    """Deposit exactly once. Safe to call repeatedly with the same payload."""
    key = idempotency_key(payload)

    # 1. Fast path: this exact content already completed in a prior run.
    prior = store.get(key)
    if prior is not None:
        logger.info("replay: key %s already deposited as %s", key[:12], prior.identifier)
        return prior

    # 2. Conditional write. The server, not the client, arbitrates duplicates.
    status, identifier = client.conditional_put(key, payload)
    if status in (200, 201):
        result = DepositResult(key=key, identifier=identifier, status="created")
    elif status in (409, 412):
        # Precondition/conflict: the record already exists server-side.
        result = DepositResult(key=key, identifier=identifier, status="existing")
    elif status in (400, 422):
        raise NonRetryableError(f"validation error {status} for key {key[:12]}")
    else:
        raise RuntimeError(f"retryable status {status} for key {key[:12]}")

    # 3. Record the outcome so future retries short-circuit at step 1.
    store.put(result)
    return result

Verification #

Correctness here means one property above all: calling guarded_deposit twice with the same payload produces one server write and returns the same identifier both times. The fake client below counts how many times it actually performs a create and returns a 412 on any repeat of a key it has already seen — modeling a server enforcing If-None-Match. The test replays the deposit and asserts the create count stays at one.

import pytest


class FakeRepo:
    """A server that enforces If-None-Match: the second PUT of a key gets 412."""

    def __init__(self) -> None:
        self.seen: dict[str, str] = {}
        self.creates = 0

    def conditional_put(self, key: str, payload: dict[str, object]) -> tuple[int, str]:
        if key in self.seen:
            return 412, self.seen[key]
        self.creates += 1
        identifier = f"10.5072/{key[:8]}"
        self.seen[key] = identifier
        return 201, identifier


def test_replay_does_not_double_deposit() -> None:
    repo = FakeRepo()
    store = IdempotencyStore(":memory:")
    payload = {"title": "Reef Temperature Series", "creators": ["Vega, A."]}

    first = guarded_deposit(repo, store, payload)
    second = guarded_deposit(repo, store, payload)  # replay

    assert first.identifier == second.identifier
    assert repo.creates == 1          # the server created exactly one record
    assert second.status == "existing" or first.identifier == second.identifier


def test_validation_error_is_terminal() -> None:
    class Bad:
        def conditional_put(self, key: str, payload: dict[str, object]) -> tuple[int, str]:
            return 422, ""

    with pytest.raises(NonRetryableError):
        guarded_deposit(Bad(), IdempotencyStore(":memory:"), {"title": ""})

Run it with pytest -q test_idempotency.py. The repo.creates == 1 assertion is the whole point: even though the deposit was invoked twice, the repository was written once. The second test confirms a 422 is never retried, because no amount of replay fixes malformed metadata.

Gotchas #

• A non-deterministic key defeats the whole scheme. Hashing a payload that includes a wall-clock timestamp, a random request ID, or unsorted dict keys yields a different key on every attempt, so the dedup store never hits and every retry double-deposits. Fix: canonicalize before hashing (sort_keys=True, fixed separators) and exclude any field that varies per attempt.
• Recording the result before the write is confirmed creates a phantom. If you put the key into the dedup store and the deposit then fails, every future replay short-circuits to a record that does not exist server-side. Fix: store the outcome only after the conditional write returns a success or precondition status, exactly as the guarded_deposit ordering shows.
• Treating 412/409 as an error re-queues a completed deposit forever. A conditional write that returns “already exists” is the success case of idempotency, not a failure; classifying it as retryable turns a finished record into a poison pill. Fix: map precondition-failed and conflict statuses to an existing result and return it.

Frequently Asked Questions #

Where should the idempotency key come from — the client or the server? #

The client, derived deterministically from content. If the server assigns the key, a client that never sees the response (the timeout case) cannot reconstruct it to retry safely, which is the exact scenario idempotency is meant to survive. A SHA-256 over the canonicalized payload gives every worker, in every run, the same key for the same content, so the conditional write and the dedup store both key on a value the client can always recompute.

Do I still need a dedup store if the server enforces If-None-Match? #

They cover different gaps. The server’s conditional write prevents a duplicate record even under concurrent retries, which is the durable guarantee. The dedup store is a client-side short-circuit that lets a replay return the prior result without a network round-trip, and it preserves the original outcome — the minted identifier — for callers that need it. Use the conditional write for correctness and the dedup store for efficiency and result recall; together they make replay both safe and cheap.

Related Guides #

• Async Batch Processing — the parent overview whose retry and dead-letter loops this idempotency layer makes safe to run.
• DOI Minting with DataCite — where an un-guarded retry can double-mint a DOI, making exactly-once effect non-negotiable.
• FAIR Principle Breakdown — the findability and reusability guarantees that content-addressed, deduplicated deposits uphold.