Repository Sync & Reconciliation: Keeping Identifier State Consistent Across Registries #

Any system that mints and manages persistent identifiers holds two copies of the truth: a local catalog that the deposit pipeline writes to, and the external registry — DataCite, or a repository like Zenodo, Figshare, or Dataverse — that the wider world resolves against. Those two copies drift. A Digital Object Identifier (DOI) gets minted as a draft and the publish call silently fails, so the catalog records it as findable while the registry still holds a draft. A curator tombstones a record upstream through the web console, and nothing tells the local system. A metadata field is edited out of band, and the next automated push overwrites a human correction. Left alone, drift compounds until the catalog is a confident record of a state that no longer exists — a direct threat to the Findable and Accessible guarantees that persistent identifiers are supposed to underwrite. This guide sits inside the Persistent Identifiers & Repository Integration area as its consistency layer: it treats the registry as the system of record for resolution and builds a scheduled reconciliation loop that measures the gap between local and upstream state, emits a minimal patch, and applies it idempotently. The concrete DataCite-specific walkthrough — fetching each DOI’s draft/registered/findable state and computing the patch — lives in reconciling DOI state across repositories; this page is the model and the machinery around it.

The Reconciliation Loop #

Reconciliation is a control loop, not a one-shot migration. On a schedule, it fetches the authoritative upstream state for a set of identifiers, compares that state against the local catalog, decides whether the two agree, and — when they do not — applies a patch that moves one side toward the other. The loop is deliberately boring: it makes no assumptions, it repeats safely, and every pass either converges the two stores or produces an auditable record of an unresolved conflict.

The loop’s value is that every pass is a measurement. If it runs and finds nothing to do, that is a positive assertion that the two stores agree for the identifiers checked — a checkpoint you can put on a dashboard. If it finds drift, it repairs the smallest possible change and records why. Because the patch is the difference between two observed states rather than a blind re-push of everything, a reconciliation run is cheap, targeted, and safe to run far more often than a full re-deposit.

Concept & Specification: State, Drift, and the Diff/Patch Model #

The unit of reconciliation is the identifier’s state: a small, comparable projection of everything that matters for consistency. For a DOI managed through the DataCite Metadata Schema, that projection is the lifecycle state (draft, registered, or findable), the landing-page URL, the metadata payload (or a hash of it), and a modification timestamp. The registry exposes all of these through its REST API; the local catalog stores its own copy plus a synced_at marker recording when it last agreed with upstream. Drift is any observable difference between the local projection and the upstream projection for the same identifier.

Three drift classes recur across every registry and repository:

• Lifecycle drift. The two sides disagree about where the identifier is in its lifecycle. The classic case is a DOI minted as a draft whose publish step failed: the catalog says findable, the registry says draft. Its mirror is an upstream tombstone — a curator retracts a record through the web console, DataCite flips it out of the findable state, and the local catalog never hears about it.
• Metadata drift. The lifecycle agrees but the payloads differ. Someone edited a creator, a title, or a rights statement directly in the repository UI, or an older automated push landed a stale version. A content hash of the normalized metadata detects this in one comparison instead of a field-by-field walk.
• Existence drift. One side has the identifier and the other does not. A DOI is missing upstream (a mint that never committed), or present upstream but absent locally (imported by a colleague, or deleted from the catalog by mistake).

The remediation model is diff then patch. Rather than pushing the entire local record on every run, reconciliation fetches the upstream state, computes the exact set of changes needed to reconcile the two, and emits a patch set — an ordered list of field-level operations. A patch is applied idempotently: applying it once and applying it three times leave the registry in the same state, so a retry after a network timeout is always safe. This is the same idempotency contract the batch layer relies on, described in Async Batch Processing, applied here to registry mutations instead of ingestion writes.

Patches are also event-sourced. Each applied patch is appended to an immutable event log keyed by identifier before the local synced_at marker is advanced. The current local state is a fold over that event stream, which means reconciliation history is fully auditable: you can replay exactly which run changed which field and why, reconstruct the catalog at any past point, and prove to a funder audit that a rights statement was corrected on a specific date. Direction of repair is a policy decision — for a lifecycle failure the local pipeline usually re-drives the upstream transition, whereas for a human edit made upstream the registry is authoritative and the correction is pulled down — and the loop records that decision rather than hard-coding one winner.

Step-by-Step Implementation #

The implementation is four ordered stages: define a comparable state projection, compute the diff, drive the reconcile loop, and apply the patch idempotently while appending to the event log. Every stage is pure and testable in isolation, which is what keeps a job that mutates a public registry trustworthy.

Step 1 — Define a comparable state projection #

Both sides must be reduced to the same shape before they can be compared. A frozen model gives you a normalized, hashable projection where equality is meaningful: two states are equal only when every reconciliation-relevant field matches. Normalizing the metadata into a stable hash means a whitespace-only or key-order difference never registers as spurious drift.

from __future__ import annotations

import hashlib
import json
from enum import Enum

from pydantic import BaseModel, ConfigDict, Field


class DoiState(str, Enum):
    DRAFT = "draft"
    REGISTERED = "registered"
    FINDABLE = "findable"
    TOMBSTONED = "tombstoned"
    ABSENT = "absent"  # not present on this side at all


class IdentifierState(BaseModel):
    """A normalized, comparable projection of one identifier's state."""

    model_config = ConfigDict(frozen=True)

    doi: str
    lifecycle: DoiState
    url: str | None = None
    metadata_hash: str | None = None
    updated: str | None = None  # ISO-8601 timestamp reported by the source

    @staticmethod
    def hash_metadata(metadata: dict[str, object]) -> str:
        """Order-independent hash of the metadata payload for drift detection."""
        canonical = json.dumps(metadata, sort_keys=True, separators=(",", ":"))
        return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

    @classmethod
    def from_record(
        cls, doi: str, lifecycle: DoiState, url: str | None,
        metadata: dict[str, object] | None, updated: str | None,
    ) -> "IdentifierState":
        return cls(
            doi=doi.lower(),
            lifecycle=lifecycle,
            url=url,
            metadata_hash=cls.hash_metadata(metadata) if metadata is not None else None,
            updated=updated,
        )

The ABSENT lifecycle value is what lets existence drift flow through the same comparison as every other kind: a DOI missing upstream is simply an upstream state whose lifecycle is ABSENT, not a special case bolted on outside the diff.

Step 2 — Compute the state diff #

The diff is a pure function of two states. It walks the reconciliation-relevant fields, and for each divergence it emits one patch operation naming the field, the observed local value, the observed upstream value, and the target the patch should move toward. Keeping this function free of I/O is deliberate: it is the part most worth exhaustively unit-testing, and a pure function is trivial to test.

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class PatchOp:
    field: str
    local: object
    upstream: object
    target: object          # value to converge on
    direction: str          # "push" (local -> registry) or "pull" (registry -> local)


def diff_state(
    local: IdentifierState, upstream: IdentifierState, *, upstream_authoritative_fields: set[str],
) -> list[PatchOp]:
    """Return the minimal ordered patch set that reconciles local with upstream."""
    ops: list[PatchOp] = []

    def add(field: str, lv: object, uv: object) -> None:
        pull = field in upstream_authoritative_fields
        ops.append(
            PatchOp(
                field=field, local=lv, upstream=uv,
                target=uv if pull else lv,
                direction="pull" if pull else "push",
            )
        )

    if local.lifecycle != upstream.lifecycle:
        add("lifecycle", local.lifecycle, upstream.lifecycle)
    if local.url != upstream.url:
        add("url", local.url, upstream.url)
    if local.metadata_hash != upstream.metadata_hash:
        add("metadata", local.metadata_hash, upstream.metadata_hash)

    return ops

An empty list is the in-sync signal — no ops means no work. The upstream_authoritative_fields argument is where the direction-of-repair policy lives: put metadata in that set when curators edit records in the repository UI and their edits should win, and lifecycle transitions default to push because the local pipeline owns publishing.

Step 3 — Drive the reconcile loop #

The driver ties fetch, diff, and apply together over a batch of identifiers. It fetches upstream state through a client that must degrade gracefully — a registry outage should downgrade an identifier to “unresolved this pass”, never crash the run — which is exactly the resilience posture set out in API Routing & Fallbacks. Each identifier is reconciled independently so that one poison record cannot stall the batch.

from __future__ import annotations

import logging
from typing import Callable, Protocol

logger = logging.getLogger("pid.reconcile")


class RegistryClient(Protocol):
    def fetch_state(self, doi: str) -> IdentifierState: ...
    def apply_patch(self, doi: str, ops: list[PatchOp]) -> None: ...


class ReconcileReport(BaseModel):
    checked: int = 0
    in_sync: int = 0
    patched: int = 0
    unresolved: list[str] = Field(default_factory=list)


def reconcile_batch(
    dois: list[str],
    load_local: Callable[[str], IdentifierState],
    client: RegistryClient,
    *,
    upstream_authoritative_fields: set[str],
    dry_run: bool = False,
) -> ReconcileReport:
    """Fetch, diff, and (optionally) patch each identifier; never abort mid-batch."""
    report = ReconcileReport()
    for doi in dois:
        report.checked += 1
        try:
            local = load_local(doi)
            upstream = client.fetch_state(doi)
        except Exception as exc:  # transient fetch failure: defer, do not fail the run
            logger.warning("fetch failed for %s: %s", doi, exc)
            report.unresolved.append(doi)
            continue

        ops = diff_state(local, upstream, upstream_authoritative_fields=upstream_authoritative_fields)
        if not ops:
            report.in_sync += 1
            continue

        logger.info("drift on %s: %s", doi, [op.field for op in ops])
        if not dry_run:
            client.apply_patch(doi, ops)
        report.patched += 1
    return report

Running with dry_run=True first is the standard operational move: it produces the full drift report — how many identifiers diverge and on which fields — without mutating a single registry record, so a human can sanity-check the blast radius before a live pass.

Step 4 — Apply the patch idempotently and record the event #

Applying the patch is the only stage that mutates external state, so it carries the idempotency and audit burden. Each patch is appended to the event log keyed by a deterministic event ID derived from the identifier and the observed states, so replaying the same reconciliation twice appends the event once. Only after the registry call and the log append succeed does the local synced_at marker advance.

from __future__ import annotations

import hashlib


def event_id(doi: str, ops: list[PatchOp]) -> str:
    """Deterministic id: same doi + same observed delta => same event."""
    material = doi + "|" + ";".join(
        f"{op.field}:{op.local!r}->{op.upstream!r}" for op in sorted(ops, key=lambda o: o.field)
    )
    return hashlib.sha256(material.encode("utf-8")).hexdigest()[:32]


def apply_and_record(
    doi: str,
    ops: list[PatchOp],
    client: RegistryClient,
    append_event: Callable[[str, str, list[PatchOp]], bool],
    mark_synced: Callable[[str], None],
) -> None:
    """Apply a patch, append exactly one event, then advance the sync marker."""
    eid = event_id(doi, ops)
    client.apply_patch(doi, ops)              # idempotent: re-applying is a no-op upstream
    appended = append_event(eid, doi, ops)    # returns False if this event id already exists
    if appended:
        logger.info("recorded reconciliation event %s for %s", eid, doi)
    mark_synced(doi)

The append_event callback is expected to be a conditional insert keyed on eid (INSERT ... ON CONFLICT DO NOTHING), so a crash between the registry call and the marker update leaves a recoverable state: the next run recomputes the identical event ID, the insert is a no-op, and the marker simply advances.

Reference: Drift Scenarios, Detection Signals, and Remediation #

Every row below is a real divergence a running catalog encounters, the concrete signal that detects it, and the remediation the patch applies. There are no placeholder rows; these are the cases a reconciliation loop is built to close.

Error Handling & Edge Cases #

The dangerous edge case in reconciliation is not a failed fetch — that is just a deferred identifier — but a lost update. Between the moment the job fetches upstream state and the moment it applies its patch, a curator can change the record from the web console. If the patch is applied blindly, it silently clobbers that human edit. The fix is optimistic concurrency: capture the registry’s version token (an ETag, a version, or the updated timestamp) at fetch time and send it back as a precondition on the write. If the token no longer matches, the registry rejects the write, and the job re-fetches, re-diffs against the newer state, and tries again.

Three further edge cases deserve explicit handling. Rate limiting: a full-catalog pass can exceed a registry’s request budget, so the fetch stage must honor Retry-After and back off — the same throttling discipline the Async Batch Processing layer applies to bulk deposits. Partial upstream reads: a registry that returns metadata but omits the lifecycle field should mark the identifier unresolved rather than diffing against a half-populated state and emitting a bogus patch. And clock skew: never treat a raw timestamp comparison as authoritative for direction of repair when the two clocks can differ by seconds — prefer an explicit version token, and use timestamps only as a tie-breaker.

Verification & Testing #

The two properties worth pinning are that the diff is truly minimal (identical states produce an empty patch, and a single-field change produces exactly one op) and that applying a patch is idempotent (a second identical apply records no new event). Because the diff is a pure function, both are fast unit tests that need no live registry.

import pytest


def _state(**kw) -> IdentifierState:
    base = dict(doi="10.1234/abc", lifecycle=DoiState.FINDABLE,
                url="https://ex.org/1", metadata_hash="h1", updated="2026-07-01T00:00:00Z")
    base.update(kw)
    return IdentifierState(**base)


def test_identical_states_produce_empty_patch() -> None:
    ops = diff_state(_state(), _state(), upstream_authoritative_fields=set())
    assert ops == []


def test_lifecycle_drift_produces_one_push_op() -> None:
    local = _state(lifecycle=DoiState.FINDABLE)
    upstream = _state(lifecycle=DoiState.DRAFT)
    ops = diff_state(local, upstream, upstream_authoritative_fields=set())
    assert len(ops) == 1
    assert ops[0].field == "lifecycle" and ops[0].direction == "push"


def test_manual_metadata_edit_pulls_from_upstream() -> None:
    local = _state(metadata_hash="h1")
    upstream = _state(metadata_hash="h2")
    ops = diff_state(local, upstream, upstream_authoritative_fields={"metadata"})
    assert ops[0].direction == "pull" and ops[0].target == "h2"


def test_event_id_is_stable_for_same_delta() -> None:
    local, upstream = _state(lifecycle=DoiState.FINDABLE), _state(lifecycle=DoiState.DRAFT)
    ops = diff_state(local, upstream, upstream_authoritative_fields=set())
    assert event_id("10.1234/abc", ops) == event_id("10.1234/abc", ops)

Run the suite with pytest -q. Wire it into CI so any change to the diff logic or the state projection must keep these invariants green; a reconciliation job that can emit a non-minimal or non-deterministic patch is one that will eventually fight a curator or thrash a registry.

Gotchas & Known Pitfalls #

• Treating the local catalog as the system of record for resolution. The registry is what the world resolves against; if reconciliation always pushes local over upstream, it will happily overwrite legitimate curator edits. Root cause: one hard-coded winner. Fix: make direction per-field policy via upstream_authoritative_fields, and default manual-edit-prone fields to pull.
• Diffing raw metadata instead of a normalized hash. Key ordering, whitespace, and null-vs-absent differences trigger endless false-positive drift and a patch that changes nothing. Root cause: comparing serialized bytes. Fix: hash a canonical, sorted, separator-normalized form, as hash_metadata does.
• Non-idempotent patch application. Re-running after a timeout double-applies transitions or appends duplicate events. Root cause: auto-increment event keys and unconditional writes. Fix: derive a deterministic event ID from the observed delta and insert it conditionally.
• Blind writes that lose concurrent updates. Applying a patch without the version token captured at fetch time silently clobbers an edit made mid-run. Root cause: no optimistic concurrency. Fix: send the ETag/version as a write precondition and re-diff on rejection.
• Reconciling the whole catalog in one unbounded pass. A single job over hundreds of thousands of DOIs blows the registry’s rate budget and never finishes. Root cause: no batching or backpressure. Fix: page the identifier set and throttle fetches, honoring Retry-After.

Frequently Asked Questions #

How often should the reconciliation job run? #

Match the cadence to how the drift arises. Lifecycle failures from your own pipeline are best caught within minutes by a post-deposit verification pass, so a short-interval job scoped to recently touched identifiers pays off immediately. Out-of-band curator edits and upstream tombstones accrue slowly, so a full-catalog sweep nightly or weekly is usually enough. Running the same loop at two cadences over two scopes — hot recent identifiers frequently, the whole catalog occasionally — is the common production pattern.

Should reconciliation ever delete a local record? #

Almost never. If an identifier is present upstream but absent locally, import it or flag it, but treat deletion as a human decision. Persistent identifiers are a promise of permanence, so the safe default when the two sides disagree about existence is to preserve and surface the record for review, not to silently drop it. A tombstone — retaining the identifier and metadata while marking it withdrawn — is the correct terminal state, never a hard delete.

How is this different from just re-pushing all local metadata on a schedule? #

A blind re-push is a write amplifier: it hammers the registry with unchanged records, races against curator edits, and gives you no signal about what actually diverged. The diff/patch loop touches only genuinely drifted fields, produces an auditable report of every change, and — crucially — can run in dry_run mode to measure drift without mutating anything. You get consistency and observability instead of consistency you cannot see.

Where does the DOI-minting side fit relative to reconciliation? #

Minting creates the identifier and its first upstream state; reconciliation keeps that state honest over the identifier’s lifetime. The two are complementary halves of the same lifecycle, and the minting mechanics — draft creation, metadata registration, and the publish transition — are covered in DOI Minting with DataCite. Reconciliation assumes those operations exist and re-drives them when it detects that one never completed.

Related Guides #

• Persistent Identifiers & Repository Integration — the parent overview that positions reconciliation as the consistency layer over minting and deposit.
• Reconciling DOI State Across Repositories — the concrete DataCite walkthrough that fetches draft/registered/findable state and computes the patch.
• DOI Minting with DataCite — how the upstream state reconciliation checks is first created.
• API Routing & Fallbacks — the graceful-degradation posture the fetch stage depends on when a registry is unreachable.