FAIR Data Automation Research Data Management

Batch Updating DataCite Metadata for Existing DOIs

Metadata is never finished. A funder reference gets added after an award lands, a landing-page URL changes when a repository migrates, a typo in a creator name is caught during an audit — and each correction has to be applied across not one DOI but hundreds or thousands of already-registered ones. This page builds a client that updates existing DataCite DOIs in bulk: a bounded-concurrency loop of PUT /dois/{id} requests that respects the account’s rate limit, isolates a per-record failure so one bad DOI does not sink the run, collects unrecoverable failures into a dead-letter list, and is safe to re-run from the top. It extends the single-record work in registering DOIs with the DataCite REST API to the scale problem, and it assumes the lifecycle and JSON:API contract set out in the DOI Minting with DataCite overview.

The prerequisites are the same DataCite Fabrica test account and prefix, Python 3.10+, and httpx with async support. The one new constraint that dominates the design is the rate limit: at batch scale you will hit HTTP 429, and the only correct response is to honour the Retry-After header rather than hammer the endpoint. The general concurrency and backpressure machinery here is the DataCite-specific instance of the patterns in the async batch processing guide.

Bounded-concurrency DataCite batch update loop A DOI worklist feeds a semaphore gate that caps concurrency, which admits each DOI to a PUT request. A decision node checks for a 2xx response. Successful updates are recorded. A 429 response routes back through the semaphore after honouring Retry-After. Permanent failures such as 422 or 404 are collected in a dead-letter list. DOI worklist many DOIs Semaphore gate limit = 8 PUT /dois/{id} update attributes 2xx? yes Updated record success 422 / 404 Dead-letter list isolate failure 429 · Retry-After

Update Scenarios and Their Outcomes

Every batch update is a PUT /dois/{id} carrying a partial or full attributes object; DataCite merges the supplied properties into the existing record. Omitting event leaves the DOI in its current state, so updating a findable DOI keeps it findable. The table maps the common update scenarios to the field written and the outcome the client must handle.

Update scenario Verb · field Outcome
Correct a creator name PUT · creators 200 — record merged, state unchanged
Change the landing-page URL PUT · url 200 — resolution updated
Add a funding reference PUT · fundingReferences 200 — property appended
Promote drafts in bulk PUT · event: publish 200 findable, or 422 if incomplete
Invalid metadata supplied PUT · any 422 — permanent, route to dead-letter
Unknown or mistyped DOI PUT /dois/{id} 404 — permanent, route to dead-letter
Rate limit exceeded PUT · any 429 + Retry-After — back off and retry
Registry or gateway fault PUT · any 5xx — transient, retry with backoff

The split that governs the whole client is permanent versus transient. A 422 or 404 is a property of the request itself and will never succeed on retry, so it is isolated to the dead-letter list. A 429, 5xx, or timeout is a property of the moment, so it is retried. Never let the two categories share a code path.

Concurrency and the rate limit are two separate dials that interact. Concurrency — the semaphore’s ceiling — governs how many requests are in flight at once; the rate limit governs how many requests DataCite will accept per unit time regardless of how you spread them. Setting concurrency modestly (a handful rather than dozens) keeps a batch comfortably under the limit and turns 429s from the common case into a rare exception the Retry-After path handles gracefully. Cranking concurrency high to finish faster is a false economy: the endpoint simply returns 429 more often, every one of those requests sleeps and retries, and the run ends up slower and noisier than a gentler one that never trips the limit.

Production Implementation

The client below is complete and runnable. An asyncio.Semaphore caps in-flight requests so the batch never exceeds the account’s concurrency budget; each DOI is updated inside its own task with try/except isolation; transient faults retry with exponential backoff and Retry-After; and permanent failures land in a dead-letter list with the reason attached. Because each update is an idempotent PUT that sets the same attributes regardless of how many times it runs, re-running the whole batch after a crash converges on the same end state and safely re-attempts only what the dead-letter list reports.

python
from __future__ import annotations

import asyncio
from dataclasses import dataclass, field
from typing import Any

import httpx

JSONAPI = "application/vnd.api+json"
TEST_BASE = "https://api.test.datacite.org"   # Fabrica test; prod: api.datacite.org
_PERMANENT = {400, 404, 409, 422}
_TRANSIENT = {429, 500, 502, 503, 504}


@dataclass(frozen=True)
class UpdateJob:
    doi: str
    attributes: dict[str, Any]   # partial DataCite attributes to merge


@dataclass
class BatchResult:
    updated: list[str] = field(default_factory=list)
    dead_letter: list[tuple[str, str]] = field(default_factory=list)  # (doi, reason)


class DataCiteBatchUpdater:
    def __init__(
        self,
        client_id: str,
        password: str,
        base_url: str = TEST_BASE,
        concurrency: int = 8,
        max_retries: int = 5,
        base_delay: float = 1.0,
    ) -> None:
        self._auth = (client_id, password)
        self._base = base_url.rstrip("/")
        self._sem = asyncio.Semaphore(concurrency)
        self._max_retries = max_retries
        self._base_delay = base_delay

    def _body(self, job: UpdateJob) -> dict[str, Any]:
        return {"data": {"type": "dois", "id": job.doi, "attributes": job.attributes}}

    async def _update_one(
        self, http: httpx.AsyncClient, job: UpdateJob, result: BatchResult
    ) -> None:
        """Update a single DOI with retry, isolating its failure from the batch."""
        async with self._sem:
            for attempt in range(self._max_retries + 1):
                try:
                    resp = await http.put(
                        f"{self._base}/dois/{job.doi}", json=self._body(job), auth=self._auth
                    )
                except (httpx.TransportError, httpx.TimeoutException) as exc:
                    if attempt == self._max_retries:
                        result.dead_letter.append((job.doi, f"transport: {exc!r}"))
                        return
                    await asyncio.sleep(self._base_delay * (2 ** attempt))
                    continue

                if resp.status_code < 300:
                    result.updated.append(job.doi)
                    return
                if resp.status_code in _PERMANENT:
                    result.dead_letter.append((job.doi, f"HTTP {resp.status_code}: {resp.text[:200]}"))
                    return
                if resp.status_code in _TRANSIENT and attempt < self._max_retries:
                    delay = float(resp.headers.get("Retry-After", self._base_delay * (2 ** attempt)))
                    await asyncio.sleep(delay)
                    continue

                result.dead_letter.append((job.doi, f"HTTP {resp.status_code} after retries"))
                return

    async def run(self, jobs: list[UpdateJob]) -> BatchResult:
        """Update every DOI under a bounded-concurrency limit; return a partitioned result."""
        result = BatchResult()
        async with httpx.AsyncClient(
            timeout=30.0, headers={"Content-Type": JSONAPI, "Accept": JSONAPI}
        ) as http:
            await asyncio.gather(*(self._update_one(http, job, result) for job in jobs))
        return result


async def _main() -> None:
    jobs = [
        UpdateJob(doi="10.5072/reef-2026-0001", attributes={"url": "https://data.example.org/reef/1"}),
        UpdateJob(doi="10.5072/reef-2026-0002", attributes={"publisher": "Marine Data Archive"}),
    ]
    updater = DataCiteBatchUpdater("YOUR.CLIENT", "your-password")
    result = await updater.run(jobs)
    print(f"updated={len(result.updated)} dead_letter={len(result.dead_letter)}")
    for doi, reason in result.dead_letter:
        print(f"  FAILED {doi}: {reason}")


if __name__ == "__main__":
    asyncio.run(_main())

Passing the shared result into each task and appending to plain lists is safe here because asyncio tasks run cooperatively on one thread — a list append never interleaves mid-operation, so no lock is needed. The dead-letter list is the run’s durable output: persist it to disk and feed it back in as the next run’s worklist to retry only what failed.

Verification

Verify the classification logic without a live account by stubbing the HTTP layer, then run a small real batch against Fabrica. The test below drives one success, one permanent failure, and one rate-limited-then-successful request through a mock transport and asserts the batch partitions them correctly.

python
import asyncio

import httpx
import pytest


@pytest.mark.asyncio
async def test_batch_partitions_success_and_permanent_failure() -> None:
    def handler(request: httpx.Request) -> httpx.Response:
        if "0001" in request.url.path:
            return httpx.Response(200, json={"data": {"id": "10.5072/x-0001"}})
        if "0002" in request.url.path:
            return httpx.Response(422, json={"errors": [{"title": "missing titles"}]})
        return httpx.Response(429, headers={"Retry-After": "0"})

    updater = DataCiteBatchUpdater("id", "pw", concurrency=4, max_retries=1)
    transport = httpx.MockTransport(handler)

    async with httpx.AsyncClient(transport=transport) as http:
        result = BatchResult()
        await asyncio.gather(
            updater._update_one(http, UpdateJob("10.5072/x-0001", {"url": "https://a"}), result),
            updater._update_one(http, UpdateJob("10.5072/x-0002", {"url": "https://b"}), result),
        )

    assert result.updated == ["10.5072/x-0001"]
    assert result.dead_letter[0][0] == "10.5072/x-0002"
    assert "422" in result.dead_letter[0][1]

Run with pytest -q. For a live check, update two real test DOIs, confirm result.updated holds both, then GET one back and assert the changed field took effect. Re-running the identical batch must report the same successes and an empty dead-letter list — the machine-readable proof the operation is idempotent.

Gotchas

  • Retrying a 429 without honouring Retry-After. Immediately re-firing a rate-limited request deepens the limit and can escalate to a temporary block. Fix: read the Retry-After header and sleep for exactly that duration before the next attempt, as _update_one does, and cap concurrency so the limit is rarely reached at all.
  • One bad DOI aborting the whole run. A bare asyncio.gather without per-task isolation propagates the first exception and cancels every sibling, losing all in-flight progress. Fix: catch inside each task and route the failure to the dead-letter list so the batch always runs to completion.
  • Sending event: publish on an update you meant to keep quiet. Including event on a PUT transitions the DOI’s state; a stray publish on a registered DOI makes it findable unexpectedly. Fix: omit event entirely for metadata-only edits so the state is preserved.