Depositing Crossref Metadata for Datasets

This is the concrete task behind the broader Crossref Metadata Deposit walkthrough: you have a dataset that supplements a published paper, and you need Crossref’s relation graph to record that link. The deliverable is a single deposit XML document — a dataset record whose relations program asserts an isSupplementTo relationship to the parent publication’s DOI — POSTed to the deposit endpoint with credentials, followed through its submission lifecycle by polling, and recovered cleanly when a record comes back as a failure. It assumes Python 3.10+, an authorized Crossref account, and that you have already read the deposit model in the overview. The reverse direction — minting the dataset’s own DOI at DataCite and asserting the relation from the article side — is covered in DOI Minting with DataCite; many workflows do both so the link resolves whichever identifier a reader starts from.

The failure this prevents is a dangling data citation. When a paper references its underlying dataset only in prose, no machine can traverse from the article to the data or back. A deposited isSupplementTo relation makes the edge first-class: discovery services, the DataCite–Crossref event pipeline, and citation trackers can all follow it. The work is entirely in getting the XML shape and the parent-DOI reference exactly right, then handling the asynchronous result honestly.

Deposit Element Reference

The document is a standard <doi_batch> envelope wrapping one <database> record with a single <dataset> inside it. The table below is the exact set of elements that carry the dataset identity and its link to the parent publication; the relation lives in the http://www.crossref.org/relations.xsd namespace (the rel prefix), everything else in the http://www.crossref.org/schema/5.3.1 deposit namespace.

Element	Parent / namespace	Value in this deposit	Purpose
`doi_batch_id`	`head` · deposit	your submission key	The identifier you poll the batch on
`timestamp`	`head` · deposit	current clock	Breaks ties against prior deposits of the same DOI
`database`	`body` · deposit	container	Top-level record type for data citations
`dataset`	`database` · deposit	`dataset_type="record"`	The dataset record itself
`titles/title`	`dataset` · deposit	dataset title	Human-readable name of the dataset
`rel:program`	`dataset` · relations	container	Holds the relation assertions
`rel:related_item`	`rel:program` · relations	one item	The parent publication being linked
`rel:inter_work_relation`	`rel:related_item` · relations	`relationship-type="isSupplementTo"` `identifier-type="doi"`	The typed edge to the parent DOI
`doi`	`doi_data` · deposit	the dataset DOI	Identifier being registered for the dataset
`resource`	`doi_data` · deposit	landing-page URL	Where the dataset DOI resolves

Production Implementation

The script below is complete and runnable end to end: it builds the XML from a typed input, submits it, polls with capped backoff, and returns a structured result that distinguishes a clean registration from a per-record failure. Credentials and the endpoint come from the environment so nothing secret is hardcoded, and the endpoint defaults to the Crossref sandbox for safe rehearsal.

python

from __future__ import annotations

import os
import time
import xml.etree.ElementTree as ET
from dataclasses import dataclass
from datetime import datetime, timezone

import httpx

CR_NS = "http://www.crossref.org/schema/5.3.1"
REL_NS = "http://www.crossref.org/relations.xsd"
ET.register_namespace("", CR_NS)
ET.register_namespace("rel", REL_NS)

DEPOSIT_URL = os.environ.get("CROSSREF_DEPOSIT_URL", "https://test.crossref.org/servlet/deposit")
STATUS_URL = os.environ.get(
    "CROSSREF_STATUS_URL", "https://test.crossref.org/servlet/submissionDownload"
)


class CrossrefSubmitError(RuntimeError):
    """Raised when the endpoint refuses to queue the batch (synchronous failure)."""


@dataclass(frozen=True)
class DatasetDeposit:
    batch_id: str
    depositor_name: str
    depositor_email: str
    registrant: str
    dataset_title: str
    dataset_doi: str
    resource_url: str
    parent_publication_doi: str

    def __post_init__(self) -> None:
        for doi in (self.dataset_doi, self.parent_publication_doi):
            if not doi.startswith("10."):
                raise ValueError(f"Not a DOI: {doi!r}")


@dataclass(frozen=True)
class BatchResult:
    status: str                 # "completed"
    failures: tuple[str, ...]   # per-record Failure messages; empty means success


def _q(tag: str) -> str:
    return f"{{{CR_NS}}}{tag}"


def _r(tag: str) -> str:
    return f"{{{REL_NS}}}{tag}"


def build_dataset_xml(dep: DatasetDeposit) -> bytes:
    """Serialize a dataset record with an isSupplementTo relation to its parent DOI."""
    batch = ET.Element(_q("doi_batch"), {"version": "5.3.1"})

    head = ET.SubElement(batch, _q("head"))
    ET.SubElement(head, _q("doi_batch_id")).text = dep.batch_id
    ET.SubElement(head, _q("timestamp")).text = datetime.now(timezone.utc).strftime("%Y%m%d%H%M%S")
    depositor = ET.SubElement(head, _q("depositor"))
    ET.SubElement(depositor, _q("depositor_name")).text = dep.depositor_name
    ET.SubElement(depositor, _q("email_address")).text = dep.depositor_email
    ET.SubElement(head, _q("registrant")).text = dep.registrant

    body = ET.SubElement(batch, _q("body"))
    database = ET.SubElement(body, _q("database"))
    dataset = ET.SubElement(database, _q("dataset"), {"dataset_type": "record"})
    titles = ET.SubElement(dataset, _q("titles"))
    ET.SubElement(titles, _q("title")).text = dep.dataset_title

    program = ET.SubElement(dataset, _r("program"))
    item = ET.SubElement(program, _r("related_item"))
    iwr = ET.SubElement(
        item,
        _r("inter_work_relation"),
        {"relationship-type": "isSupplementTo", "identifier-type": "doi"},
    )
    iwr.text = dep.parent_publication_doi

    doi_data = ET.SubElement(dataset, _q("doi_data"))
    ET.SubElement(doi_data, _q("doi")).text = dep.dataset_doi
    ET.SubElement(doi_data, _q("resource")).text = dep.resource_url

    return ET.tostring(batch, encoding="utf-8", xml_declaration=True)


def submit(xml_bytes: bytes, batch_id: str) -> str:
    """POST the deposit; return the batch_id, or raise on a synchronous refusal."""
    data = {
        "operation": "doMDUpload",
        "login_id": os.environ["CROSSREF_LOGIN_ID"],
        "login_passwd": os.environ["CROSSREF_LOGIN_PASSWD"],
    }
    files = {"fname": (f"{batch_id}.xml", xml_bytes, "application/xml")}
    with httpx.Client(timeout=30.0) as client:
        resp = client.post(DEPOSIT_URL, data=data, files=files)
    if resp.status_code >= 400 or "error" in resp.text.lower():
        raise CrossrefSubmitError(f"Not queued ({resp.status_code}): {resp.text[:200]}")
    return batch_id


def _parse(xml_text: str) -> BatchResult | None:
    """None while queued/in_process; a BatchResult once the batch is terminal."""
    root = ET.fromstring(xml_text)
    status = root.attrib.get("status", "unknown")
    if status in {"queued", "in_process"}:
        return None
    failures = tuple(
        (msg.text or "unknown failure")
        for rec in root.iter("record_diagnostic")
        if rec.attrib.get("status") == "Failure"
        for msg in rec.iter("msg")
    )
    return BatchResult(status=status, failures=failures)


def poll(batch_id: str, *, max_wait: float = 300.0) -> BatchResult:
    """Poll the submission with capped backoff until it is terminal."""
    params = {
        "usr": os.environ["CROSSREF_LOGIN_ID"],
        "pwd": os.environ["CROSSREF_LOGIN_PASSWD"],
        "file_name": batch_id,
        "type": "result",
    }
    delay, waited = 5.0, 0.0
    with httpx.Client(timeout=30.0) as client:
        while waited < max_wait:
            resp = client.get(STATUS_URL, params=params)
            resp.raise_for_status()
            result = _parse(resp.text)
            if result is not None:
                return result
            time.sleep(delay)
            waited += delay
            delay = min(delay * 2, 60.0)
    raise TimeoutError(f"Batch {batch_id} still not terminal after {max_wait}s")


def deposit_dataset(dep: DatasetDeposit) -> BatchResult:
    """Full path: build, submit, poll, and surface record-level failures."""
    xml_bytes = build_dataset_xml(dep)
    batch_id = submit(xml_bytes, dep.batch_id)
    result = poll(batch_id)
    if result.failures:
        # Record-level failure inside a completed batch: do NOT blindly resubmit.
        raise CrossrefSubmitError("; ".join(result.failures))
    return result

The critical design choice is that deposit_dataset distinguishes two failure modes. A CrossrefSubmitError from submit is synchronous — bad credentials, malformed XML, an unauthorized prefix — and is terminal until you fix the cause. A CrossrefSubmitError raised after poll carries the record diagnostic from a completed batch: the batch processed, but the record was rejected, most often because the parent DOI has a higher-timestamp deposit or the dataset DOI collides. That distinction is what tells an operator whether to fix-and-resubmit or to reconcile a single record.

Verification

Exercise the build and parse steps offline — no credentials, no network — so the two most error-prone parts are pinned in CI. Assert that the relation serializes with the right type and target, and that the parser refuses to call a queued batch finished.

python

def test_relation_targets_parent_publication() -> None:
    dep = DatasetDeposit(
        batch_id="ds-batch-01",
        depositor_name="Data Team",
        depositor_email="deposit@example.org",
        registrant="Example University",
        dataset_title="Reef Temperature Time Series 2024",
        dataset_doi="10.5555/dataset.reef.2024",
        resource_url="https://example.org/datasets/reef-2024",
        parent_publication_doi="10.5555/article.reef.2024",
    )
    xml_text = build_dataset_xml(dep).decode("utf-8")
    assert 'relationship-type="isSupplementTo"' in xml_text
    assert "10.5555/article.reef.2024" in xml_text
    assert 'dataset_type="record"' in xml_text


def test_queued_batch_is_not_terminal() -> None:
    assert _parse('<doi_batch_diagnostic status="in_process"/>') is None


def test_completed_with_failure_is_surfaced() -> None:
    doc = (
        '<doi_batch_diagnostic status="completed">'
        '<record_diagnostic status="Failure"><msg>Relation target not found</msg>'
        "</record_diagnostic></doi_batch_diagnostic>"
    )
    result = _parse(doc)
    assert result is not None and result.failures == ("Relation target not found",)

Run it with pytest -q. A passing run confirms the deposit points at the correct parent DOI and that a still-processing batch will never be mistaken for a completed one — the two mistakes that turn a data-citation deposit into a silent no-op.

Gotchas

A relation to a non-existent parent DOI fails at the record level, not at submit. If the parent publication’s DOI is not yet registered, the batch still reaches completed, but the record carries a Failure. Fix: register or confirm the parent DOI first, and treat poll failures as reconcilable per record rather than resubmitting the whole batch.
Reusing a doi_batch_id or timestamp overwrites the wrong thing. Crossref keeps the highest-timestamp deposit for a DOI, so a reused or stale timestamp can make a correction vanish. Fix: generate the timestamp from the clock on every build, as build_dataset_xml does, and use a fresh batch id per submission.
Depositing the dataset here duplicates a DataCite identifier if you also mint one there. A dataset should carry exactly one DOI. Fix: pick one registration agency for the dataset DOI — if it lives at DataCite, assert the relation from the article side per DOI Minting with DataCite rather than minting a second identifier here.

Crossref Metadata Deposit — the full deposit model, endpoint semantics, and submission lifecycle this task specializes.
DOI Minting with DataCite — the reciprocal path for minting the dataset’s own DOI and asserting the relation from the data side.

Depositing Crossref Metadata for Datasets #

Deposit Element Reference #

Production Implementation #

Verification #

Gotchas #

Related Guides #

Depositing Crossref Metadata for Datasets

Deposit Element Reference

Production Implementation

Verification

Gotchas

Related Guides