Persistent Identifiers & Repository Integration for Scientific Research Data #

A dataset that has been validated at the ingestion gate and written to immutable storage is durable, but it is not yet citable. Nothing points at it, nothing resolves to it, and no external index knows it exists. Turning that private archived object into a public record of science is a distinct engineering problem: a persistent identifier has to be minted and bound to a stable landing page, the bytes and their metadata have to be deposited into one or more discipline repositories, the identifier has to be registered with the citation infrastructure that journals and funders query, and — because none of those systems share a transaction — the state scattered across them has to be continuously reconciled back into agreement. When any of these steps is done by hand, the symptoms are familiar: DOIs that resolve to a 404, a Zenodo record whose metadata contradicts the DataCite record for the same identifier, a dataset the funder’s reporting portal cannot find because it was never registered where the funder looks. This overview treats minting, deposit, registration, and reconciliation as one pipeline with explicit contracts between stages, and it links to the detailed implementation guides for each. It is written for the Python automation engineers and research data managers who own the layer that sits directly downstream of Data Ingestion & Metadata Enrichment and directly on top of the topology described in Core Architecture & FAIR Mapping.

Architecture Overview #

The identifier-and-deposit layer has a consistent shape regardless of which registries and repositories a given institution uses. A validated dataset enters, a draft identifier is minted so a citable string exists before anything is public, the object and its metadata are deposited into one or more repositories, the identifier is published to its findable state and begins resolving, and a background reconciliation loop compares the state held in each external system against the platform’s own record of truth and repairs any drift. The diagram below shows that pipeline and its feedback edge.

Each arrow is a contract, not a mere sequence. The edge into the mint stage carries a durably archived object and a verified checksum, because the platform will not mint against bytes it cannot prove exist; the edge into deposit carries a draft identifier that is safe to abandon; the edge into publish is the point of no return, because a findable DOI cannot be deleted; and the feedback edge carries a diff between what the platform recorded and what each external registry actually holds. Reading the pipeline this way makes the ordering non-negotiable: mint before deposit so the landing page can embed its own identifier, deposit before publish so the identifier never resolves to nothing, and reconcile forever because the external systems will drift no matter how careful the write path is.

Identifier Minting with the DataCite Metadata Schema #

For research data the dominant identifier is the DOI, and for datasets the DOI is almost always registered through DataCite rather than Crossref. Minting is governed by two standards working together. The DataCite Metadata Schema defines the fields a DOI record must carry — mandatory identifier, creators, titles, publisher, publicationYear, and resourceType, plus a deep set of recommended properties for related identifiers, funding references, and geolocation. The DataCite REST API defines how those fields are submitted, using a JSON:API envelope in which a DOI is a resource of type dois. The mechanics of registering, updating, and versioning DOIs against that API — including how each schema field maps onto a JSON:API attribute — are worked through in the DOI Minting with DataCite walkthrough.

The single most important property of a correct minting routine is that it is two-phase. A DOI in DataCite moves through three states: draft, registered, and findable. A draft DOI is inert — it does not resolve, it is not indexed, and it can be deleted outright. A findable DOI resolves, is exposed in DataCite search and to downstream harvesters, and is permanent. The safe pattern reserves a draft first, uses the returned identifier while assembling and archiving the object, and only transitions the DOI to findable once the landing page it points at is live and returns a 200. A crash at any point before publication leaves nothing but a disposable draft; a crash after publication leaves a permanent, resolvable record, which is exactly what you want.

from __future__ import annotations

import httpx
from dataclasses import dataclass

DATACITE_API = "https://api.datacite.org"


@dataclass(frozen=True)
class MintResult:
    doi: str
    state: str  # "draft" | "findable"


class RegistryError(RuntimeError):
    ...


def _datacite_attributes(record, prefix: str) -> dict[str, object]:
    """Build a minimal, schema-valid DataCite Metadata Schema payload."""
    return {
        "prefix": prefix,
        "titles": [{"title": record.title}],
        "creators": [{"name": c} for c in record.creators],
        "publisher": record.publisher,
        "publicationYear": record.year,
        "types": {"resourceTypeGeneral": "Dataset"},
        "url": record.landing_page,  # where the findable DOI will resolve
        "rightsList": [{"rightsIdentifier": record.license_spdx}],
    }


def mint_doi(client: httpx.Client, record, prefix: str, auth: tuple[str, str]) -> MintResult:
    """Two-phase DOI mint against the DataCite REST API.

    Phase 1 reserves an inert draft DOI. Phase 2 publishes it to the findable
    state only after the object is archived and its landing page returns 200.
    Draft DOIs are deletable; findable DOIs are permanent, so the irreversible
    transition runs last and the whole routine is safe to retry.
    """
    # Phase 1 — create a draft. Omitting "event" leaves state == "draft".
    draft = client.post(
        f"{DATACITE_API}/dois",
        auth=auth,
        json={"data": {"type": "dois", "attributes": _datacite_attributes(record, prefix)}},
    )
    if draft.status_code not in (200, 201):
        raise RegistryError(f"draft creation failed: {draft.status_code} {draft.text}")
    doi = draft.json()["data"]["id"]

    # ... caller archives the object and asserts the landing page is live here ...

    # Phase 2 — publish. "event": "publish" transitions draft -> findable.
    published = client.put(
        f"{DATACITE_API}/dois/{doi}",
        auth=auth,
        json={"data": {"type": "dois", "attributes": {"event": "publish"}}},
    )
    if published.status_code != 200:
        raise RegistryError(f"publish failed for {doi}: {published.status_code}")
    state = published.json()["data"]["attributes"]["state"]
    if state != "findable":
        raise RegistryError(f"{doi} reached unexpected state {state!r}")
    return MintResult(doi=doi, state=state)

The routine is idempotent in the way that matters: re-running the publish PUT on an already-findable DOI is a no-op that returns the same state, so a retry after a network timeout never produces a second identifier or a corrupt record. The typed record that feeds _datacite_attributes should be the same validated object produced by the ingestion gate, whose construction and field constraints are specified in Pydantic schema validation — reusing it here means the DOI metadata cannot disagree with the metadata the archive was admitted under.

Repository Deposit Automation Behind a Common Adapter #

Minting an identifier makes a dataset citable; depositing it into a repository makes it downloadable, discoverable in disciplinary search, and — for many funders — compliant. Most platforms deposit to more than one target: Zenodo for a general-purpose CERN-backed archive that mints its own DOIs, Figshare for a commercial repository with strong institutional-portal integration, and Dataverse for the open-source platform many universities self-host. Their APIs differ in every incidental detail and agree on the essential shape: create a draft record, upload one or more files, attach metadata, and publish. The engineering move that keeps this maintainable is to hide those differences behind a single adapter interface so the pipeline speaks one vocabulary and each backend translates. The provider-specific request flows, authentication schemes, and metadata mappings for each target are detailed in the Repository Deposit Automation guide.

from __future__ import annotations

from dataclasses import dataclass
from typing import Protocol, runtime_checkable


@dataclass(frozen=True)
class DepositResult:
    repository: str          # "zenodo" | "figshare" | "dataverse"
    deposition_id: str       # provider-native record id
    concept_doi: str | None  # version-independent DOI, if the provider mints one
    version_doi: str | None  # this exact version's DOI
    landing_url: str


@runtime_checkable
class RepositoryAdapter(Protocol):
    """Uniform contract every repository backend must satisfy.

    Every method keys off a caller-supplied external_id so a retried deposit
    re-attaches to the in-flight record instead of creating a duplicate. This
    is what makes the whole deposit path safe to run at-least-once.
    """

    name: str

    def create_draft(self, metadata: dict[str, object], external_id: str) -> str: ...
    def upload_file(self, deposition_id: str, path: str, sha256: str) -> None: ...
    def publish(self, deposition_id: str) -> DepositResult: ...
    def get_state(self, deposition_id: str) -> str: ...


def deposit(adapter: RepositoryAdapter, record, files, external_id: str) -> DepositResult:
    """Provider-agnostic deposit: create -> upload -> publish, idempotently.

    The same call graph drives Zenodo, Figshare, or Dataverse; only the
    injected adapter changes. Upload passes a precomputed SHA-256 so the
    adapter can verify the remote copy before publish.
    """
    deposition_id = adapter.create_draft(record.repository_metadata(), external_id)
    for f in files:
        adapter.upload_file(deposition_id, f.path, f.sha256)
    return adapter.publish(deposition_id)

Two design points repay the effort. First, the external_id threaded through every method is the idempotency key: each adapter records it against the provider’s native record id, so a retried deposit resumes the existing draft rather than creating a duplicate — the same at-least-once discipline the ingestion consumer relies on, applied to the outbound side. Second, DepositResult distinguishes a concept DOI from a version DOI because Zenodo, and increasingly other platforms, mint both: a stable concept DOI that always resolves to the latest version and a version-specific DOI for the exact deposit. A pipeline that stores only one cannot later answer “cite this exact version,” so the adapter surface makes both first-class. The concrete provider walkthroughs — automating Zenodo batch uploads, publishing to Figshare, and depositing to Dataverse — implement exactly this interface.

Because deposit is a multi-step remote operation, it is the stage most exposed to partial failure: a draft is created, three of five files upload, and the connection drops. The get_state method exists precisely so the reconciliation loop can find a draft stuck between create and publish and drive it forward or tear it down deterministically, rather than leaving an orphaned half-deposit no one notices.

Crossref Metadata Deposit and How It Relates to DataCite #

DataCite and Crossref are sibling DOI registration agencies under the same DOI Foundation and the same Handle System resolver, but they serve different corners of the scholarly record. DataCite is the registry of record for datasets, software, samples, and other research outputs; Crossref is the registry of record for journal articles, conference papers, and books. For a pure dataset the DataCite DOI is usually the whole story. The reason Crossref enters a data-management pipeline at all is relationships: when a dataset underpins a published article, or a data paper describes a dataset, the citation graph that funders and bibliometric services traverse is stitched together from relatedIdentifier assertions on the DataCite side and equivalent relation metadata deposited to Crossref on the article side. Getting those cross-references to agree — same relation type, both directions asserted — is what makes a dataset show up in an article’s “associated data” panel. The deposit format, the XML schema Crossref expects, and the bidirectional relation patterns are covered in the Crossref Metadata Deposit guide.

The operational distinction that trips teams up is the deposit model. The DataCite REST API is synchronous and JSON-based: a 2xx response means the record is accepted and its state is immediately queryable. Crossref’s deposit path is historically asynchronous and XML-based: a deposit returns a submission id, and success or a schema-validation failure arrives later as a separate deposit report that must be polled or received by callback. A pipeline that treats a Crossref HTTP 200 on submission as “done” will silently accumulate rejected deposits. Model Crossref deposits as jobs with a pending state that only closes when the deposit report confirms it, and never mint a Crossref-side relation that names a DataCite DOI which is still a draft — a dangling cross-reference is worse than a missing one.

Choosing an Identifier Scheme #

DOI is the default for anything meant to be cited in the formal literature, but it is not the only persistent identifier and not always the right one. Two alternatives matter. The Archival Resource Key (ARK) is a scheme designed for institutional custody: ARKs are free to mint, carry no registration-agency fee, support suffix passthrough and inflections for metadata, and are governed by the assigning institution rather than a central registry — which makes them well suited to high-volume internal objects, specimens, and intermediate artifacts that need a stable name but not a formal citation. The Handle System is the lower-level resolution protocol that DOIs themselves are built on; minting bare Handles directly is appropriate when an institution runs its own Handle service and wants resolvable identifiers without tying them to either DataCite or Crossref semantics. The decision is not global — a mature platform mints DOIs for citable published datasets and ARKs for the high-volume tail of internal objects — and the criteria-by-criteria comparison, including cost, resolution guarantees, metadata obligations, and governance, is laid out in the Identifier Scheme Selection analysis, with a concrete ARK implementation in minting and resolving ARK identifiers.

The rule of thumb worth internalizing is that permanence and cost move together. A DOI’s permanence is backed by a registration agency and a contractual obligation to maintain resolution, which is why it costs money and why you must never mint one against an object you are not prepared to preserve indefinitely. An ARK’s permanence is backed only by the assigning institution’s own commitment — cheaper and more flexible, but exactly as durable as that institution’s stewardship. Choosing a scheme is therefore choosing who is responsible for resolution in twenty years, and that choice should be made per object class, encoded as policy, and enforced at mint time rather than left to whoever runs the deposit script. Where that policy lives alongside repository selection, funder mandates, and retention rules is the subject of Open Science Infrastructure Planning, which this identifier layer must satisfy.

Reconciliation & Synchronization of Identifier State #

The uncomfortable truth of this layer is that there is no distributed transaction across the platform’s database, DataCite, and three repositories. Each write can succeed or fail independently, external systems can be edited out-of-band by a curator, and a repository can re-mint or update a DOI on its own schedule. State drift is therefore not an anomaly to be prevented but a steady-state condition to be continuously repaired. A reconciliation job is the mechanism: on a schedule, it enumerates the platform’s own record of every identifier, fetches the current state from each external system, computes a diff, and applies a deterministic repair — publish a draft that should be findable, patch metadata that has fallen out of sync, flag a DOI that resolves to a dead landing page, and open an incident for anything it cannot resolve automatically. The lifecycle it must track for every DOI is shown below, and the full reconciliation algorithm — including how to represent the record of truth and how to make repairs idempotent — is detailed in Repository Sync & Reconciliation and its worked example, reconciling DOI state across repositories.

The state machine is what makes reconciliation deterministic rather than a pile of special cases. Every discrepancy the loop finds is a mismatch between the recorded state and the state the registry reports, and every legal repair is one of the labelled transitions: a draft that should have gone findable is republished; a findable DOI whose landing page 404s is a drift the loop escalates because the fix is upstream in the archive; a DOI a curator withdrew to tombstoned out-of-band is reconciled by updating the platform’s own record to match. The one transition the loop must never invent is deletion of a findable DOI, because that transition does not exist — a published DOI can only move to tombstoned, never disappear.

Failure Modes & Operational Runbook #

This layer is judged by how it behaves when a registry is slow, a quota is exhausted, or a deposit half-completes. The table is the operational core: the failures an on-call engineer will actually see, the signal that surfaces each, and the first remediation step.

Three of these cause the quietest data-integrity failures. Deposit partial failure is the most common and least visible: nothing errored loudly, a draft simply never got published, and it stays invisible until a researcher asks where their data went. The defense is that every deposition has a terminal state and an SLA, and anything non-terminal past its SLA is swept by the reconciliation loop. Metadata drift is subtle because both sides can be “right” — a curator legitimately corrects an author name in Zenodo while the platform still holds the original — so the decision of which side wins must be explicit and encoded, not resolved ad hoc. Tombstoned DOIs are dangerous only if the pipeline treats them as errors and tries to “fix” them by re-minting; the correct behavior is to accept the tombstone as a valid terminal state and reconcile the local record to it. The circuit-breaker and cache-fallback machinery that keeps registry downtime from becoming deposit downtime is shared with the resolution layer and specified in API Routing & Fallbacks; rate-limit handling with token-bucket throttling and resumable checkpoints is the same discipline applied to the outbound path.

Underpinning the runbook is the same observability posture the rest of the platform uses: structured logs on every mint, deposit, and transition; a metric for the count of identifiers in each lifecycle state; and a nightly reconciliation report that must reach zero unexplained discrepancies. Detection is what turns each of these failure modes from a silent corruption into a routine, replayable event.

Security & Access Control #

Repository and registry integration concentrates exactly the credentials an attacker wants: a DataCite account can mint or withhold identifiers under an institution’s prefix, and a repository token can publish or delete records. Treat these as high-value secrets — store them in a managed secret store rather than environment files, scope each token to the narrowest capability the pipeline needs, and prefer per-repository service accounts so a compromised Figshare token cannot touch DataCite. Automation should authenticate against the institution’s OIDC or SAML provider and exchange for short-lived tokens on service-to-service calls, so a leaked credential expires rather than persisting; the broader access model, encryption, and audit-logging requirements are specified in Security & Access Control.

Two integration-specific rules matter beyond generic secret hygiene. First, minting is a privileged, irreversible action, so the transition from draft to findable should be gated by the same policy checks the ingestion boundary applies — an object without a resolvable landing page and a valid license must never reach findable. Second, access decisions and identifier state are decoupled: a DOI stays resolvable and its metadata public even when the underlying bytes are embargoed, so the deposit adapter must set the repository’s access condition independently of publication, and the reconciliation loop must treat an embargoed-but-findable record as correct rather than as drift. Every mint, publish, and deposit belongs in an immutable audit log, because the questions auditors ask — who minted this DOI, when did it become findable, why does it point where it does — can only be answered from a record that cannot be edited after the fact.

FAQ #

Should I mint the DOI before or after depositing to the repository? #

Mint a draft first, deposit second, and publish last. Reserving a draft DOI up front gives you a citable string to embed in the landing page and in the metadata you deposit, so the repository record and the DOI record agree from the start. Because a draft is inert and deletable, nothing is lost if the deposit fails — you abandon the draft. Only after the object is archived and its landing page returns a 200 do you transition the DOI to findable, which is the irreversible step. Note that some repositories, Zenodo among them, will mint the DOI for you as part of publishing; in that case you consume their identifier rather than minting your own, but the ordering discipline is identical.

What is the difference between DataCite and Crossref, and do I need both? #

DataCite is the DOI registration agency for datasets, software, and other research outputs; Crossref is the agency for journal articles and books. If you are only publishing datasets, DataCite alone is sufficient. Crossref becomes relevant when a dataset is linked to a published article and you want that relationship to appear in citation graphs — the article’s Crossref record and the dataset’s DataCite record assert complementary relatedIdentifier relations. The practical gotcha is that Crossref’s deposit path is asynchronous and XML-based, so a deposit’s real success arrives in a later deposit report, not in the initial HTTP response. See the Crossref Metadata Deposit guide.

How do I avoid creating duplicate deposits when a job retries? #

Thread a stable idempotency key — the external_id in the adapter interface above — through every call, and have each adapter record it against the provider’s native record id. On a retry, the adapter looks up the key, finds the in-flight draft, and resumes it instead of creating a new one. Combined with acknowledging progress only after each step durably completes, this makes the deposit path safe to run at-least-once, which is the only realistic guarantee across unreliable network calls. The resumable patterns are the outbound counterpart of those in async batch processing.

A DOI resolves but its landing page returns a 404 — what do I do? #

That is landing-page rot, and it is an upstream problem: the identifier is fine, but the object it points at moved or was lost. Do not touch the DOI. Quarantine the record, restore or re-point the landing page in the archive, verify that the resolved URL returns a 200, and only then clear the incident. A nightly reconciliation job that probes resolved URLs is what catches this before a citing author does. The reconciliation algorithm is detailed in Repository Sync & Reconciliation.

When should I use an ARK instead of a DOI? #

Use a DOI for anything meant to be formally cited and preserved indefinitely, and an Archival Resource Key (ARK) for the high-volume internal objects — specimens, intermediate artifacts, working records — that need a stable, resolvable name but not a registration-agency-backed citation. ARKs are free to mint and governed by your institution, which makes them cheaper and more flexible but exactly as durable as your own stewardship. The trade-off is really about who guarantees resolution in twenty years; the full comparison is in Identifier Scheme Selection.

Can I delete a DOI that was minted by mistake? #

Only if it is still a draft. A draft DOI is inert and can be deleted outright. Once a DOI reaches the findable state it is permanent and cannot be deleted — the only forward transition is to tombstone it, which keeps the DOI resolving to a tombstone page that explains the withdrawal. This is exactly why minting is two-phase: you do all your validation while the identifier is still a disposable draft, and you cross into the irreversible findable state only when you are certain. Treat a tombstoned DOI as a valid terminal state, not an error to be repaired.

Related #

• DOI Minting with DataCite — registering, updating, and versioning DOIs against the DataCite REST API and Metadata Schema.
• Repository Deposit Automation — the common adapter and provider implementations for Zenodo, Figshare, and Dataverse.
• Crossref Metadata Deposit — depositing article-to-dataset relations and how Crossref complements DataCite.
• Identifier Scheme Selection — choosing between DOI, ARK, and the Handle System per object class.
• Repository Sync & Reconciliation — the drift-repair loop that keeps identifier state consistent across systems.
• Core Architecture & FAIR Mapping — the platform topology and resolution layer this identifier work plugs into.

This overview is the map for the identifier and deposit layer of the scientific research data management platform; start here for the mint-to-reconcile pipeline, then follow any stage into its detailed implementation guide.