Repository Deposit Automation: A Unified Pipeline for Zenodo, Figshare, and Dataverse #

Research groups rarely deposit to a single archive. A grant may require Zenodo for a software release, an institutional Figshare portal for supplementary figures, and a Dataverse installation for the tabular data that underpins a paper — often the same dataset, mirrored three ways. Hand-driven deposits do not survive that fan-out: each repository has its own authentication scheme, its own file-upload mechanism, and its own moment at which a Digital Object Identifier is minted, so teams end up maintaining three brittle scripts that drift apart and silently publish inconsistent records. This guide builds the alternative — one deposit pipeline that treats Zenodo, Figshare, and Dataverse as interchangeable backends behind a single Python interface, advancing every dataset through the same lifecycle and capturing the DOI each archive returns. It sits inside the Persistent Identifiers & Repository Integration area as the deposit execution layer, downstream of the identifier decisions and upstream of reconciliation. The per-backend mechanics live in three companion walkthroughs — automating Zenodo batch uploads with the REST API, publishing datasets to Figshare via the API, and depositing to Dataverse via the Native API — while this page defines the abstraction that unifies them.

The Common Deposit Lifecycle #

Despite their surface differences, all three archives expose the same five-stage lifecycle, and a unified pipeline is possible precisely because that shape is invariant. A deposit begins by creating a draft — an empty, unpublished record that reserves an internal identifier and, in most installations, a placeholder DOI. Files are then uploaded and checksum-verified, so the bytes the archive stores are provably the bytes you sent. Metadata is attached to the draft: title, creators with their Open Researcher and Contributor ID (ORCID) identifiers, a license drawn from the SPDX License List, and a resource type. The draft is then published, an irreversible transition that freezes the files and metadata and instructs the registration agency to make the DOI findable. Finally the pipeline captures the minted DOI from the publish response and records it, closing the loop back to the identifier bookkeeping that the DOI minting with DataCite guide governs.

The value of naming these stages precisely is that each maps to a distinct failure domain. A draft-creation failure is an authentication or quota problem; an upload failure is a network or checksum problem; a publish failure is a validation problem where the archive rejects incomplete metadata. Keeping the stages separate means a pipeline can retry each with the right strategy instead of blindly re-running the whole deposit — which, without idempotency, would duplicate records and burn DOIs.

A Repository-Adapter Abstraction #

The unifying construct is an adapter: a small object that knows how to speak one archive’s dialect while exposing the same four verbs to the pipeline. Python’s typing.Protocol expresses this as structural typing — any class that provides create_draft, upload, set_metadata, and publish with the right signatures satisfies the contract, without inheriting from a base class. That keeps each backend implementation independent and testable in isolation, and it lets the orchestrator hold a list[RepositoryAdapter] and iterate without caring which archive is which.

from __future__ import annotations

from dataclasses import dataclass, field
from pathlib import Path
from typing import Protocol, runtime_checkable


@dataclass(frozen=True)
class DepositMetadata:
    """Backend-neutral metadata; each adapter maps it to the archive's schema."""
    title: str
    creators: list[dict[str, str]]           # [{"name": "Curie, Marie", "orcid": "0000-..."}]
    description: str
    license_spdx: str                        # e.g. "CC-BY-4.0"
    resource_type: str = "dataset"
    keywords: list[str] = field(default_factory=list)


@dataclass(frozen=True)
class DraftHandle:
    """Opaque reference to an unpublished record on one backend."""
    backend: str
    deposit_id: str
    upload_target: str                       # bucket URL, upload service URL, or persistentId


@dataclass(frozen=True)
class FileRef:
    filename: str
    remote_checksum: str                     # digest the archive computed and returned


@dataclass(frozen=True)
class MintedRecord:
    backend: str
    deposit_id: str
    doi: str
    record_url: str


@runtime_checkable
class RepositoryAdapter(Protocol):
    """The uniform surface every deposit backend must implement."""

    name: str

    def create_draft(self, metadata: DepositMetadata) -> DraftHandle: ...

    def upload(self, draft: DraftHandle, path: Path, *, md5: str) -> FileRef: ...

    def set_metadata(self, draft: DraftHandle, metadata: DepositMetadata) -> None: ...

    def publish(self, draft: DraftHandle) -> MintedRecord: ...

Marking the Protocol runtime_checkable lets a test assert isinstance(adapter, RepositoryAdapter) to confirm a backend is wired up before a run, catching a missing method at startup rather than mid-deposit. The DepositMetadata record is deliberately backend-neutral: the mapping onto DataCite-style creators, Figshare authors, or a Dataverse citation block happens inside each adapter, which is where the archive-specific knowledge belongs.

Idempotency via Content Hashes #

The single most important property of an automated deposit pipeline is that re-running it is safe. Networks drop, tokens expire, and CI jobs get re-triggered; without idempotency, every restart risks a second Zenodo deposition, a duplicate Figshare article, and a wasted, permanently registered DOI. The pipeline earns idempotency the same way the batch layer in async batch processing does — with content addressing. Compute a deterministic deposit key from the sorted set of file digests plus a normalized metadata hash, and keep a small persistent ledger mapping that key to the record already created. Before creating a draft, look the key up: if a published record exists, the deposit is a no-op; if a draft exists, resume it rather than starting over.

from __future__ import annotations

import hashlib
import json
from pathlib import Path


def file_md5(path: Path, chunk: int = 1 << 20) -> str:
    """Stream a file through MD5 so multi-gigabyte inputs stay memory-constant."""
    digest = hashlib.md5()
    with path.open("rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            digest.update(block)
    return digest.hexdigest()


def deposit_key(files: list[Path], metadata: DepositMetadata) -> str:
    """A stable key over content + metadata: identical inputs -> identical key."""
    file_part = sorted(f"{p.name}:{file_md5(p)}" for p in files)
    meta_part = json.dumps(
        {"title": metadata.title, "license": metadata.license_spdx,
         "creators": metadata.creators},
        sort_keys=True, separators=(",", ":"),
    )
    payload = "|".join(file_part) + "||" + meta_part
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()

MD5 is not a security choice here — it is the checksum every one of these archives computes server-side and returns, so using it locally lets the pipeline compare digests directly without a translation step. The SHA-256 deposit key is separate: it is the pipeline’s own dedup primary key, and it changes if any file byte or any load-bearing metadata field changes, which correctly forces a new version rather than silently overwriting.

Resumable Uploads #

Large deposits fail mid-transfer, and re-sending gigabytes because byte one megabyte after the last checkpoint dropped is unacceptable. Resumability comes from two facts: the deposit key identifies the in-progress draft, and each archive can be queried for the files it already holds. On restart, the pipeline reloads the DraftHandle from its ledger, lists the remote files, and uploads only those whose local MD5 is absent from the remote set. Zenodo’s bucket API and Dataverse’s file API are naturally idempotent per filename, and Figshare’s part-based upload lets you skip already-completed parts — so “resume” reduces to “diff local against remote, then upload the difference.” The serializing progress checkpoints for resumable jobs walkthrough covers durably persisting that ledger across process restarts.

Step-by-Step Implementation #

The orchestrator below drives any adapter through the lifecycle, consulting the ledger for idempotency and verifying every checksum before it will publish. The four steps map one-to-one to the Protocol methods.

Step 1 — Create or resume the draft #

Resolve the deposit key first, then either reuse an existing draft or ask the adapter to create one. This is the idempotency gate: a re-run with unchanged inputs finds the existing record and never mints a second identifier.

from __future__ import annotations

import logging
from pathlib import Path

logger = logging.getLogger("deposit.pipeline")


class DepositLedger:
    """Minimal persistent map: deposit_key -> DraftHandle | MintedRecord."""

    def __init__(self, store: dict[str, dict[str, str]]) -> None:
        self._store = store

    def find(self, key: str) -> dict[str, str] | None:
        return self._store.get(key)

    def remember_draft(self, key: str, draft: DraftHandle) -> None:
        self._store[key] = {"state": "draft", "backend": draft.backend,
                            "deposit_id": draft.deposit_id, "target": draft.upload_target}

    def remember_record(self, key: str, record: MintedRecord) -> None:
        self._store[key] = {"state": "published", "doi": record.doi,
                            "url": record.record_url, "deposit_id": record.deposit_id}


def create_or_resume(
    adapter: RepositoryAdapter, ledger: DepositLedger, key: str, metadata: DepositMetadata
) -> DraftHandle:
    existing = ledger.find(key)
    if existing and existing["state"] == "draft":
        logger.info("Resuming existing draft %s on %s", existing["deposit_id"], adapter.name)
        return DraftHandle(existing["backend"], existing["deposit_id"], existing["target"])
    draft = adapter.create_draft(metadata)
    ledger.remember_draft(key, draft)
    return draft

Step 2 — Upload files with checksum verification #

Upload each file, then assert the digest the archive returns equals the digest you computed locally. A mismatch means the bytes were corrupted in transit; the pipeline must re-upload rather than proceed, because a published record with a silently corrupted file is a FAIR-integrity failure that no downstream consumer can detect.

from __future__ import annotations


class ChecksumMismatch(RuntimeError):
    """The archive's returned digest disagrees with the local one."""


def upload_verified(
    adapter: RepositoryAdapter, draft: DraftHandle, path: Path, *, max_attempts: int = 3
) -> FileRef:
    local_md5 = file_md5(path)
    for attempt in range(1, max_attempts + 1):
        ref = adapter.upload(draft, path, md5=local_md5)
        if ref.remote_checksum.lower() == local_md5.lower():
            logger.info("Verified %s (%s) on %s", path.name, local_md5, adapter.name)
            return ref
        logger.warning(
            "Checksum mismatch for %s on attempt %d/%d (local=%s remote=%s)",
            path.name, attempt, max_attempts, local_md5, ref.remote_checksum,
        )
    raise ChecksumMismatch(f"{path.name}: {max_attempts} uploads all failed verification")

Step 3 — Attach metadata #

With bytes verified, map the backend-neutral DepositMetadata onto the archive’s schema and write it to the draft. Each adapter owns this translation, so the orchestrator simply delegates. Attaching metadata after the files means a validation error surfaces against a draft that is otherwise ready to publish.

def attach_metadata(
    adapter: RepositoryAdapter, draft: DraftHandle, metadata: DepositMetadata
) -> None:
    adapter.set_metadata(draft, metadata)
    logger.info("Metadata attached to %s draft %s", adapter.name, draft.deposit_id)

Step 4 — Publish and capture the minted DOI #

Publishing is irreversible, so it runs last and only after every prior stage succeeded. The adapter returns a MintedRecord carrying the DOI, which the pipeline writes back to the ledger, flipping the deposit key’s state to published so any future run short-circuits.

def run_deposit(
    adapter: RepositoryAdapter,
    ledger: DepositLedger,
    files: list[Path],
    metadata: DepositMetadata,
) -> MintedRecord:
    key = deposit_key(files, metadata)
    prior = ledger.find(key)
    if prior and prior["state"] == "published":
        logger.info("Deposit %s already published as %s; skipping", key[:12], prior["doi"])
        return MintedRecord(adapter.name, prior["deposit_id"], prior["doi"], prior["url"])

    draft = create_or_resume(adapter, ledger, key, metadata)
    remote_files = {r.filename: r.remote_checksum for r in _list_remote(adapter, draft)}
    for path in files:
        if path.name in remote_files and remote_files[path.name].lower() == file_md5(path).lower():
            continue  # resume: already uploaded and verified
        upload_verified(adapter, draft, path)
    attach_metadata(adapter, draft, metadata)
    record = adapter.publish(draft)
    ledger.remember_record(key, record)
    logger.info("Published %s -> %s", adapter.name, record.doi)
    return record


def _list_remote(adapter: RepositoryAdapter, draft: DraftHandle) -> list[FileRef]:
    """Adapters expose a list_files helper; default to empty for a fresh draft."""
    lister = getattr(adapter, "list_files", None)
    return lister(draft) if callable(lister) else []

How Each Backend Differs #

The adapters differ only inside these four methods, and the table below is the exact map of what changes. Zenodo authenticates with a bearer token and streams whole files to a per-deposition bucket URL; Figshare authenticates the same way but requires a multipart, MD5-per-part upload through a separate upload service; Dataverse uses a custom header and a multipart form against a persistent identifier. All three mint DOIs through DataCite, but they diverge on when the DOI becomes findable and how versioning is expressed.

Error Handling & Edge Cases #

Deposit failures split cleanly into two categories that demand opposite responses. Retryable faults — HTTP 429 rate limiting, 502/503 from an overloaded archive, or a transient checksum mismatch — should back off with jitter and try again, exactly as the idempotent retry patterns for batch deposits guide details. Terminal faults — a 400 rejecting malformed metadata, a 403 from an expired token, or a 409 conflict on an already-published record — must stop the deposit and surface for a human, because retrying them only wastes quota. The most dangerous edge case is a partial publish: the archive registers the DOI but the network drops before the pipeline reads the response. Because publishing is idempotent per draft on all three backends, the fix is to re-fetch the draft’s state on restart — if it reports published, harvest the DOI from the record rather than re-publishing.

import time
import random

import httpx


def with_backoff(fn, *, max_attempts: int = 5, base: float = 1.0):
    """Retry retryable HTTP faults; let terminal ones (4xx except 429) propagate."""
    for attempt in range(1, max_attempts + 1):
        try:
            return fn()
        except httpx.HTTPStatusError as exc:
            code = exc.response.status_code
            terminal = code < 500 and code != 429
            if terminal or attempt == max_attempts:
                raise
            delay = base * 2 ** (attempt - 1) + random.uniform(0, 0.5)
            time.sleep(delay)
    raise RuntimeError("unreachable")

Verification & Testing #

Assert the orchestration logic without touching a live archive by driving it with an in-memory fake adapter that satisfies the Protocol. The test below proves the two properties that matter most: a re-run of an already-published deposit mints nothing new, and a corrupted upload is rejected before publish.

from pathlib import Path

import pytest


class FakeAdapter:
    name = "fake"

    def __init__(self, corrupt: bool = False) -> None:
        self.corrupt = corrupt
        self.publishes = 0

    def create_draft(self, metadata: DepositMetadata) -> DraftHandle:
        return DraftHandle("fake", "dep-1", "mem://bucket")

    def upload(self, draft: DraftHandle, path: Path, *, md5: str) -> FileRef:
        return FileRef(path.name, "0" * 32 if self.corrupt else md5)

    def set_metadata(self, draft: DraftHandle, metadata: DepositMetadata) -> None:
        pass

    def publish(self, draft: DraftHandle) -> MintedRecord:
        self.publishes += 1
        return MintedRecord("fake", "dep-1", "10.5072/zenodo.1", "https://example.org/1")

    def list_files(self, draft: DraftHandle) -> list[FileRef]:
        return []


def test_republish_is_idempotent(tmp_path: Path) -> None:
    f = tmp_path / "data.csv"
    f.write_bytes(b"a,b\n1,2\n")
    meta = DepositMetadata("T", [{"name": "Curie, M."}], "d", "CC-BY-4.0")
    ledger = DepositLedger({})
    adapter = FakeAdapter()
    run_deposit(adapter, ledger, [f], meta)
    run_deposit(adapter, ledger, [f], meta)      # replay
    assert adapter.publishes == 1                # published exactly once


def test_corrupt_upload_never_publishes(tmp_path: Path) -> None:
    f = tmp_path / "data.csv"
    f.write_bytes(b"a,b\n1,2\n")
    meta = DepositMetadata("T", [{"name": "Curie, M."}], "d", "CC-BY-4.0")
    adapter = FakeAdapter(corrupt=True)
    with pytest.raises(ChecksumMismatch):
        run_deposit(adapter, DepositLedger({}), [f], meta)
    assert adapter.publishes == 0

Run the suite with pytest -q. Because the fake adapter satisfies the same Protocol as the real Zenodo, Figshare, and Dataverse adapters, a green run proves the orchestration is correct independent of any network, and the real adapters can then be tested against each archive’s sandbox instance in an integration tier.

Gotchas & Known Pitfalls #

• Re-running a deposit without a ledger double-mints DOIs. Root cause: the create-draft call is not guarded by a content-derived key, so a CI retry starts a fresh deposition. Fix: compute the SHA-256 deposit key first and short-circuit on a published ledger entry, as run_deposit does.
• Trusting the upload without checking the returned checksum. Root cause: assuming an HTTP 201 means the bytes arrived intact. Fix: compare the archive’s returned MD5 against the locally computed one and re-upload on mismatch before you publish.
• Publishing before metadata validation. Root cause: attaching metadata after publish, when the record is already frozen. Fix: order the lifecycle so set_metadata and its validation run against the draft; publish only once the draft is complete.
• Assuming a new deposit is a new version. Root cause: creating a fresh deposition for an updated dataset instead of using the backend’s version action. Fix: route updates through newversion on Zenodo, an article edit on Figshare, or a major/minor publish on Dataverse so the concept DOI and lineage are preserved.
• Hard-coding one archive’s DOI-reservation timing. Root cause: reading the DOI at draft time on a backend that only makes it findable at publish. Fix: capture the DOI from the publish response, and treat any pre-reservation as provisional until the record is published.

Frequently Asked Questions #

Can one pipeline really deposit the same dataset to all three archives? #

Yes, and that is the point of the adapter abstraction. Because Zenodo, Figshare, and Dataverse share the create-draft, upload, set-metadata, publish lifecycle, a single orchestrator can hold one adapter per archive and run the same DepositMetadata through each. What differs — authentication, upload mechanics, and DOI timing — is confined to each adapter’s four methods, so the pipeline logic, idempotency ledger, and checksum verification are written once. You will get three distinct DOIs, one per archive, which is expected; reconciling them into a single logical record is the job of the reconciling DOI state across repositories walkthrough.

How do content hashes make the deposit idempotent? #

The pipeline derives a deterministic deposit key from the sorted file MD5 digests plus a normalized hash of the load-bearing metadata, then keeps a ledger mapping that key to the record it produced. A re-run recomputes the same key from the same inputs, finds the existing entry, and either resumes an in-progress draft or skips a published one entirely. Because the key changes whenever a file byte or a metadata field changes, an actual content change correctly forces a new version instead of silently overwriting the old record.

What happens if an upload is interrupted halfway through? #

Nothing is lost. The draft persists on the archive, and its handle persists in the ledger keyed by the deposit key. On restart the pipeline reloads the draft, lists the files the archive already holds, compares each remote MD5 against the local one, and uploads only the missing or mismatched files. This diff-then-upload approach turns a failed multi-gigabyte transfer into a cheap resume rather than a full restart.

Do all three repositories mint DOIs the same way? #

They all register through DataCite, but they diverge on timing. Zenodo pre-reserves an inactive DOI at draft creation and activates it on publish; Dataverse assigns a draft persistent identifier at dataset creation and makes it findable when you publish a major or minor version; Figshare requires an explicit reserve_doi call and activates it on publish. The practical rule is to treat any pre-publish DOI as provisional and to capture the authoritative identifier from the publish response, which the DOI minting with DataCite guide covers in depth.

Related Guides #

• Persistent Identifiers & Repository Integration — the parent area that positions this deposit layer between identifier selection and reconciliation.
• Automating Zenodo batch uploads with the REST API — the bucket-based upload and new-version flow for the Zenodo adapter.
• Publishing datasets to Figshare via the API — the multipart, MD5-per-part upload behind the Figshare adapter.
• Depositing to Dataverse via the Native API — the citation-block JSON and major/minor publish behind the Dataverse adapter.
• Open License Configuration — resolving the SPDX license label that every adapter attaches to its metadata.