FAIR Data Automation Research Data Management

Depositing to Dataverse via the Native API

Dataverse is the archive of choice for many institutional repositories, and its Native API is powerful — but the create step is unforgiving, because a dataset is defined by a deeply nested citation-metadata JSON where every field carries its own typeName, typeClass, and multiple flag. Omit a required field or mistype a compound value and the API rejects the whole dataset with a validation error. This walkthrough runs the Dataverse Native API end to end: build the datasetVersion JSON for the citation metadata block, create the dataset inside a parent collection, add files with multipart uploads and MD5 verification, then publish a major or minor version. It is the concrete Dataverse backend behind the Repository Deposit Automation guide and follows the same create-upload-metadata-publish lifecycle. You need a Dataverse API token, the alias of a collection you can publish into, Python 3.10+, and httpx; test against https://demo.dataverse.org. The persistent identifier assigned at creation is registered through DataCite when you publish, as the DOI minting with DataCite guide describes.

Dataverse Native API deposit flow Four stages run left to right: create dataset posts the datasetVersion citation JSON into a parent collection and returns a draft persistentId; add files uploads each file by multipart form with MD5 verification; publish sends type major or minor; and the terminal box marks the DOI as findable. Create dataset datasetVersion JSON Add files multipart + MD5 Publish type=major|minor DOI findable version 1.0

The Citation Metadata Block Fields

Every dataset must carry a valid citation metadata block, and five fields are required for the dataset to save: title, author, datasetContact, dsDescription, and subject. Primitive fields hold a scalar; compound fields hold a list of sub-field dictionaries; subject is a controlled vocabulary whose values must match Dataverse’s list exactly. The table is the exact shape of each field inside datasetVersion.metadataBlocks.citation.fields.

Field (typeName) typeClass multiple Required Example value
title primitive false yes Reef Temperature Series 2019–2024
author compound true yes authorName = Curie, Marie, authorAffiliation = Institut Curie
datasetContact compound true yes datasetContactName = Curie, Marie, datasetContactEmail = data@example.org
dsDescription compound true yes dsDescriptionValue = Hourly sea-surface temperatures…
subject controlledVocabulary true yes ["Earth and Environmental Sciences"]
keyword compound true no keywordValue = oceanography
depositor primitive false no Curie, Marie

The citation block is only the mandatory core; a Dataverse installation may also enable domain-specific blocks such as geospatial, social-science, or life-science metadata, each added as a sibling key under metadataBlocks. Those extra blocks follow the same typeName/typeClass/multiple grammar, so the helper that builds the citation fields extends naturally to them. Keeping the required five correct is what matters for a save to succeed; the optional fields and extra blocks enrich discoverability without changing the deposit flow.

Production Implementation

The client below assembles the citation fields with a small helper so the nesting stays readable, creates the dataset in a parent collection, adds files via multipart/form-data, verifies each returned MD5, and publishes. File upload and publish both address the dataset by its persistent identifier through the :persistentId path form.

python
from __future__ import annotations

import hashlib
import json
from pathlib import Path
from typing import Any

import httpx

DATAVERSE = "https://demo.dataverse.org"


def compute_md5(path: Path, chunk: int = 1 << 20) -> str:
    """Stream the file through MD5 so large uploads stay memory-constant."""
    digest = hashlib.md5()
    with path.open("rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            digest.update(block)
    return digest.hexdigest()


class DataverseError(RuntimeError):
    """Raised on an unrecoverable Dataverse API response."""


def _primitive(name: str, value: str) -> dict[str, Any]:
    return {"typeName": name, "typeClass": "primitive", "multiple": False, "value": value}


def _compound_sub(name: str, value: str) -> dict[str, Any]:
    return {name: {"typeName": name, "typeClass": "primitive", "multiple": False, "value": value}}


def build_citation(
    title: str, author: str, affiliation: str, contact_email: str, description: str, subject: str
) -> dict[str, Any]:
    """Assemble the required citation metadata block for a new dataset."""
    fields = [
        _primitive("title", title),
        {"typeName": "author", "typeClass": "compound", "multiple": True,
         "value": [{**_compound_sub("authorName", author),
                    **_compound_sub("authorAffiliation", affiliation)}]},
        {"typeName": "datasetContact", "typeClass": "compound", "multiple": True,
         "value": [{**_compound_sub("datasetContactName", author),
                    **_compound_sub("datasetContactEmail", contact_email)}]},
        {"typeName": "dsDescription", "typeClass": "compound", "multiple": True,
         "value": [_compound_sub("dsDescriptionValue", description)]},
        {"typeName": "subject", "typeClass": "controlledVocabulary", "multiple": True,
         "value": [subject]},
    ]
    return {"metadataBlocks": {"citation": {"displayName": "Citation Metadata", "fields": fields}}}


def create_dataset(client: httpx.Client, parent_alias: str, dataset_version: dict[str, Any]) -> str:
    """POST the datasetVersion into a collection; returns the draft persistentId."""
    resp = client.post(f"/api/dataverses/{parent_alias}/datasets",
                       json={"datasetVersion": dataset_version})
    if resp.status_code >= 400:
        raise DataverseError(f"Dataset creation failed: {resp.text}")
    return resp.json()["data"]["persistentId"]     # e.g. "doi:10.5072/FK2/ABCDEF"


def add_file(client: httpx.Client, pid: str, path: Path, description: str = "") -> str:
    """Upload one file via multipart and return the checksum Dataverse computed."""
    local_md5 = compute_md5(path)
    json_data = json.dumps({"description": description, "restrict": False})
    with path.open("rb") as fh:
        resp = client.post(
            "/api/datasets/:persistentId/add",
            params={"persistentId": pid},
            files={"file": (path.name, fh)},
            data={"jsonData": json_data},
        )
    if resp.status_code >= 400:
        raise DataverseError(f"File add failed for {path.name}: {resp.text}")
    data_file = resp.json()["data"]["files"][0]["dataFile"]
    remote = (data_file.get("checksum", {}) or {}).get("value") or data_file.get("md5", "")
    if remote.lower() != local_md5.lower():
        raise DataverseError(f"Checksum mismatch for {path.name}: {remote} != {local_md5}")
    return remote


def publish_dataset(client: httpx.Client, pid: str, version_type: str = "major") -> dict[str, Any]:
    """Publish the dataset; the first release must be a major version."""
    resp = client.post("/api/datasets/:persistentId/actions/:publish",
                       params={"persistentId": pid, "type": version_type})
    resp.raise_for_status()
    return resp.json()["data"]


def deposit_dataset(
    token: str, parent_alias: str, files: list[Path], **meta: str
) -> dict[str, str]:
    """Full flow: build citation -> create -> add files -> publish major."""
    headers = {"X-Dataverse-key": token}
    with httpx.Client(base_url=DATAVERSE, headers=headers, timeout=120.0) as client:
        version = build_citation(
            title=meta["title"], author=meta["author"], affiliation=meta["affiliation"],
            contact_email=meta["contact_email"], description=meta["description"],
            subject=meta["subject"],
        )
        pid = create_dataset(client, parent_alias, version)
        for path in files:
            add_file(client, pid, path, description=f"Data file {path.name}")
        published = publish_dataset(client, pid, version_type="major")
        return {"persistent_id": pid, "version": f"{published['versionNumber']}.0"}


if __name__ == "__main__":
    result = deposit_dataset(
        token="YOUR_TOKEN",
        parent_alias="my-collection",
        files=[Path("reef_sst.csv")],
        title="Reef Temperature Series 2019–2024",
        author="Curie, Marie",
        affiliation="Institut Curie",
        contact_email="data@example.org",
        description="Hourly sea-surface temperatures from 12 reef stations.",
        subject="Earth and Environmental Sciences",
    )
    print(result["persistent_id"])

Draft, Publish, and Version Semantics

Dataverse separates a dataset’s editable state from its citable state, and knowing where that boundary sits prevents the two most common automation mistakes. Creating a dataset, or editing a published one, always produces a draft version that only you and your collaborators can see; the persistent identifier exists but is not yet findable through DataCite. Publishing is the transition that registers or updates the DOI and exposes the version to the world. Because that transition is irreversible for a given version number, the pipeline should add and verify every file against the draft first, then publish once — exactly the order the Repository Deposit Automation guide prescribes.

The type parameter encodes the version increment. The very first publication must be major, producing version 1.0; later edits to a published dataset publish as minor (1.1, 1.2) for metadata or additive file changes, or major (2.0) when the change is significant enough that downstream users should re-review. Dataverse tracks these numbers automatically, so an updater only decides which increment applies. Files carry their own MD5 fixity, recorded at add time and re-verified by the installation, which is why the add_file guard compares the returned checksum before the pipeline is allowed to move on to publish. Keeping the fixity check at the file boundary means a corrupted upload is caught while the dataset is still an unpublished draft and can be fixed without minting anything.

Verification

Confirm the citation block is well formed and that the file-add checksum guard works, both offline, using httpx.MockTransport. The test builds the citation JSON, asserts the five required fields are present, and proves a mismatched checksum is rejected.

python
from pathlib import Path

import httpx
import pytest


def test_citation_has_required_fields() -> None:
    version = build_citation(
        "T", "Curie, Marie", "Institut Curie", "d@example.org", "desc",
        "Earth and Environmental Sciences",
    )
    names = {f["typeName"] for f in version["metadataBlocks"]["citation"]["fields"]}
    assert {"title", "author", "datasetContact", "dsDescription", "subject"} <= names


def test_add_file_rejects_bad_checksum(tmp_path: Path) -> None:
    f = tmp_path / "data.csv"
    f.write_bytes(b"a,b\n1,2\n")

    def handler(request: httpx.Request) -> httpx.Response:
        return httpx.Response(200, json={"data": {"files": [
            {"dataFile": {"checksum": {"type": "MD5", "value": "0" * 32}}}
        ]}})

    transport = httpx.MockTransport(handler)
    with httpx.Client(base_url="https://demo.dataverse.org", transport=transport) as client:
        with pytest.raises(DataverseError):
            add_file(client, "doi:10.5072/FK2/ABCDEF", f)

Run it with pytest -q. Against a live demo.dataverse.org token, deposit_dataset returns the dataset’s persistentId (a doi: value) and the published version 1.0, visible in the collection you targeted.

Gotchas

  • An unlisted subject value. subject is a controlled vocabulary; a free-text string such as "Oceanography" is rejected because it is not in Dataverse’s list. Fix: use an exact vocabulary term such as Earth and Environmental Sciences.
  • Publishing the first version as minor. Dataverse requires the initial release to be a major version, so type=minor on a never-published dataset fails. Fix: publish the first version with type=major, and reserve minor for later edits to a published dataset.
  • Building compound fields as flat strings. Writing author as a plain string instead of a list of sub-field dictionaries produces a validation error. Fix: nest each compound value as {"authorName": {…}, "authorAffiliation": {…}}, exactly as build_citation does.