Core Architecture & FAIR Mapping for Scientific Research Data Management #

Scientific research data management breaks down at the boundary between how instruments produce data and how funders, journals, and institutional review boards expect it to be described, preserved, and cited. A microscope writes proprietary binary frames; a national funder demands a resolvable Digital Object Identifier (DOI) with machine-readable provenance within a fixed embargo window. When that gap is bridged by hand — spreadsheets of accession numbers, one-off migration scripts, README files that drift out of sync with the archive — the result is unfindable data, broken landing pages, and audit failures that surface only when a grant renewal is on the line. This overview treats the whole span from ingestion to exposure as a single engineered system: the layered infrastructure that persists and indexes data, the pipeline patterns that move it deterministically, the compliance checkpoints that enforce FAIR at write time, the identifier and routing layer that keeps it findable, and the failure modes an operations team must be ready to remediate. It is written for the Python automation engineers, research data managers, and academic IT teams who own that system in production, and it links directly to the detailed implementation guides for each layer.

Architecture Overview #

Every reliable research data platform shares the same coarse topology: a validation gate at the edge, immutable persistence beneath it, a semantic index that makes the data queryable, a resolution service that turns identifiers into addresses, and an API surface that serves both humans and automated harvesters. The diagram below shows that end-to-end flow, from the moment an instrument or laboratory information management system (LIMS) hands off a payload to the point a researcher or a citation-tracking crawler retrieves it.

The value of drawing the topology explicitly is that each arrow becomes an interface contract. Data does not “flow” between components as an implementation detail; it crosses a boundary where a schema is enforced, a checksum is recorded, or a provenance edge is written. The sections that follow walk each boundary in turn and connect it to the standard that governs it and the implementation guide that details it.

Layered Infrastructure & Service Boundary Mapping #

The transition from ad hoc storage to compliant infrastructure begins with a layered model in which each layer has one responsibility and a stable contract with the layer above it. Collapsing these responsibilities into a single service — the usual failure of a first-generation repository — is what makes later compliance retrofits so expensive.

Persistence layer. At the foundation, immutable object storage and version-controlled repositories establish physical durability. Ceph, AWS S3 with Object Lock, and Git-LFS provide cryptographic integrity guarantees and append-only write semantics that prevent silent corruption and accidental overwrite. The contract this layer exposes upward is narrow and deliberately dumb: given bytes and a checksum, return an opaque, permanent storage key. It knows nothing about metadata or FAIR; that ignorance is what lets it scale to petabytes without becoming a bottleneck.

Semantic indexing layer. Above persistence, an indexing layer translates raw files and binary payloads into queryable structures — a knowledge graph of typed entities, or an inverted index over extracted metadata fields. Separating this from ingestion is critical: compute-heavy metadata extraction and triple materialization must never block a high-throughput instrument stream. This layer is where controlled vocabularies live, and its correctness depends entirely on the transformation rules defined in Metadata Schema Mapping, which converts heterogeneous source records into a canonical internal representation before anything is indexed.

Resolution layer. Persistent identifiers — DOIs, Handles, ARKs — are resolved here into landing pages, metadata documents, or direct download endpoints. Resolution is a distinct layer rather than a property of the API because it has different availability and caching characteristics: identifiers must resolve for decades, long after the ingestion path that minted them has been rewritten.

API and exposure layer. The outermost boundary negotiates content types, enforces rate limits and authentication, and serves versioned contracts to downstream consumers. Its design is detailed in API Routing & Fallbacks, which specifies how requests are routed to primary endpoints, mirrors, or local caches under degradation.

The reason to insist on these boundaries is that FAIR maps onto them almost one-to-one. The structural decomposition in the FAIR Principle Breakdown translates each abstract sub-principle into a concrete component: persistent identifiers land in the resolution layer, machine-readable metadata feeds the index, standardized vocabularies drive schema validation at the ingestion gate, and licensing metadata governs the access decisions made in the exposure layer. Treating compliance requirements as interface contracts rather than administrative guidelines is what lets a platform enforce FAIR at the point of ingestion instead of asserting it in a report afterward.

Core Pipeline Patterns #

Automated compliance cannot rest on synchronous, monolithic scripts that ingest, validate, enrich, and archive in one call stack — a single malformed record or a slow resolver stalls the entire run. Production platforms decompose the pipeline into decoupled stages connected by a durable buffer. Message brokers such as Apache Kafka or RabbitMQ absorb high-velocity instrument streams so that downstream processors consume at a sustainable rate, and orchestration frameworks such as Apache Airflow or Prefect schedule the periodic reconciliation jobs that keep the index synchronized with storage. The trade-offs between event-driven and batch execution, and the retry and checkpoint patterns that make each safe, are covered in depth across the data ingestion and metadata enrichment guides — in particular async batch processing for high-throughput ingestion.

Two properties are non-negotiable at this layer: idempotency (reprocessing a record must not create a duplicate archive entry or mint a second identifier) and explicit failure routing (a record that cannot be processed must land in a dead-letter queue with enough context to remediate, never silently dropped). The pattern below shows a Pydantic V2 validation gate that turns an untrusted inbound payload into a typed, compliance-checked object — the single choke point every record must pass before it is allowed near persistent storage.

from __future__ import annotations

import hashlib
from datetime import datetime, timezone

from pydantic import BaseModel, Field, ValidationError, field_validator


class DatasetRecord(BaseModel):
    """Canonical, compliance-checked representation of an inbound dataset.

    Acts as the ingestion validation gate: a payload that cannot be coerced
    into this model never reaches persistence and is routed to the DLQ.
    """

    dataset_id: str = Field(min_length=1)
    title: str = Field(min_length=3)
    creators: list[str] = Field(min_length=1)  # ORCID iDs or bare names
    checksum_sha256: str = Field(pattern=r"^[0-9a-f]{64}$")
    license_spdx: str  # SPDX identifier, e.g. "CC-BY-4.0"
    created_at: datetime

    @field_validator("created_at")
    @classmethod
    def _tz_aware_utc(cls, v: datetime) -> datetime:
        # Naive timestamps silently compare wrong across timezones; force UTC.
        if v.tzinfo is None:
            raise ValueError("created_at must be timezone-aware")
        return v.astimezone(timezone.utc)


def admit(payload: dict[str, object]) -> DatasetRecord:
    """Validate a raw payload at the ingestion boundary.

    Raises ValidationError on any structural or vocabulary failure so the
    caller can route the offending record to the dead-letter queue.
    """
    record = DatasetRecord.model_validate(payload)
    return record


def compliance_certificate(record: DatasetRecord) -> str:
    """Derive a deterministic certificate hash pinning the validated state.

    Attached to the archived object so audits can prove *what* passed the gate.
    """
    canonical = record.model_dump_json()
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

The Pydantic model is not decoration; it is the enforcement point. Field constraints reject a malformed checksum before a byte is written, the SPDX license field forces a machine-readable rights declaration, and the timezone validator eliminates a whole class of silent comparison bugs. The exact validator library, error-code taxonomy, and test strategy for this gate are specified in Pydantic schema validation; the tabular ingestion path that feeds many of these records is covered in pandas data pipelines, and the instrument-specific extraction that precedes validation in lab notebook parsing.

The second pattern is the consumer loop itself. It must acknowledge a broker message only after the record is durably persisted, and it must route validation failures to a quarantine topic rather than crashing or discarding.

from __future__ import annotations

import logging
from collections.abc import Iterable
from typing import Protocol

from pydantic import ValidationError

logger = logging.getLogger("ingest")


class Broker(Protocol):
    def poll(self) -> Iterable[tuple[str, dict[str, object]]]: ...
    def ack(self, offset: str) -> None: ...
    def to_dlq(self, offset: str, payload: dict[str, object], reason: str) -> None: ...


class Store(Protocol):
    def exists(self, dataset_id: str) -> bool: ...
    def put(self, record: DatasetRecord, certificate: str) -> None: ...


def run_consumer(broker: Broker, store: Store) -> None:
    """Idempotent, at-least-once ingestion consumer with DLQ routing."""
    for offset, payload in broker.poll():
        try:
            record = admit(payload)
        except ValidationError as exc:
            # Permanent, record-level failure: quarantine, do not retry in place.
            broker.to_dlq(offset, payload, reason=exc.json())
            broker.ack(offset)
            continue

        if store.exists(record.dataset_id):
            # Reprocessing is safe: skip the write, still ack to advance.
            logger.info("duplicate %s ignored (idempotent)", record.dataset_id)
            broker.ack(offset)
            continue

        store.put(record, compliance_certificate(record))
        broker.ack(offset)  # ack only after durable write

Because the write is guarded by an existence check and the acknowledgement follows the write, this loop is safe to restart at any point — the defining requirement of an at-least-once system that must never double-archive or double-mint.

Compliance Enforcement as Pipeline Checkpoints #

Compliance embedded into the pipeline topology is durable; compliance applied retroactively is a perpetual backlog. The governing idea is that every dataset entering the archive already carries a verifiable compliance certificate — the hash produced by the validation gate above — so an auditor never has to re-derive whether a record met policy at deposit time.

Concretely, mandates decompose into checkpoints at specific boundaries. FAIR sub-principles are enforced at the ingestion gate (an identifier, machine-readable metadata, a vocabulary-conformant subject, and an explicit license must all be present to pass). Funder mandates layer additional required fields on top: the mapping from an agency policy such as the NIH Data Management and Sharing policy onto concrete metadata is worked through in aligning NIH data-sharing policies with FAIR principles, part of the broader funder mandate alignment work. Institutional rules — retention windows, sensitivity classification, permitted repositories — are expressed as policy objects evaluated at the same gate, following the data governance frameworks model.

When a record fails a checkpoint, it is not dropped: it is routed to a quarantine topic where an automated remediation workflow either applies a deterministic enrichment rule or escalates to human review. The routing itself is shown below.

The license field deserves special attention. A dataset with no rights statement is legally unreusable regardless of how findable it is, so the gate treats a missing or unrecognized SPDX License List identifier as a hard failure. The mechanics of turning an author’s intent into a valid license record — and reconciling it with repository defaults — are detailed in open license configuration.

Identity & Resolution #

Findable data depends on identifiers that resolve reliably for the lifetime of the record, which vastly outlasts any single version of the software that minted them. Research platforms assign DOIs (typically registered through the DataCite Metadata Schema for research outputs), Handles, or ARKs, and each identifier must resolve to a stable landing page, a metadata document, or a download endpoint. Minting is a two-phase operation — reserve a draft identifier, then publish it once the object is durably archived — so that a crash between the two never leaves a live identifier pointing at nothing.

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class MintResult:
    identifier: str
    state: str  # "draft" | "findable"


class RegistryError(RuntimeError):
    ...


def mint_doi(client, record: DatasetRecord, prefix: str) -> MintResult:
    """Two-phase DOI mint: reserve a draft, publish only after archival.

    A draft identifier is inert and free to abandon; publishing is the point
    of no return, so it runs last and is idempotent on the client side.
    """
    draft = client.reserve(prefix=prefix, metadata={
        "titles": [{"title": record.title}],
        "creators": [{"name": c} for c in record.creators],
        "rightsList": [{"rightsIdentifier": record.license_spdx}],
    })
    # ... object is already durably archived before this line ...
    published = client.publish(draft.identifier)  # idempotent: safe to retry
    if published.state != "findable":
        raise RegistryError(f"{draft.identifier} did not reach findable state")
    return MintResult(identifier=published.identifier, state="findable")

Resolution and the API gateway that fronts it are where availability engineering lives. The gateway negotiates content — serving application/ld+json to a machine harvester and HTML to a browser from the same URL — enforces per-consumer rate limits, and, critically, degrades gracefully when an upstream registry is slow or unreachable. The routing rules that make this deterministic (prefer a local metadata cache, fall through to a mirror, and only then query the upstream registry) are specified in API Routing & Fallbacks. Versioned API contracts round out the layer so that a breaking change to an internal schema never disrupts a downstream citation tracker or a partner institution’s harvester.

Failure Modes & Operational Runbook #

A research data platform is judged in production by how it behaves when a dependency degrades, not by how it behaves on the happy path. The table below is the operational core of this overview: the failure modes an on-call engineer will actually see, the signal that detects each, and the first remediation step.

Two failure modes deserve elaboration because they are the ones most often missed until they cause data loss. Schema drift is silent by construction: firmware updates, ELN vendor releases, and evolving community ontologies change the shape of incoming payloads without any error being raised, because the new records are merely different, not malformed. The defense is statistical — compare the field-coverage distribution of each source against a recorded baseline and alert when coverage drops below a configurable threshold, then route the drifted stream to quarantine rather than admitting degraded metadata. Resolver downtime is dangerous because it is intermittent: a DOI resolver that times out for ten minutes must not cause ten minutes of failed deposits. A per-endpoint circuit breaker that opens on elevated error rates and serves from the local PID cache turns an upstream outage into a transparent, replayable delay.

Underpinning the whole runbook is observability. Structured JSON logs, distributed tracing (OpenTelemetry), and metric collection (Prometheus) make every checkpoint and every fallback event visible, and a nightly reconciliation job verifies that every published dataset still has a valid identifier, an intact checksum, and a resolvable metadata endpoint. Detection is what converts these failure modes from outages into routine, remediable events.

Security & Access Control #

FAIR does not mean open to everyone. Human-subjects data, proprietary instrument outputs, and embargoed pre-publication results all require that findability and accessibility be decoupled: a record can be discoverable while its bytes remain gated. Access decisions therefore live in the metadata layer, where license, data-use agreement, sensitivity label, and access tier are declared explicitly and evaluated programmatically. An attribute-based access control (ABAC) model checks contextual claims — user affiliation, project membership, sensitivity classification — before granting read or write, with role-based rules layered on for coarse administrative boundaries. The full policy model, including encryption at rest and in transit, immutable audit logging, and cryptographic signing of metadata records, is specified in Security & Access Control.

Automation ties into institutional identity rather than reinventing it: pipelines authenticate against the institution’s OIDC or SAML provider and use short-lived tokens for service-to-service calls, so a compromised worker cannot hold long-lived credentials. Access state is not static — when a dataset transitions from embargoed to open, an automated workflow rewrites its access control list and triggers re-indexing so discovery reflects the new visibility immediately. Enforcing least privilege across fallback tiers, so that a secondary mirror or local cache never inherits the elevated permissions of the primary endpoint, keeps a degradation event from becoming a privilege-escalation event.

FAQ #

How do I handle a dataset that arrives with a partial or non-standard license field? #

Treat a missing or unrecognized license as a hard validation failure, not a warning. The ingestion gate rejects any record whose license is not a valid SPDX License List identifier and routes it to quarantine, where a remediation rule either maps a known synonym (for example an internal shorthand to CC-BY-4.0) or escalates to the depositor. Admitting a rights-less record makes the data legally unreusable no matter how findable it is; see open license configuration for the reconciliation rules.

What happens when DataCite or another registry is unreachable during minting? #

Nothing is lost. Minting is two-phase — reserve a draft, then publish — and the per-endpoint circuit breaker opens when the registry error rate crosses its threshold. New identifiers are buffered as drafts and existing ones are served from the local PID cache, so deposits continue and the publish step replays automatically once the registry recovers. The routing and fallback specifics are in API Routing & Fallbacks.

How do I detect schema drift before it corrupts the index? #

Do not rely on validation errors — drifted records are different, not malformed. Record a baseline field-coverage distribution per source and compare each incoming batch against it; when coverage for a required field drops below a configurable threshold, alert and route the source to quarantine rather than admitting it. The mapping is then patched following Metadata Schema Mapping.

Why enforce idempotency if the broker already delivers each message once? #

Because durable brokers guarantee at-least-once, not exactly-once, delivery: a consumer crash between a write and its acknowledgement will redeliver the message. Guarding the write with an existence check and acknowledging only after a durable write makes reprocessing a no-op, which is the only safe basis for restarts, replays, and horizontal scaling. The patterns are detailed in async batch processing.

Where should compliance checks live — at ingestion or at publication? #

At ingestion. A record that passes the gate carries a compliance certificate pinning exactly what was validated, so audits never re-derive policy conformance after the fact. Checking only at publication leaves a growing backlog of non-conformant records in the store and turns every audit into a full re-scan. The checkpoint model is described in the compliance section above and in the FAIR Principle Breakdown.

How is access controlled without breaking findability for restricted data? #

Decouple the two. Metadata and its identifier stay discoverable so the record can be cited and requested, while an ABAC policy evaluated in the exposure layer gates the actual bytes on affiliation, project membership, and sensitivity label. Transitions between embargoed and open state trigger an automated ACL rewrite and re-index; the full model is in Security & Access Control.

Related #

• FAIR Principle Breakdown — how each FAIR sub-principle maps to a concrete pipeline component and validation rule.
• Metadata Schema Mapping — declarative crosswalks from source formats into the canonical internal representation.
• API Routing & Fallbacks — deterministic routing, circuit breakers, and cache/mirror fallback for the exposure layer.
• Security & Access Control — ABAC/RBAC, encryption, audit logging, and signed metadata.
• Data Ingestion & Metadata Enrichment — the sibling area covering validation, batch processing, and instrument parsing that feeds this architecture.
• Open Science Infrastructure Planning — governance frameworks, funder mandates, and repository strategy that this architecture must satisfy.

This overview is the map for the whole scientific research data management platform; start here for the topology, then follow any layer into its detailed implementation guide.