Security & Access Control Engineering for FAIR Research Data Workflows #

Implementing robust security and access control inside scientific data pipelines means reconciling two mandates that pull in opposite directions: the FAIR requirement for open, machine-actionable accessibility, and the strict compliance boundaries imposed by institutional review boards, funding agencies, and data-protection regulation. FAIR does not mean public — the Accessible principle is explicit that data may sit behind an authentication and authorization layer as long as the metadata stays resolvable and the access protocol stays open. This page sits inside the Core Architecture & FAIR Mapping overview and owns the enforcement layer those gates depend on: it turns “access control is evaluated before storage, not after exposure” into the zero-trust boundaries, automated credential lifecycles, and policy-as-code that make the guarantee real. It is written for academic IT teams, research data managers, and Python automation engineers who already run an ingestion pipeline and need to know exactly where authentication, authorization, and auditability are implemented — and how each one fails safely. The design constraint throughout is that these controls run as continuous, automated processes, never as manual gatekeeping that would stall ingestion or enrichment.

Concept & Specification: The Accessible Sub-Principles as Enforceable Controls #

The access layer implements a small set of named standards, each of which has an internal implementation guide rather than an outbound link. Federated identity is issued as OAuth 2.0 access tokens carried in the OpenID Connect layer, serialized as signed JSON Web Tokens (JWT) whose signatures verify against a JSON Web Key Set (JWKS). Authorization rules are expressed declaratively in the Open Policy Agent (OPA) engine using the Rego policy language, evaluated at a Policy Decision Point and enforced at a Policy Enforcement Point. Tamper-evident lineage is written as W3C PROV-O provenance backed, where cross-organization verification is required, by W3C Verifiable Credentials. Licensing that gates reuse resolves against the SPDX License List, contributor identity against ORCID, and institutional affiliation against the Research Organization Registry (ROR).

These map directly onto the Accessible sub-principles that the FAIR Principle Breakdown enforces at its validation gate. A1 requires retrieval by identifier over a standardized protocol; A1.1 requires that protocol to be open and free, which the zero-trust layer satisfies by terminating TLS 1.3 rather than any proprietary transport; A1.2 requires the protocol to support authentication and authorization where necessary, which is exactly the ABAC decision this page implements; and A2 requires the metadata to persist even when the data is withdrawn, which the audit ledger and tombstone manifest guarantee. Treating each of these as a typed control — a token that must validate, a policy that must permit, a provenance edge that must be written — is what lets a machine assert access compliance instead of a curator asserting it by hand.

Step-by-Step Implementation #

The access layer advances a request through four ordered controls. Each control implements a group of Accessible sub-principles, and a request that fails any control is rejected and audited rather than silently downgraded, so an unauthorized read can never reach storage.

Step 1 — Zero-trust boundary: identity-aware proxy and edge policy evaluation (A1, A1.1) #

Research data ecosystems rarely live inside a single trusted perimeter. Datasets move between institutional repositories, cloud compute, and third-party analysis platforms, so boundary enforcement begins with explicit network segmentation and an identity-aware proxy that intercepts every request before it reaches storage or compute. Access decisions are evaluated at the edge by an attribute-based access control (ABAC) engine that inspects request context, dataset classification, and user entitlements in real time. Zero-trust here means continuous credential validation, mutual TLS for service-to-service calls, and least-privilege scoping for automated ingestion agents — any deviation from expected token scopes or source ranges suspends the workflow and routes an alert to security operations. The proxy must enforce encryption in transit without becoming a latency bottleneck, which is why policy evaluation is co-located with the gateway rather than round-tripped to a distant authorization service. Because these boundaries wrap the same routing layer described in API routing & fallbacks, the policy decision point must sit before the fallback chain, so a degraded upstream never becomes a path around authorization.

Step 2 — Automated token lifecycle: acquisition, validation, and refresh (A1.2) #

Academic IT teams typically operate federated identity providers that issue OAuth 2.0 or OpenID Connect tokens for researchers, service accounts, and automated agents. Python automation must acquire, rotate, and revoke credentials without human intervention, drawing short-lived tokens from a secret store (HashiCorp Vault, AWS Secrets Manager, or an institutional key-management service) rather than baking them into environment variables. Refresh logic uses exponential backoff with jitter and a circuit breaker so an identity-provider outage does not trigger an authentication storm; on a 401 Unauthorized or 403 Forbidden, the agent attempts a silent refresh, revalidates the new token against the JWKS endpoint, and resumes.

The routine below caches a token until just before expiry, validates its RS256 signature against the JWKS before ever using it, and retries transient failures with resilient backoff. Signature validation is not optional: accepting an unverified JWT is equivalent to trusting anyone who can reach the token endpoint.

from __future__ import annotations

import time
import logging

import jwt
import requests
from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential,
    retry_if_exception_type,
    before_sleep_log,
)

logger = logging.getLogger(__name__)


class TokenManager:
    """Acquire, cache, and cryptographically validate short-lived OAuth 2.0 tokens."""

    def __init__(self, client_id: str, client_secret: str, token_url: str, jwks_url: str) -> None:
        self.client_id = client_id
        self.client_secret = client_secret
        self.token_url = token_url
        self.jwks_url = jwks_url
        self._token_cache: str | None = None
        self._expiry: float = 0.0

    def _fetch_token(self) -> dict[str, object]:
        payload = {
            "grant_type": "client_credentials",
            "client_id": self.client_id,
            "client_secret": self.client_secret,
            "scope": "fair:read fair:write metadata:resolve",
        }
        response = requests.post(self.token_url, data=payload, timeout=10)
        response.raise_for_status()
        return response.json()

    @retry(
        stop=stop_after_attempt(4),
        wait=wait_exponential(multiplier=1, min=2, max=30),
        retry=retry_if_exception_type((requests.HTTPError, requests.ConnectionError)),
        before_sleep=before_sleep_log(logger, logging.WARNING),
    )
    def get_valid_token(self) -> str:
        # Serve from cache until 60s before expiry to absorb clock skew.
        if self._token_cache and time.time() < self._expiry - 60:
            return self._token_cache

        token_data = self._fetch_token()
        access_token = str(token_data["access_token"])
        self._validate_signature(access_token)     # never trust an unverified JWT
        self._token_cache = access_token
        self._expiry = time.time() + float(token_data.get("expires_in", 3600))
        return self._token_cache

    def _validate_signature(self, token: str) -> None:
        signing_key = jwt.PyJWKClient(self.jwks_url).get_signing_key_from_jwt(token)
        jwt.decode(token, signing_key.key, algorithms=["RS256"], audience="research-pipeline-api")

This pattern keeps agents from ever holding long-lived credentials in memory: the cache is short, the refresh is silent, and every token is signature-checked before use. It is the concrete mechanism behind sub-principle A1.2 — the authorization layer FAIR permits without breaching openness.

Step 3 — Policy-as-code authorization: ABAC through Open Policy Agent (A1.2) #

Static role-based access control (RBAC) does not scale across heterogeneous research environments where dataset sensitivity, funder restrictions, and institutional affiliation intersect dynamically. Policy-as-code with Open Policy Agent evaluates declarative Rego rules against every request, deployed as a sidecar or inline middleware that intercepts a pipeline step before transformation, metadata extraction, or cross-repository sync.

A representative Rego policy evaluates user affiliation, dataset classification, and the requested operation, defaulting to deny so an unmatched request is refused rather than admitted:

package fair.pipeline.authz

default allow := false

allow if {
    input.user.role == "researcher"
    input.dataset.classification == "public"
    input.action in ["read", "resolve_metadata"]
}

allow if {
    input.user.role == "principal_investigator"
    input.dataset.classification in ["restricted", "controlled"]
    input.user.institution == input.dataset.owner_institution
    input.action in ["read", "write", "annotate"]
}

# Machine-actionable enforcement: deny anything not explicitly permitted.
deny if {
    not allow
    input.action != "health_check"
}

Because policy is code, it is version-controlled, tested in staging, and promoted through CI/CD, which eliminates the configuration drift and manual-approval bottlenecks that plague hand-edited access-control lists. The classification tags this policy reads — public, restricted, controlled — are the same ones asserted at ingestion, so the boundary defined by your data governance frameworks is enforced end to end rather than restated at every service.

Step 4 — Cryptographic provenance and immutable audit trails (A2, R1.2) #

FAIR workflows must maintain verifiable lineage from raw ingestion through enrichment, transformation, and publication. Cryptographic provenance makes every access event, metadata update, and derivative generation tamper-evident: the pipeline computes a SHA-256 digest at each transformation stage, attaches it to the metadata payload, and appends it to an append-only ledger or immutable object-storage tier. Hash-chaining each entry to the previous one — or issuing W3C Verifiable Credentials for cross-organization consumers — lets a downstream reader verify integrity and access history without trusting a central authority. The access layer cross-references these digests against the policy decision so that unauthorized lineage tampering is caught, not just unauthorized reads. This is what satisfies A2 (metadata survives even when the data is withdrawn) and feeds the R1.2 provenance requirement, and it is the audit substrate that an IRB review or a grant report reconstructs from.

from __future__ import annotations

import json
import hashlib
from datetime import datetime, timezone


def append_audit_entry(
    prev_hash: str,
    actor: str,
    action: str,
    dataset_id: str,
    decision: str,
) -> dict[str, str]:
    """Append a hash-chained audit record; each entry seals the one before it."""
    entry = {
        "prev_hash": prev_hash,                                   # links to prior entry
        "timestamp": datetime.now(timezone.utc).isoformat(),     # tz-aware UTC, never local
        "actor": actor,                                          # ORCID URI or service account id
        "action": action,                                       # read | write | annotate | resolve
        "dataset_id": dataset_id,
        "decision": decision,                                   # permit | deny, from the PDP
    }
    # The entry hash covers prev_hash, so any earlier tampering invalidates the chain.
    entry_hash = hashlib.sha256(json.dumps(entry, sort_keys=True).encode("utf-8")).hexdigest()
    return {**entry, "entry_hash": entry_hash}

Reference: Control-to-Sub-Principle Mapping #

The table below is the canonical mapping this page enforces. Every row is a control a request must pass; the mechanism column names the exact component that implements it. There are no aspirational rows — each corresponds to code or configuration in the access layer.

Error Handling & Edge Cases #

Access failures must be non-blocking for the pipeline as a whole and fully traceable for the individual request. A 401/403 triggers a single silent token refresh and revalidation before the request is retried; a persistent authorization denial is written to the audit chain with the failing policy path and the requesting identity, never surfaced as a bare exception. Token refresh follows jittered exponential backoff capped at a small attempt count so a transient identity-provider outage degrades to queued retries rather than a thundering herd. Because a denied request is still an auditable event, a curator can later reconcile a misclassified dataset instead of discovering a silent gap.

The most dangerous edge case in the authorization layer is the fail-open one: a policy engine that returns “permit” when it cannot reach its data, or a gateway that bypasses the decision point when the sidecar is unhealthy. Guard against it by making the Rego bundle default-deny (as above), by treating an unreachable Policy Decision Point as a deny rather than a skip, and by asserting that the health_check carve-out can never return dataset bytes. For the resolution path, a fallback that serves a cached record must propagate its source and provenance so a registry outage is visible, not laundered into an apparently authoritative 200 OK — the same discipline the resilient router in API routing & fallbacks applies. That router pairs with strict schema validation at the boundary so a fallback response cannot introduce semantic drift:

from __future__ import annotations

import requests
from pydantic import BaseModel, ValidationError
from tenacity import retry, stop_after_attempt, wait_fixed, retry_if_exception_type


class MetadataRecord(BaseModel):
    pid: str
    title: str
    schema_version: str
    access_level: str
    provenance_hash: str


class ResilientMetadataRouter:
    """Resolve a PID across a primary and ordered fallbacks, validating every response."""

    def __init__(self, primary_url: str, fallback_urls: list[str]) -> None:
        self.primary_url = primary_url
        self.fallback_urls = fallback_urls

    @retry(
        stop=stop_after_attempt(3),
        wait=wait_fixed(2),
        retry=retry_if_exception_type((requests.Timeout, requests.HTTPError)),
    )
    def _request_with_timeout(self, url: str, pid: str) -> dict[str, object]:
        response = requests.get(f"{url}/resolve/{pid}", timeout=8)
        response.raise_for_status()
        return response.json()

    def resolve_metadata(self, pid: str) -> MetadataRecord:
        endpoints = [self.primary_url] + self.fallback_urls
        last_error: Exception | None = None
        for url in endpoints:
            try:
                payload = self._request_with_timeout(url, pid)
                return MetadataRecord(**payload)          # reject structurally invalid fallbacks
            except (requests.RequestException, ValidationError) as exc:
                last_error = exc
                continue
        raise RuntimeError(f"All metadata endpoints failed for PID {pid}: {last_error}")

Verification & Testing #

Correctness of an access layer is asserted, not assumed, because the failure mode is silent by definition — an over-permissive policy leaks data without raising an error. The test below pins the two contracts that matter most: an unauthorized action must be denied, and a tampered audit entry must break the hash chain. Wire both into CI so a policy edit or a schema change fails the build instead of quietly widening access.

from __future__ import annotations


def evaluate_policy(user_role: str, classification: str, action: str) -> bool:
    """Local mirror of the Rego default-deny rule, used to pin authorization in tests."""
    if user_role == "researcher" and classification == "public":
        return action in {"read", "resolve_metadata"}
    if user_role == "principal_investigator" and classification in {"restricted", "controlled"}:
        return action in {"read", "write", "annotate"}
    return False


def test_researcher_cannot_write_restricted() -> None:
    """A researcher writing a restricted dataset must be denied (default-deny holds)."""
    assert evaluate_policy("researcher", "restricted", "write") is False


def test_audit_chain_detects_tampering() -> None:
    """Mutating a sealed entry must change its recomputed hash, breaking the chain."""
    entry = append_audit_entry("GENESIS", "https://orcid.org/0000-0002-1825-0097",
                               "read", "ds-001", "permit")
    tampered = {**entry, "decision": "deny"}          # attacker flips the recorded decision
    import json, hashlib
    recomputed = hashlib.sha256(
        json.dumps({k: v for k, v in tampered.items() if k != "entry_hash"},
                   sort_keys=True).encode("utf-8")
    ).hexdigest()
    assert recomputed != tampered["entry_hash"]

Run the suite with pytest -q; a green run is the machine-readable assertion that the default-deny policy and the tamper-evident ledger both hold.

Gotchas & Known Pitfalls #

• Skipping JWT signature validation. Decoding a token to read its claims without verifying the RS256 signature against the JWKS trusts anyone who can forge a payload. Root cause: jwt.decode(..., options={"verify_signature": False}) copied from a debugging snippet. Fix: always resolve the signing key via PyJWKClient and pin algorithms=["RS256"] and the expected audience, as TokenManager does.
• Fail-open authorization. A gateway that admits the request when the Policy Decision Point is unreachable converts an outage into a data breach. Root cause: treating “no decision” as “no objection.” Fix: make the Rego bundle default-deny and configure the enforcement point to deny on timeout or engine error.
• RBAC-only access control. Static roles cannot express “same institution as the dataset owner” or “funder permits reuse,” so teams bolt on ad-hoc exceptions that drift. Root cause: modeling access as identity alone. Fix: evaluate attributes (user, dataset classification, action, affiliation) in ABAC, keeping roles as one input among several.
• Long-lived tokens in environment variables. A static bearer token in .env or a CI secret that never rotates is a standing credential an attacker can replay indefinitely. Root cause: convenience over lifecycle. Fix: mint short-lived client-credentials tokens from a secret store at run time and cache them only until just before expiry.
• Timezone-naive audit timestamps. An audit entry without an offset makes embargo windows and retention math non-deterministic across regions and can reorder a chain. Root cause: datetime.now() instead of an aware UTC value. Fix: stamp every entry with datetime.now(timezone.utc) before hashing, as the ledger does.

Frequently Asked Questions #

Does putting data behind authentication break FAIR compliance? #

No. The Accessible principle governs how data and metadata are retrieved, not whether they are public. Sub-principle A1.2 explicitly allows an authentication and authorization layer where access must be restricted, and A2 requires only that the metadata stay resolvable even when the data itself is withheld. A dataset can be fully FAIR while under embargo or restricted to authorized institutions, provided the protocol is open (A1.1) and the metadata persists. The FAIR Principle Breakdown enforces exactly this distinction at its validation gate.

Should I use RBAC or ABAC for research dataset access? #

Use attribute-based access control (ABAC) as the decision model and keep roles as one attribute among several. Research access rules depend on dataset classification, funder restrictions, and whether the requester shares the owner’s institution — conditions that static roles cannot express without a combinatorial explosion of exceptions. Encoding them as Rego policy in Open Policy Agent keeps the rules declarative, version-controlled, and testable, and it evaluates the request context at the edge instead of hard-coding entitlements into application code.

Where should the policy decision be evaluated in the pipeline? #

Before storage or compute access, at an identity-aware proxy co-located with the gateway, and ahead of any resilient-routing fallback chain. Evaluating authorization at the edge keeps latency low and guarantees that a degraded upstream never becomes a path around the Policy Decision Point. The one carve-out — a health_check action — must be structurally incapable of returning dataset bytes.

How do I make an access log admissible for an IRB or grant audit? #

Write an append-only, hash-chained audit trail where each entry seals the previous one with a SHA-256 digest and carries a timezone-aware UTC timestamp, the actor’s ORCID or service identity, the action, and the policy decision. Because any retroactive edit changes the recomputed hash and breaks the chain, the ledger is tamper-evident without a central trust authority, and any historical state can be reconstructed for review. For cross-organization verification, issue the same evidence as W3C Verifiable Credentials.

Related Guides #

• Core Architecture & FAIR Mapping — the parent overview mapping this access layer onto storage, indexing, and resolution service boundaries.
• FAIR Principle Breakdown — the validation gate where these ABAC/RBAC and encryption controls enforce the Accessible sub-principles.
• API routing & fallbacks — the resilient resolution control plane that authorization must sit in front of.
• Setting up secure data boundaries for academic IT — the step-by-step boundary build that operationalizes these controls in an institutional deployment.
• Data governance frameworks — the classification and consent model whose tags these policies read.