Overview

FairDataHive is a self-hosted research data management system that lets teams register, describe, version, publish, and share datasets following open standards. It replaces ad-hoc file shares and spreadsheet catalogs with a structured, searchable, and interoperable repository.

Catalog Management

Create and publish versioned dataset records with rich DCAT metadata.

Dual-mode Search

Keyword search (FTS + tag matching) and semantic vector search over published catalog resources.

FAIR Scoring

Automatic per-resource FAIR assessment with actionable improvement suggestions.

RDF Export

Machine-readable Turtle and JSON-LD exports per resource and catalog-wide.

OAI-PMH

Standards-based harvesting endpoint for external portal integration.

Access Control

Private resources with owner-managed access request workflows.

Architecture

FairDataHive follows a layered service architecture. The backend stack runs in Docker Compose; the React UI runs locally with Vite during development, or as static files served by the API after npm run build. There is no containerized Vite service — that pattern duplicated port 5173 and caused a second dev server on 5174.

LayerTechnologyRole
APIFastAPI 0.111, Python 3.12REST API, business logic, async I/O
ORM / MigrationsSQLAlchemy 2.0 async, AlembicSchema management, typed queries
Primary databasePostgreSQL 16 + pgvectorRelational storage, FTS indexes, vector search
Object storageRustFS (S3 API; also MinIO/AWS/Ceph)File uploads, presigned download URLs
CacheRedis 7Search result caching, TTL invalidation
Embeddingssentence-transformers (in-process)768-dim vectors for semantic search
RDFrdflib 7Turtle / JSON-LD / OAI-DC generation
SHACLpyshacl 0.25DCAT-AP shape validation
UIReact 18, Vite, TypeScriptBrowser data management interface
AuthKeycloak 24 (optional)OIDC/JWT identity provider

Host port mapping (Docker Compose)

ServiceContainer portHost portNotes
API80008002Docs at /docs; Swagger at /swagger; UI at /ui
PostgreSQL54325434Avoids conflict with local Postgres on 5432
RustFS S390009002Set S3_PUBLIC_ENDPOINT=http://localhost:9002
RustFS console90019003Loopback only; local login rustfsadmin / rustfsadmin
Redis6379noneInternal compose network only
Keycloak80808080Optional: docker compose --profile auth up -d auth
React UI (dev)—5173Local only: cd frontend && npm run dev (strictPort)

Service topology

Browser
  │
  ├── :5173   Vite dev server (host: npm run dev) ──────────┐
  │              /docs → docs/index.html (local)           │
  │              proxies /api, /swagger → :8002            │
  ├── :8002/ui   production build (frontend/dist)          ├──► API (Docker :8002)
  └── :8002/docs guide · :8002/swagger OpenAPI             │
                │                                           │
                ├── postgres   (host :5434)
                ├── rustfs     (host :9002 / :9003)
                ├── redis      (internal)
                └── auth       (optional profile, :8080)

docker compose up -d  →  backend only (no Node/Vite container)

Repository layout

fairdatahive/
├── app/
│   ├── config.py          ← Pydantic Settings (all env vars)
│   ├── database.py        ← Async SQLAlchemy engine
│   ├── main.py            ← FastAPI factory, middleware, router mount
│   ├── core/              ← Auth, exceptions, pagination, constants
│   ├── models/            ← SQLAlchemy ORM models (source of truth)
│   ├── schemas/           ← Pydantic v2 request / response contracts
│   ├── routers/           ← HTTP routing (thin layer, no logic)
│   └── services/          ← Business logic (resource, search, fair…)
├── alembic/               ← Database migrations
├── frontend/              ← React + TypeScript UI (Vite, scripts/dev-server.mjs)
├── docs/                  ← This documentation (index.html)
├── static/                ← context.jsonld, dcat_ap_shapes.ttl
├── templates/             ← Jinja2 HTML landing page
├── tests/                 ← pytest-asyncio test suite
├── docker/                ← entrypoint.sh
├── docker-compose.yml
├── Dockerfile
└── .env.example

Quick Start

Prerequisites

  • Docker ≥ 24 and Docker Compose v2
  • Ports 5173, 8002, 9002, 9003, 5434 available on the host (Redis is internal-only)

Start the full stack

  1. Clone the repository and enter the project directory.
    bashgit clone <repo-url>
    cd fairdatahive
  2. Create your environment file.
    bashcp .env.example .env

    The defaults work out of the box for local development. See Configuration for a full reference.

  3. Build and start all services.
    bashdocker compose build
    docker compose up -d

    Starts postgres, rustfs, redis, and api only — not a Vite/Node UI container. The API runs Alembic migrations on startup; wait ~15 seconds on first boot.

  4. Verify the API is healthy.
    bashcurl http://localhost:8002/api/v1/health
    # {"status":"ok","components":{"database":"ok","cache":"ok","storage":"ok"}}
  5. Start the React UI (separate terminal).
    bashcd frontend
    cp .env.example .env
    npm install
    npm run dev

    Open http://localhost:5173. Vite proxies /api, /swagger, and /openapi.json to :8002. Project documentation is served at /docs from the docs/ folder. Only one dev server should run on port 5173; npm run dev clears any orphan on 5174 automatically.

  6. Optional — production-like UI without Vite.
    bashcd frontend && npm run build
    docker compose build api && docker compose up -d --force-recreate api

    Open http://localhost:8002/ui/ (root / redirects there).

Key URLs after startup

ServiceURL
React UI (dev)http://localhost:5173 — npm run dev in frontend/
React UI (built)http://localhost:8002/ui/ — after npm run build
Documentationhttp://localhost:5173/docs (or :8002/docs via API)
Swagger / OpenAPIhttp://localhost:8002/swagger (proxied from Vite as /swagger)
RustFS consolehttp://localhost:9003 — rustfsadmin / rustfsadmin (loopback)
Catalog (Turtle)http://localhost:8002/api/v1/catalog.ttl
OAI-PMH endpointhttp://localhost:8002/api/v1/oai?verb=Identify
Prometheus metricshttp://localhost:8002/metrics
ℹ

Dev auth mode — the default .env sets DEV_AUTH_ENABLED=true, which bypasses JWT verification and treats every request as a fixed dev-user. This makes local development frictionless. Disable it for any shared or production deployment.

⚠

One UI dev server — use only http://localhost:5173. If you previously saw the same app on 5174, a second Vite had started when 5173 was busy. Run npm run dev:stop in frontend/, then npm run dev. Do not run a Docker Vite container alongside local npm run dev (that service was removed from Compose).

Configuration

All configuration is driven by environment variables (loaded from .env by Pydantic Settings). No code changes are needed to adapt FairDataHive to your environment.

Database

VariableDefaultDescription
DATABASE_URLpostgresql+asyncpg://fairdatahive:fairdatahive@postgres:5432/fairdatahiveAsync PostgreSQL connection string

Object storage (RustFS / S3)

VariableDefaultDescription
S3_ENDPOINThttp://rustfs:9000S3 endpoint used by the API container (Docker hostname)
S3_PUBLIC_ENDPOINThttp://localhost:9002Browser-facing URL embedded in presigned download links (host port from compose).
S3_ACCESS_KEYrustfsadminRestricted application credential (not the server root in production)
S3_SECRET_KEYrustfsadminApplication credential secret
S3_BUCKETfairdatahive-catalogBucket name for file uploads
S3_REGIONus-east-1SigV4 signing region
S3_ADDRESSING_STYLEpathPath-style addressing for self-hosted S3
PROVISION_STORAGEtrueEntrypoint creates the bucket if missing
MAX_UPLOAD_SIZE_MB500Maximum per-file upload size in megabytes
PRESIGNED_URL_EXPIRY_SECONDS3600Presigned download URL lifetime

Legacy MINIO_* names are accepted for one transition release; when both are set, S3_* wins. Cutover from an existing MinIO volume is documented in docs/storage-migration.md.

Authentication

VariableDefaultDescription
KEYCLOAK_URLhttp://auth:8080Keycloak base URL for JWKS fetching
KEYCLOAK_REALMfairdatahiveRealm name
DEV_AUTH_ENABLEDtrue in .env.exampleSkip JWT and use a fixed dev identity (dev-user). Set false for production.
DEV_AUTH_SUBdev-userSubject claim used when dev auth is active
DEV_AUTH_NAMEDevelopment UserDisplay name for the dev user

Search and catalog

VariableDefaultDescription
REDIS_URLredis://redis:6379/0Redis connection URL for search result caching
EMBEDDING_MODELmulti-qa-mpnet-base-dot-v1sentence-transformers model for semantic search
SEMANTIC_SEARCH_THRESHOLD0.30Minimum cosine similarity for a semantic hit
FAIR_SCORE_PUBLISH_THRESHOLD0.40Minimum overall FAIR score required to publish

CORS

VariableDefault
CORS_ORIGINShttp://localhost:5173,http://localhost:3000,http://localhost:8002

Optional integrations

VariableDescription
SMTP_HOST / SMTP_PORT / SMTP_USER / SMTP_PASSWORDEmail delivery for access notifications
DATACITE_REPO_ID / DATACITE_PASSWORD / DATACITE_DOI_PREFIXDataCite Fabrica API credentials for DOI minting
DATACITE_API_URLDefaults to https://api.test.datacite.org (sandbox)

Deployment

FairDataHive is designed as a self-hosted data management service: metadata and search live in PostgreSQL, binaries in S3-compatible object storage, and the API orchestrates FAIR cataloguing, access control, and interoperability endpoints. This section describes how to deploy that stack reliably for lab, consortium, or institutional use — not only “start containers”, but the operational choices that keep catalogs findable, downloadable, and auditable over time.

Topology and components

Treat FairDataHive as a small platform, not a single process. Each component has a distinct failure domain and scaling profile:

ComponentRolePersistenceNotes
API (FastAPI)REST, RDF, OAI-PMH, presigned URLs, FAIR scoringStatelessCPU/RAM for embeddings on semantic search; rebuild image on release
PostgreSQL 16 + pgvectorCatalog metadata, FTS, vector index, auditDurable volumeBack up regularly; extensions required at init
RustFS / S3Uploaded files and ZIP extractsDurable volume or external bucketMust be reachable from API and browsers (see public endpoint)
RedisSearch result cacheOptional persistenceSafe to flush; cache rebuilds on traffic
Keycloak (optional)OIDC/JWT issuerIts own DB in real deploymentsCompose profile auth is dev-oriented only
React UIHuman catalog interfaceStatic files in frontend/distServed by API at /ui or separate CDN

The bundled docker-compose.yml maps host ports to avoid clashes on developer machines (8002 API, 5434 Postgres, 9002/9003 RustFS). In production behind a reverse proxy you typically expose 443 only and keep data services on an internal network.

Prerequisites

  • Container runtime — Docker 24+ and Compose v2 (or an equivalent orchestrator).
  • Host resources — minimum 4 GB RAM for a pilot (8 GB+ recommended once semantic search and concurrent uploads matter). Disk: Postgres + object storage grow with catalog size; plan object storage separately from DB.
  • Network — stable DNS for the public catalog URL; outbound HTTPS if using DataCite or external IdP.
  • Secrets store — production credentials must not live in git; use a vault, compose secrets, or orchestrator secrets.
  • TLS termination — HTTPS at the edge (reverse proxy or load balancer) for UI, API, and presigned download URLs.

Deployment models

Single-host Compose

Fastest path: one VM runs Postgres, RustFS, Redis, and API together. Ideal for pilots, lab deployments, and consortium sandboxes.

Compose + external managed services

Run only the API container (or API + Redis) and point DATABASE_URL and S3 settings at managed RDS/Cloud SQL and institutional S3.

Kubernetes / Nomad

Split stateful sets (Postgres operator, S3-compatible object store) from a stateless API Deployment. Use Ingress for TLS and path routing to /api/v1, /ui, /docs.

Regardless of model, preserve the two-endpoint S3 pattern: an internal endpoint for the API container and a public endpoint embedded in presigned links returned to browsers. Mixing these is the most common production defect.

Docker Compose (single host)

The reference stack is defined in docker-compose.yml. Migrations run automatically when the API container starts (RUN_MIGRATIONS=true in the entrypoint).

  1. Prepare configuration.
    bashcd fairdatahive
    cp .env.example .env
    # Edit .env — see “Production environment” below
  2. Build the API image (after any code or dependency change):
    bashdocker compose build api
  3. Start data services and wait for Postgres.
    bashdocker compose up -d postgres rustfs redis
    docker compose ps   # postgres should show “healthy”
  4. Start the API.
    bashdocker compose up -d api
    docker compose logs -f api   # watch migrations, then “Application startup complete”
  5. Verify health and catalog surface.
    bashcurl -s http://localhost:8002/api/v1/health | jq .
    curl -s http://localhost:8002/api/v1/oai?verb=Identify | head

    Open documentation and Swagger on the host port you publish.

⚠

Do not run a Dockerized Vite dev server in production. Use a static build served at /ui (see below). A second Node container caused duplicate ports and root-owned caches in development; it is intentionally removed from Compose.

Serving the React UI

The catalog UI is a static SPA. For production, build once and let the API serve it from frontend/dist (mounted when the directory exists at image build time or copied in).

bashcd frontend
npm ci
npm run build
cd ..
docker compose build api
docker compose up -d --force-recreate api

Users open https://<your-host>/ui/ (the API redirects / when the build is present). Set CORS_ORIGINS to your public UI origin if the UI is hosted on another hostname; when served from the same API host, CORS is less critical but token-based API calls from browsers still require correct origins if split.

For CDN deployment, upload frontend/dist/* and point the CDN at the same VITE_API_BASE public API URL used at build time, or rebuild with VITE_API_BASE=https://catalog.example.org/api/v1.

Object storage

RustFS in Compose (default)

The API talks to RustFS on the internal Docker network (S3_ENDPOINT=http://rustfs:9000). Browsers must never receive that hostname in presigned URLs. Sign against S3_PUBLIC_ENDPOINT so the SigV4 host matches what the browser hits.

VariableCompose / devProduction example
S3_ENDPOINThttp://rustfs:9000Internal service URL (K8s DNS, VPC endpoint)
S3_PUBLIC_ENDPOINThttp://localhost:9002https://objects.example.org
S3_BUCKETfairdatahive-catalogInstitutional bucket name; create before first upload

After changing S3_PUBLIC_ENDPOINT, recreate the API container. Test by publishing a resource, downloading a file, and confirming the presigned URL host matches what clients can resolve. Keep the RustFS console private.

External S3 (AWS, Ceph, institutional cluster)

FairDataHive uses S3-compatible APIs via aioboto3. Point S3_ENDPOINT at the vendor endpoint, set keys and bucket, and set S3_PUBLIC_ENDPOINT to the same public base users hit (or CloudFront in front of the bucket). Ensure CORS on the bucket allows GET from your catalog origin if browsers fetch objects directly. Set PROVISION_STORAGE=false when CreateBucket is denied.

Database and migrations

PostgreSQL must include the vector extension (image pgvector/pgvector:pg16 in Compose). The API runs alembic upgrade head on startup when RUN_MIGRATIONS=true.

  • First deploy — start Postgres, then API once; confirm logs show migrations applied.
  • Upgrades — pull new image, run docker compose up -d --force-recreate api; entrypoint migrates before serving traffic.
  • Manual migration — docker compose run --rm --entrypoint alembic api upgrade head
  • Backup — snapshot the postgres-data volume or use pg_dump on a schedule; test restore on a staging instance before you need it.
  • Connection string — use postgresql+asyncpg:// form in DATABASE_URL; pool sizing is handled by SQLAlchemy defaults unless you tune at the proxy layer (PgBouncer recommended for large fleets).
ℹ

Metadata tombstones remain in Postgres when resources are deleted; files are removed from object storage. Backup policy should cover both DB and bucket for full recovery.

Authentication in production

Disable development authentication everywhere except local laptops:

envDEV_AUTH_ENABLED=false
KEYCLOAK_URL=https://idp.example.org
KEYCLOAK_REALM=fairdatahive

The API validates JWTs via JWKS (PyJWT, RS256, issuer checked). Configure your IdP (Keycloak or institutional OIDC) to issue tokens with a stable sub claim — that value becomes owner_sub / publisher_sub on records. The Compose auth profile (docker compose --profile auth up -d auth) runs Keycloak in start-dev mode; for production, run Keycloak (or another IdP) as its own hardened deployment with a dedicated database and TLS.

UI users paste or store access tokens in the catalog header, or you integrate your institution’s OAuth2 proxy in front of the UI and API. Plan token lifetime and refresh according to your security policy; FairDataHive does not implement refresh-token flows itself.

Reverse proxy and TLS

Expose a single public hostname (e.g. catalog.example.org) and route paths to the API container. Typical path map:

PathBackend
/api/v1/*API (catalog REST)
/ui/*Static UI (or dedicated CDN origin)
/docsProject documentation (StaticFiles)
/swagger, /openapi.jsonOpenAPI UI and schema
/metricsPrometheus (restrict by network ACL)
/api/v1/oaiOAI-PMH harvesters (allow institutional IPs)

Set the public site URL in configuration:

envBASE_URL=https://catalog.example.org
CORS_ORIGINS=https://catalog.example.org

Terminate TLS at the proxy (nginx, Traefik, Caddy, cloud LB). Increase client_max_body_size (nginx) to at least your MAX_UPLOAD_SIZE_MB plus overhead for multipart uploads.

Production environment

Example production-oriented .env (adjust hostnames and secrets):

env# Core
DATABASE_URL=postgresql+asyncpg://fairdatahive:***@postgres:5432/fairdatahive
REDIS_URL=redis://redis:6379/0
BASE_URL=https://catalog.example.org
RUN_MIGRATIONS=true

# Object storage
S3_ENDPOINT=http://rustfs:9000
S3_PUBLIC_ENDPOINT=https://objects.example.org
S3_ACCESS_KEY=***
S3_SECRET_KEY=***
S3_BUCKET=fairdatahive-catalog
S3_REGION=us-east-1
S3_ADDRESSING_STYLE=path
PROVISION_STORAGE=true
MAX_UPLOAD_SIZE_MB=500
PRESIGNED_URL_EXPIRY_SECONDS=3600

# Auth — required in production
DEV_AUTH_ENABLED=false
KEYCLOAK_URL=https://idp.example.org
KEYCLOAK_REALM=fairdatahive
CORS_ORIGINS=https://catalog.example.org

# Search / FAIR
EMBEDDING_MODEL=multi-qa-mpnet-base-dot-v1
FAIR_SCORE_PUBLISH_THRESHOLD=0.4
SEMANTIC_SEARCH_THRESHOLD=0.30

# Optional
SMTP_HOST=smtp.example.org
SMTP_USER=***
SMTP_PASSWORD=***
EMAIL_FROM=noreply@catalog.example.org
DATACITE_REPO_ID=***
DATACITE_PASSWORD=***
DATACITE_DOI_PREFIX=10.12345
DATACITE_API_URL=https://api.datacite.org
!

Never deploy with default RustFS credentials (rustfsadmin) or DEV_AUTH_ENABLED=true on a network reachable by others. Rotate Postgres and Redis passwords; restrict admin consoles (RustFS console, Keycloak) to management VPNs.

Operations: backup, upgrade, monitoring

Backup strategy

  • PostgreSQL — nightly logical dump or volume snapshot; retain 30+ days for catalog compliance.
  • Object storage — versioning/replication on the bucket; lifecycle rules for cost.
  • Configuration — version-control .env templates without secrets; store secrets in a vault.

Upgrades

  1. Snapshot DB and note current image tag.
  2. Build/pull new API image; run migrations on staging first.
  3. docker compose up -d --force-recreate api (or rolling update in K8s).
  4. Smoke-test: health, search, upload, download presign, OAI Identify.
  5. Rebuild UI if API paths or env changed: npm run build then recreate API.

Monitoring

The API exposes Prometheus metrics at /metrics (instrumentator). Scrape from your monitoring network only. Combine with:

  • GET /api/v1/health — database and Redis connectivity
  • Container logs — structured via application logging; watch migration and presign errors
  • Disk usage on Postgres and object storage volumes
  • Redis memory — eviction is acceptable for cache but indicates heavy search load

Semantic search loads the sentence-transformers model on first use; expect a CPU/RAM spike after cold start — plan readiness probes with generous start_period if you autoscale API pods.

Security checklist

AreaAction
AuthenticationDEV_AUTH_ENABLED=false; institutional OIDC; short-lived tokens
TransportTLS 1.2+ everywhere; HSTS on public catalog host
Object storageUnique keys; bucket policy least privilege; correct public endpoint
DatabaseStrong password; no public port; encrypted volume at rest if required by policy
NetworkRedis and Postgres internal-only; restrict /metrics and the RustFS console
UploadsEnforce MAX_UPLOAD_SIZE_MB; scan at proxy or storage layer if policy requires AV
Private resourcesReview access-request workflow; enable SMTP for auditability of grants
DependenciesRebuild images on security patches; pin base image digests in regulated environments

For incident response, catalog metadata and audit logs live in Postgres; download events are recorded per distribution. Pair FairDataHive logs with reverse-proxy access logs for a full trace of who retrieved which presigned URL.


React UI

The frontend is a standalone Vite + React 18 + TypeScript application located in frontend/. It communicates entirely through the REST API and requires no separate backend of its own.

How to run the UI

ModeCommandURL
Development (recommended)npm run devhttp://localhost:5173 — hot reload, API proxy
Production-likenpm run build then recreate APIhttp://localhost:8002/ui/
Preview buildnpm run previewhttp://localhost:4173 — static preview only

Development server

bash# Terminal 1 — backend
cd fairdatahive && docker compose up -d

# Terminal 2 — frontend
cd fairdatahive/frontend
cp .env.example .env     # VITE_DEV_AUTH=true to match API dev auth
npm install
npm run dev              # guarded by scripts/dev-server.mjs → http://127.0.0.1:5173

vite.config.ts sets strictPort: true on port 5173 and binds 127.0.0.1. The npm run dev script stops any orphan process on 5174 before starting (legacy second Vite instances).

Production build

bashcd fairdatahive/frontend
npm run build            # outputs to frontend/dist/

cd ..
docker compose build api && docker compose up -d --force-recreate api

When frontend/dist/ exists, the API mounts it at /ui and redirects / to http://localhost:8002/ui/ without a separate Node process.

Stop / restart dev server

bashcd fairdatahive/frontend
npm run dev:stop   # frees 5173 and any orphan 5174
npm run dev        # single Vite on http://127.0.0.1:5173 only

Frontend environment variables

VariableDefaultDescription
VITE_API_BASEemptyFull API base URL. Leave empty to use the Vite proxy (/api → localhost:8002).
VITE_DEV_AUTHtrueWhen true, seeds the token field with dev on first load.
VITE_API_PROXYhttp://localhost:8002Proxy target for vite.config.ts when overriding the default API port.

UI Usage Guide

Header controls

The top bar contains navigation links, a scope selector, and an authentication token field.

  • Scope selector — switches between the public catalog (all published resources) and a named project scope. When a project ID is entered, browse, search, and create operations are scoped to that project.
  • Token field — paste a JWT access token here for Keycloak-authenticated requests. With DEV_AUTH_ENABLED=true, this field can be left empty or set to any value.

Search

The Search page supports three modes selectable from the dropdown next to the search box. Only published, non-private resources in the current scope are searchable. See Search engine for implementation detail.

  • Auto (default) — keyword search first; if fewer than 3 hits, semantic results are merged in.
  • Keyword — best for titles, themes, and keyword tags (including short tags like a or comma-separated a,b,c,d). Uses PostgreSQL full-text search plus a token fallback when FTS misses (e.g. single-letter English stop words).
  • Semantic — best for natural-language questions (e.g. “air pollution readings from city sensors”). Matches by meaning via embeddings stored at publish time; requires PostgreSQL + pgvector.

Facet filters for theme, license, and format appear once a query is submitted.

Creating a resource

The Create page registers a new DCAT dataset in the current scope. You need at least one distribution to publish; metadata and files can be combined in a single save.

  1. Navigate to Create in the top navigation.
  2. Fill in the metadata form.
    • Title — display name in the catalog and search results.
    • Description (min 50 characters) — what the data is and how to use it.
    • Keywords (min 2, comma-separated) — discovery tags for search and facets.
    • Theme — research domain from the catalog vocabulary.
    • License — SPDX-style licence; required before publish.
    • Private resource — when checked, downloads require owner approval via access requests.

    Additional fields (provenance, spatial/temporal coverage, DOI, language) are available via the API when creating or patching a resource.

  3. Add at least one distribution. Each option adds a separate dcat:Distribution; you may use one or several.
    • File upload — data stored in FairDataHive (S3/RustFS). Files upload as you select them; ZIP archives are unpacked and each member can be linked. Download on the resource page uses a presigned URL.
    • External URL — a link to data hosted elsewhere (Zenodo, Figshare, a lab file server, a landing page). Recorded as access_url only. Use for static files or human-oriented pages, not for describing a programmatic API.
    • API / data service URL — the root of a REST or other machine-facing service. Recorded as both access_url (where to reach the data) and a linked dcat:DataService with endpoint_url (for RDF export and harvesters). The Create form sets both to the same URL. Use when consumers call an API rather than download a fixed file.
    ↔

    External URL vs API — same hostname, different intent: external = “get the file or page here”; API = “access via a data service”. Example external: https://zenodo.org/…/files/data.csv. Example API: https://api.example.com/v1/measurements.

  4. Save as draft or publish immediately. Enabling Publish immediately runs the publish guard after saving (licence, at least one distribution, FAIR score ≥ threshold). If the guard fails, the record stays a draft and failing checks are shown on the resource page.

Resource detail page

The resource page shows:

  • Full metadata and version history (sidebar)
  • FAIR score breakdown with suggestions (sidebar)
  • Distributions with Download buttons (generates a presigned URL)
  • Actions: Publish (for drafts), Landing page link, RDF (Turtle) link
  • Access request button (for private resources the user cannot download)

API — Catalog

All API routes are prefixed with /api/v1.

GET/catalogList up to 100 published resources. Filter by ?scope=.
GET/catalog.ttlFull catalog as W3C DCAT Turtle.
GET/catalog.jsonldFull catalog as DCAT JSON-LD.
GET/catalog/statsAggregate counts — resources, versions, distributions, by-theme, by-license.
GET/licensesSupported license vocabulary (SPDX-based).
GET/themesAvailable research domain themes.
GET/healthLiveness probe — reports database and cache status.

API — Resources

POST/resourcesCreate a new resource (draft). Returns a ResourceVersionRead.auth
GET/resourcesPaginated list of published resources. Supports ?scope=&page=&size=.
GET/resources/{version_id}Get a specific version. Supports content negotiation: text/turtle or application/ld+json.
PATCH/resources/{version_id}Update a draft's metadata. Only the owner may patch.auth
POST/resources/{version_id}/publishPublish a draft. Runs the publish guard (license + distribution + FAIR score).auth
POST/resources/{version_id}/deprecateMark a version as deprecated.auth
GET/resources/{base_id}/versionsAll versions of a resource ordered by date.
GET/resources/{base_id}/latestRedirect (307) to the current published version.
POST/resources/{base_id}/versionsCreate a new version from an existing published resource.auth
DELETE/resources/{base_id}Tombstone a resource — metadata is retained, files and downloads are removed.auth
GET/resources/{version_id}/fair-scoreCompute and return the FAIR score for a version.
GET/resources/{version_id}.ttlResource metadata as Turtle.
GET/resources/{version_id}.jsonldResource metadata as JSON-LD.
GET/resources/{version_id}/landingHTML landing page with embedded JSON-LD and FAIR score.
GET/resources/{version_id}/dcat-ap-reportSHACL validation report (Turtle) against DCAT-AP 3 shapes.
POST/resources/{version_id}/mint-doiMint a DOI via the DataCite Fabrica API.auth

Create resource — request body

json{
  "scope": "public",               // "public" or a project ID string
  "title": "Urban air quality measurements",
  "description": "Hourly PM2.5 and NO2 readings from city sensor stations…",
  "keywords": ["air quality", "PM2.5", "sensors"],
  "theme": "EarthScience",
  "license_id": "CC-BY-4.0",
  "language": "en",
  "provenance": "Collected by a municipal sensor network, 2024…",
  "is_private": false,
  "datasets": [
    {
      "distributions": [
        {
          "dist_type": "upload",
          "upload_token": "public/draft-dev-user-abc/abc123-data.csv",
          "media_type": "text/csv",
          "checksum_sha256": "…",
          "byte_size": 45231
        }
      ]
    }
  ]
}
⚠

Publish guard — a draft cannot be published unless all three conditions are met: (1) a license_id is set, (2) at least one distribution exists, and (3) the overall FAIR score ≥ FAIR_SCORE_PUBLISH_THRESHOLD (default 0.40).

API — Files

Files are uploaded independently before (or during) resource creation. The returned upload_token is the S3 object key and must be included in a DistributionCreate payload when creating or patching a resource.

POST/files/uploadMultipart file upload. Returns object key, SHA-256 checksum, byte size, and extracted files for ZIPs.auth
GET/distributions/{id}/downloadGenerate a presigned download URL. Enforces authentication and access rights for private resources.

Upload flow

bash# 1. Upload the file
curl -F "file=@data.csv" \
     "http://localhost:8002/api/v1/files/upload?scope=public"

# Response:
# { "upload_token": "public/draft-dev-user-abc/8fa3-data.csv", … }

# 2. Reference the token in the resource payload
# "distributions": [{ "dist_type": "upload", "upload_token": "public/…" }]

ZIP archives are automatically extracted. Each member file is uploaded individually and returned as extracted_files in the response. You can add each extracted file as a separate distribution.

API — Access Control

GET/resources/{version_id}/accessCheck download access for the current user.
POST/access-requestsSubmit an access request to the resource owner.auth
GET/access-requests/incomingList access requests for resources the current user owns.auth
GET/access-requests/outgoingList the current user's own submitted requests.auth
POST/access-requests/{id}/acceptAccept a request — adds requester's sub to access_rights.auth
POST/access-requests/{id}/rejectReject a pending request.auth
DELETE/access-requests/{id}/revokeRevoke a previously accepted request.auth

Access requests expire automatically after ACCESS_REQUEST_EXPIRY_DAYS (default 30). A background task runs every 6 hours to purge expired pending requests. When a request is accepted or rejected, an in-app notification is created for the requester. If SMTP is configured, an email is also sent.

API — Interoperability

GET/oaiOAI-PMH endpoint. Supports all 6 verbs: Identify, ListMetadataFormats, ListSets, ListIdentifiers, ListRecords, GetRecord. Formats: oai_dc, dcat.
GET/api/v1/context.jsonldShared JSON-LD context document.

OAI-PMH usage examples

bash# Identify the repository
curl "http://localhost:8002/api/v1/oai?verb=Identify"

# List all records (Dublin Core)
curl "http://localhost:8002/api/v1/oai?verb=ListRecords&metadataPrefix=oai_dc"

# Harvest a specific scope as a set
curl "http://localhost:8002/api/v1/oai?verb=ListRecords&metadataPrefix=dcat&set=project-alpha"

# Incremental harvest from a date
curl "http://localhost:8002/api/v1/oai?verb=ListRecords&metadataPrefix=oai_dc&from=2025-01-01"

Data Model

The catalog model follows W3C DCAT 3 concepts.

Resource

A logical dataset identified by a stable base_resource_id. Owns all versions. Has a scope and an owner (user sub).

ResourceVersion

One snapshot of metadata and files. States: draft → published → deprecated. Carries all DCAT fields.

Dataset

Logical grouping of distributions within a version. Maps to dcat:Dataset.

Distribution

A single access point — upload (S3), external link (access_url), or api service (access_url plus dcat:DataService). See Creating a resource for when to use each.

Version lifecycle

POST /resources  →  draft
                    │
                    ▼  (publish guard passes)
              POST /publish  →  published  ←──── current version
                                    │
                                    ▼  (owner creates new version)
                              POST /versions  →  new draft
                                    │
                                    ▼
                              POST /publish  →  new published  ←── current version
                              (previous version transitions to deprecated)

DELETE /resources  →  tombstone  (data_deleted=true, metadata retained)

Scopes

Every resource carries a scope string — either "public" or an arbitrary project identifier. Scopes are not enforced by membership; they function as a namespace and filter. Any authenticated user may create resources in any scope.

FAIR Scoring

Each resource version receives a FAIR score in the range [0.0, 1.0] computed as the average of four dimension scores. Each dimension score is the fraction of passing checks within that dimension.

F

Findable

  • Has a non-empty title
  • Description ≥ 50 characters
  • At least 2 keywords
  • Theme is set
  • DOI is registered
A

Accessible

  • At least one distribution
  • Access rights defined (for private resources)
  • Access or download URL present on all distributions
I

Interoperable

  • License is set
  • Language (BCP-47) is set
  • IANA media type on all distributions
  • Publisher profile is resolvable
R

Reusable

  • Provenance description present (≥ 20 chars)
  • Rights statement present
  • SHA-256 checksum on all uploaded files
ℹ

The overall FAIR score must meet or exceed FAIR_SCORE_PUBLISH_THRESHOLD (default 0.40) before a draft can be published. The API returns actionable suggestions for each failing check.

Authentication

FairDataHive uses Bearer JWT authentication. Tokens are validated against the JWKS endpoint of a configured Keycloak realm. Authorization is resource-centric — the JWT sub claim is used as the owner and permission anchor; no Keycloak roles are enforced by the application.

JWT validation flow

  1. Request arrives with Authorization: Bearer <token>.
  2. API fetches JWKS from Keycloak (cached for 1 hour).
  3. Token is verified with PyJWT (RS256, issuer checked, audience not checked).
  4. Claims are mapped to CurrentUser(sub, email, name, preferred_username).
  5. The sub claim is stored as owner_sub / publisher_sub on created records.

Endpoint protection summary

ScopeRequirement
Read published resources, search, catalog, RDF, OAI-PMHNone — open access
Download from a public distributionNone
Download from a private distributionAuthenticated + listed in access_rights or owner
Create / patch / publish / delete resources, upload filesAuthenticated (any valid user)
Mutate another user's resourceBlocked — 403 Forbidden

Development mode

Setting DEV_AUTH_ENABLED=true bypasses all JWT processing. Every request — including those without any Authorization header — is treated as dev-user. Do not use this in production or any shared deployment.

⚠

The React UI does not include an OIDC login flow. For Keycloak-authenticated deployments, obtain a token externally (Keycloak login page, client credentials, etc.) and paste it into the token field in the header bar.

Search Engine

Catalog discovery uses two complementary strategies: keyword search (lexical, indexed text) and semantic search (vector similarity). The unified endpoint GET /api/v1/search selects or combines them according to mode.

Keyword search (PostgreSQL FTS + token fallback)

A GIN index over to_tsvector('english', …) covers title, description, theme, and keywords (migrations 0002_search_indexes, 0004_expand_fts_index). The API builds the same combined text for plainto_tsquery and ts_rank. A two-step query (IDs + scores first, then entity load) avoids ORM eager-load conflicts.

English FTS drops very short tokens and can miss comma-separated keyword lists. When FTS alone would fail, a token fallback also runs:

  • The query is split on commas, semicolons, and whitespace (e.g. a,b,c,d → four terms).
  • All tokens must match (same intent as plainto_tsquery AND behaviour).
  • Tokens longer than two characters: case-insensitive substring match on title, description, theme, and keywords.
  • Short tokens (one–two characters): match keyword array elements only (exact element, case-insensitive), so a finds a keyword a but not the letter “a” inside fairdata.

A hit qualifies if either FTS or the token fallback matches. Use mode=keyword when searching by explicit tags or abbreviations.

Semantic search (pgvector)

Semantic search finds resources whose meaning is close to the query, not necessarily the same words.

  1. Indexing (on publish) — After a resource passes the publish guard, a background task concatenates title, description, keywords (space-separated), and theme (newline-separated blocks), then encodes the text with the configured sentence-transformers model (EMBEDDING_MODEL, default multi-qa-mpnet-base-dot-v1). The 768-dimensional vector is stored in resource_version.embedding (pgvector). Embeddings are not refreshed on later metadata edits unless the resource is published again.
  2. Query — The user query is encoded with the same model (CPU-bound; first use downloads model weights to the Hugging Face cache and loads the model in-process).
  3. Retrieval — PostgreSQL ranks rows by cosine similarity (1 − (embedding <=> query_vector)), using an HNSW index (idx_rv_embedding). Only rows with a non-null embedding and similarity ≥ SEMANTIC_SEARCH_THRESHOLD (default 0.30) are returned, ordered by score descending.

Semantic search requires PostgreSQL with pgvector; it is disabled on SQLite (unit tests). Private published resources are excluded from catalog search the same as for keyword mode.

Auto mode

Keyword search runs first. If the result count is fewer than 3, semantic search is also run. Unique semantic results are appended to the keyword list (no duplicate version IDs). The response uses "mode_used": "auto". Vague natural-language queries with few lexical hits benefit most; precise keyword or tag queries should use mode=keyword.

Caching

Results are cached in Redis keyed by a SHA-256 hash of the query parameters. TTLs are 300 seconds for keyword results and 120 seconds for semantic results. The cache for a scope is invalidated whenever a resource in that scope is published or deleted. After deploying search changes, restart the API and expect cached responses to expire within the TTL (or flush Redis in development).


Local Development (without Docker)

You need a running PostgreSQL 16 instance with the pgvector extension enabled, plus Redis and RustFS (or any S3-compatible endpoint).

bash# Create and activate a virtual environment
python3.12 -m venv .venv
source .venv/bin/activate

# Install all dependencies including dev tools
pip install -r requirements.txt
pip install pytest pytest-asyncio httpx aiosqlite

# Set environment variables (override .env or export directly)
export DATABASE_URL="postgresql+asyncpg://fairdatahive:fairdatahive@localhost:5432/fairdatahive"
export DEV_AUTH_ENABLED=true

# Run migrations
alembic upgrade head

# Start the API with hot reload
uvicorn app.main:app --reload --port 8002

# In another terminal — React UI (proxies to :8002)
cd frontend && cp .env.example .env && npm install && npm run dev

Testing

The test suite uses pytest-asyncio with an in-memory SQLite database. No PostgreSQL, Redis, or object-storage instance is required to run the tests.

bash# Run all tests
TESTING=1 pytest tests/ -q

# Run a specific test file
TESTING=1 pytest tests/test_search.py -v

# Skip integration tests (require real PostgreSQL + pgvector)
TESTING=1 pytest tests/ -m "not integration" -q
Test fileCoverage area
test_resources.pyCRUD, publish guard, versioning, tombstones
test_search.pyKeyword search, suggest, catalog stats, licenses, themes
test_fair.pyFAIR score computation across all dimensions
test_rdf.pyTurtle parsing, catalog graph, content negotiation
test_access.pyAccess request lifecycle, private download guard
test_audit.pyDownload logging, notifications, stats endpoint
test_oai.pyOAI-PMH verbs, deleted record handling
test_health.pyHealth endpoint, static assets, root response

Troubleshooting

SymptomCauseFix
Authentication required on all requests DEV_AUTH_ENABLED not active in the running container Confirm .env exists and set the variable; rebuild: docker compose build api && docker compose up -d --force-recreate api
Download link opens http://rustfs:9000/… in browser Presigned URL uses Docker hostname Set S3_PUBLIC_ENDPOINT=http://localhost:9002 in .env and recreate the API container
Port 9000 already in use Another S3 service on the host (e.g. old MinIO stack) Compose maps RustFS to loopback 9002 (S3) and 9003 (console). Use S3_PUBLIC_ENDPOINT=http://localhost:9002.
Search returns 500 with a query string Known FTS issue (fixed) — image out of date Rebuild: docker compose build api && docker compose up -d --force-recreate api
Port 5432 already in use Another PostgreSQL on the host Compose maps to host port 5434 (5434:5432). Connect via localhost:5434.
Port 8000 or 8001 already in use Another API on the host (e.g. legacy stack on 8001) Compose maps to host port 8002 (8002:8000). Use http://localhost:8002.
Port 6379 bind error on redis Old compose published Redis on the host Redis is no longer published to the host. Run docker compose down then docker compose up -d.
Same page on 5173 and 5174 Two Vite dev servers (second bound 5174 when 5173 was busy) cd frontend && npm run dev:stop then npm run dev. Use only 5173. Compose no longer includes a Docker Vite service.
Port 5173 is already in use Another Vite or process holds 5173 cd frontend && npm run dev:stop, or end the other terminal's npm run dev. Check: ss -tlnp | grep 5173
Nothing at http://localhost:5173 Vite not running cd frontend && npm run dev (API on 8002), or use http://localhost:8002/ui/ after npm run build
EACCES on node_modules/.vite Root-owned cache (e.g. old Docker Vite container) From fairdatahive/: docker run --rm -v "$(pwd)/frontend:/app" -w /app node:20-alpine rm -rf /app/node_modules/.vite/deps then npm run dev
Keyword search returns no results Query not in catalog text, or API image out of date Rebuild API; try full words or theme names. Partial matches use ILIKE fallback when FTS is empty.
Temporary failure in name resolution for postgres .env missing or postgres not running Run cp .env.example .env then docker compose up -d postgres and wait for healthy
Migrations fail on API startup Database not ready or schema drift Wait for healthy Postgres, then: docker compose run --rm --entrypoint alembic api upgrade head. Migrations run automatically via RUN_MIGRATIONS=true in the API entrypoint.
Publish fails with FAIR score error Score below 0.40 threshold Check GET /resources/{id}/fair-score for failing checks and suggestions

Useful diagnostic commands

bash# Check dev auth is active
curl http://localhost:8002/api/v1/users/me
# Should return {"sub":"dev-user",…} without any Authorization header

# Check service health
curl http://localhost:8002/api/v1/health

# View API logs
docker compose logs api --tail 50 -f

# Confirm env inside the container
docker compose exec api printenv DEV_AUTH_ENABLED

# Stop stray Vite on 5173 / 5174
cd frontend && npm run dev:stop

# Rebuild API after code or migration changes
docker compose build api && docker compose up -d --force-recreate api

FairDataHive v1.0.0  ·  Built on FastAPI, PostgreSQL, pgvector, RustFS, Redis  ·  Get started