FairDataHive
A FAIR-compliant research data catalog built on W3C DCAT 3, FastAPI, and PostgreSQL.
Overview
FairDataHive is a self-hosted research data management system that lets teams register, describe, version, publish, and share datasets following open standards. It replaces ad-hoc file shares and spreadsheet catalogs with a structured, searchable, and interoperable repository.
Catalog Management
Create and publish versioned dataset records with rich DCAT metadata.
Dual-mode Search
Keyword search (FTS + tag matching) and semantic vector search over published catalog resources.
FAIR Scoring
Automatic per-resource FAIR assessment with actionable improvement suggestions.
RDF Export
Machine-readable Turtle and JSON-LD exports per resource and catalog-wide.
OAI-PMH
Standards-based harvesting endpoint for external portal integration.
Access Control
Private resources with owner-managed access request workflows.
Architecture
FairDataHive follows a layered service architecture. The backend stack runs in Docker
Compose; the React UI runs locally with Vite during development, or as static files
served by the API after npm run build. There is no containerized Vite
service — that pattern duplicated port 5173 and caused a second dev
server on 5174.
| Layer | Technology | Role |
|---|---|---|
| API | FastAPI 0.111, Python 3.12 | REST API, business logic, async I/O |
| ORM / Migrations | SQLAlchemy 2.0 async, Alembic | Schema management, typed queries |
| Primary database | PostgreSQL 16 + pgvector | Relational storage, FTS indexes, vector search |
| Object storage | RustFS (S3 API; also MinIO/AWS/Ceph) | File uploads, presigned download URLs |
| Cache | Redis 7 | Search result caching, TTL invalidation |
| Embeddings | sentence-transformers (in-process) | 768-dim vectors for semantic search |
| RDF | rdflib 7 | Turtle / JSON-LD / OAI-DC generation |
| SHACL | pyshacl 0.25 | DCAT-AP shape validation |
| UI | React 18, Vite, TypeScript | Browser data management interface |
| Auth | Keycloak 24 (optional) | OIDC/JWT identity provider |
Host port mapping (Docker Compose)
| Service | Container port | Host port | Notes |
|---|---|---|---|
| API | 8000 | 8002 | Docs at /docs; Swagger at /swagger; UI at /ui |
| PostgreSQL | 5432 | 5434 | Avoids conflict with local Postgres on 5432 |
| RustFS S3 | 9000 | 9002 | Set S3_PUBLIC_ENDPOINT=http://localhost:9002 |
| RustFS console | 9001 | 9003 | Loopback only; local login rustfsadmin / rustfsadmin |
| Redis | 6379 | none | Internal compose network only |
| Keycloak | 8080 | 8080 | Optional: docker compose --profile auth up -d auth |
| React UI (dev) | — | 5173 | Local only: cd frontend && npm run dev (strictPort) |
Service topology
Browser
│
├── :5173 Vite dev server (host: npm run dev) ──────────┐
│ /docs → docs/index.html (local) │
│ proxies /api, /swagger → :8002 │
├── :8002/ui production build (frontend/dist) ├──► API (Docker :8002)
└── :8002/docs guide · :8002/swagger OpenAPI │
│ │
├── postgres (host :5434)
├── rustfs (host :9002 / :9003)
├── redis (internal)
└── auth (optional profile, :8080)
docker compose up -d → backend only (no Node/Vite container)
Repository layout
fairdatahive/
├── app/
│ ├── config.py ← Pydantic Settings (all env vars)
│ ├── database.py ← Async SQLAlchemy engine
│ ├── main.py ← FastAPI factory, middleware, router mount
│ ├── core/ ← Auth, exceptions, pagination, constants
│ ├── models/ ← SQLAlchemy ORM models (source of truth)
│ ├── schemas/ ← Pydantic v2 request / response contracts
│ ├── routers/ ← HTTP routing (thin layer, no logic)
│ └── services/ ← Business logic (resource, search, fair…)
├── alembic/ ← Database migrations
├── frontend/ ← React + TypeScript UI (Vite, scripts/dev-server.mjs)
├── docs/ ← This documentation (index.html)
├── static/ ← context.jsonld, dcat_ap_shapes.ttl
├── templates/ ← Jinja2 HTML landing page
├── tests/ ← pytest-asyncio test suite
├── docker/ ← entrypoint.sh
├── docker-compose.yml
├── Dockerfile
└── .env.example
Quick Start
Prerequisites
- Docker ≥ 24 and Docker Compose v2
- Ports
5173,8002,9002,9003,5434available on the host (Redis is internal-only)
Start the full stack
-
Clone the repository and enter the project directory.
bash
git clone <repo-url> cd fairdatahive -
Create your environment file.
bash
cp .env.example .envThe defaults work out of the box for local development. See Configuration for a full reference.
-
Build and start all services.
bash
docker compose build docker compose up -dStarts postgres, rustfs, redis, and api only — not a Vite/Node UI container. The API runs Alembic migrations on startup; wait ~15 seconds on first boot.
-
Verify the API is healthy.
bash
curl http://localhost:8002/api/v1/health # {"status":"ok","components":{"database":"ok","cache":"ok","storage":"ok"}} -
Start the React UI (separate terminal).
bash
cd frontend cp .env.example .env npm install npm run devOpen http://localhost:5173. Vite proxies
/api,/swagger, and/openapi.jsonto:8002. Project documentation is served at/docsfrom thedocs/folder. Only one dev server should run on port 5173;npm run devclears any orphan on 5174 automatically. -
Optional — production-like UI without Vite.
bash
cd frontend && npm run build docker compose build api && docker compose up -d --force-recreate apiOpen http://localhost:8002/ui/ (root
/redirects there).
Key URLs after startup
| Service | URL |
|---|---|
| React UI (dev) | http://localhost:5173 — npm run dev in frontend/ |
| React UI (built) | http://localhost:8002/ui/ — after npm run build |
| Documentation | http://localhost:5173/docs (or :8002/docs via API) |
| Swagger / OpenAPI | http://localhost:8002/swagger (proxied from Vite as /swagger) |
| RustFS console | http://localhost:9003 — rustfsadmin / rustfsadmin (loopback) |
| Catalog (Turtle) | http://localhost:8002/api/v1/catalog.ttl |
| OAI-PMH endpoint | http://localhost:8002/api/v1/oai?verb=Identify |
| Prometheus metrics | http://localhost:8002/metrics |
Dev auth mode — the default .env sets DEV_AUTH_ENABLED=true, which bypasses JWT verification and treats every request as a fixed dev-user. This makes local development frictionless. Disable it for any shared or production deployment.
One UI dev server — use only http://localhost:5173. If you previously saw the same app on 5174, a second Vite had started when 5173 was busy. Run npm run dev:stop in frontend/, then npm run dev. Do not run a Docker Vite container alongside local npm run dev (that service was removed from Compose).
Configuration
All configuration is driven by environment variables (loaded from .env by
Pydantic Settings). No code changes are needed to adapt FairDataHive to your environment.
Database
| Variable | Default | Description |
|---|---|---|
DATABASE_URL | postgresql+asyncpg://fairdatahive:fairdatahive@postgres:5432/fairdatahive | Async PostgreSQL connection string |
Object storage (RustFS / S3)
| Variable | Default | Description |
|---|---|---|
S3_ENDPOINT | http://rustfs:9000 | S3 endpoint used by the API container (Docker hostname) |
S3_PUBLIC_ENDPOINT | http://localhost:9002 | Browser-facing URL embedded in presigned download links (host port from compose). |
S3_ACCESS_KEY | rustfsadmin | Restricted application credential (not the server root in production) |
S3_SECRET_KEY | rustfsadmin | Application credential secret |
S3_BUCKET | fairdatahive-catalog | Bucket name for file uploads |
S3_REGION | us-east-1 | SigV4 signing region |
S3_ADDRESSING_STYLE | path | Path-style addressing for self-hosted S3 |
PROVISION_STORAGE | true | Entrypoint creates the bucket if missing |
MAX_UPLOAD_SIZE_MB | 500 | Maximum per-file upload size in megabytes |
PRESIGNED_URL_EXPIRY_SECONDS | 3600 | Presigned download URL lifetime |
Legacy MINIO_* names are accepted for one transition release; when both are set,
S3_* wins. Cutover from an existing MinIO volume is documented in
docs/storage-migration.md.
Authentication
| Variable | Default | Description |
|---|---|---|
KEYCLOAK_URL | http://auth:8080 | Keycloak base URL for JWKS fetching |
KEYCLOAK_REALM | fairdatahive | Realm name |
DEV_AUTH_ENABLED | true in .env.example | Skip JWT and use a fixed dev identity (dev-user). Set false for production. |
DEV_AUTH_SUB | dev-user | Subject claim used when dev auth is active |
DEV_AUTH_NAME | Development User | Display name for the dev user |
Search and catalog
| Variable | Default | Description |
|---|---|---|
REDIS_URL | redis://redis:6379/0 | Redis connection URL for search result caching |
EMBEDDING_MODEL | multi-qa-mpnet-base-dot-v1 | sentence-transformers model for semantic search |
SEMANTIC_SEARCH_THRESHOLD | 0.30 | Minimum cosine similarity for a semantic hit |
FAIR_SCORE_PUBLISH_THRESHOLD | 0.40 | Minimum overall FAIR score required to publish |
CORS
| Variable | Default |
|---|---|
CORS_ORIGINS | http://localhost:5173,http://localhost:3000,http://localhost:8002 |
Optional integrations
| Variable | Description |
|---|---|
SMTP_HOST / SMTP_PORT / SMTP_USER / SMTP_PASSWORD | Email delivery for access notifications |
DATACITE_REPO_ID / DATACITE_PASSWORD / DATACITE_DOI_PREFIX | DataCite Fabrica API credentials for DOI minting |
DATACITE_API_URL | Defaults to https://api.test.datacite.org (sandbox) |
Deployment
FairDataHive is designed as a self-hosted data management service: metadata and search live in PostgreSQL, binaries in S3-compatible object storage, and the API orchestrates FAIR cataloguing, access control, and interoperability endpoints. This section describes how to deploy that stack reliably for lab, consortium, or institutional use — not only “start containers”, but the operational choices that keep catalogs findable, downloadable, and auditable over time.
On this page
Topology and components
Treat FairDataHive as a small platform, not a single process. Each component has a distinct failure domain and scaling profile:
| Component | Role | Persistence | Notes |
|---|---|---|---|
| API (FastAPI) | REST, RDF, OAI-PMH, presigned URLs, FAIR scoring | Stateless | CPU/RAM for embeddings on semantic search; rebuild image on release |
| PostgreSQL 16 + pgvector | Catalog metadata, FTS, vector index, audit | Durable volume | Back up regularly; extensions required at init |
| RustFS / S3 | Uploaded files and ZIP extracts | Durable volume or external bucket | Must be reachable from API and browsers (see public endpoint) |
| Redis | Search result cache | Optional persistence | Safe to flush; cache rebuilds on traffic |
| Keycloak (optional) | OIDC/JWT issuer | Its own DB in real deployments | Compose profile auth is dev-oriented only |
| React UI | Human catalog interface | Static files in frontend/dist | Served by API at /ui or separate CDN |
The bundled docker-compose.yml maps host ports to avoid clashes on developer
machines (8002 API, 5434 Postgres, 9002/9003
RustFS). In production behind a reverse proxy you typically expose 443 only
and keep data services on an internal network.
Prerequisites
- Container runtime — Docker 24+ and Compose v2 (or an equivalent orchestrator).
- Host resources — minimum 4 GB RAM for a pilot (8 GB+ recommended once semantic search and concurrent uploads matter). Disk: Postgres + object storage grow with catalog size; plan object storage separately from DB.
- Network — stable DNS for the public catalog URL; outbound HTTPS if using DataCite or external IdP.
- Secrets store — production credentials must not live in git; use a vault, compose secrets, or orchestrator secrets.
- TLS termination — HTTPS at the edge (reverse proxy or load balancer) for UI, API, and presigned download URLs.
Deployment models
Single-host Compose
Fastest path: one VM runs Postgres, RustFS, Redis, and API together. Ideal for pilots, lab deployments, and consortium sandboxes.
Compose + external managed services
Run only the API container (or API + Redis) and point DATABASE_URL and S3 settings at managed RDS/Cloud SQL and institutional S3.
Kubernetes / Nomad
Split stateful sets (Postgres operator, S3-compatible object store) from a stateless API Deployment. Use Ingress for TLS and path routing to /api/v1, /ui, /docs.
Regardless of model, preserve the two-endpoint S3 pattern: an internal endpoint for the API container and a public endpoint embedded in presigned links returned to browsers. Mixing these is the most common production defect.
Docker Compose (single host)
The reference stack is defined in docker-compose.yml. Migrations run automatically
when the API container starts (RUN_MIGRATIONS=true in the entrypoint).
-
Prepare configuration.
bash
cd fairdatahive cp .env.example .env # Edit .env — see “Production environment” below -
Build the API image (after any code or dependency change):
bash
docker compose build api -
Start data services and wait for Postgres.
bash
docker compose up -d postgres rustfs redis docker compose ps # postgres should show “healthy” -
Start the API.
bash
docker compose up -d api docker compose logs -f api # watch migrations, then “Application startup complete” -
Verify health and catalog surface.
bash
curl -s http://localhost:8002/api/v1/health | jq . curl -s http://localhost:8002/api/v1/oai?verb=Identify | headOpen documentation and Swagger on the host port you publish.
Do not run a Dockerized Vite dev server in production. Use a static build served at /ui (see below). A second Node container caused duplicate ports and root-owned caches in development; it is intentionally removed from Compose.
Serving the React UI
The catalog UI is a static SPA. For production, build once and let the API serve it from
frontend/dist (mounted when the directory exists at image build time or copied in).
bashcd frontend
npm ci
npm run build
cd ..
docker compose build api
docker compose up -d --force-recreate api
Users open https://<your-host>/ui/ (the API redirects / when
the build is present). Set CORS_ORIGINS to your public UI origin if the UI is
hosted on another hostname; when served from the same API host, CORS is less critical but
token-based API calls from browsers still require correct origins if split.
For CDN deployment, upload frontend/dist/* and point the CDN at the same
VITE_API_BASE public API URL used at build time, or rebuild with
VITE_API_BASE=https://catalog.example.org/api/v1.
Object storage
RustFS in Compose (default)
The API talks to RustFS on the internal Docker network (S3_ENDPOINT=http://rustfs:9000).
Browsers must never receive that hostname in presigned URLs. Sign against
S3_PUBLIC_ENDPOINT so the SigV4 host matches what the browser hits.
| Variable | Compose / dev | Production example |
|---|---|---|
S3_ENDPOINT | http://rustfs:9000 | Internal service URL (K8s DNS, VPC endpoint) |
S3_PUBLIC_ENDPOINT | http://localhost:9002 | https://objects.example.org |
S3_BUCKET | fairdatahive-catalog | Institutional bucket name; create before first upload |
After changing S3_PUBLIC_ENDPOINT, recreate the API container. Test by
publishing a resource, downloading a file, and confirming the presigned URL host matches
what clients can resolve. Keep the RustFS console private.
External S3 (AWS, Ceph, institutional cluster)
FairDataHive uses S3-compatible APIs via aioboto3. Point S3_ENDPOINT
at the vendor endpoint, set keys and bucket, and set S3_PUBLIC_ENDPOINT to the
same public base users hit (or CloudFront in front of the bucket). Ensure CORS on the bucket
allows GET from your catalog origin if browsers fetch objects directly.
Set PROVISION_STORAGE=false when CreateBucket is denied.
Database and migrations
PostgreSQL must include the vector extension (image
pgvector/pgvector:pg16 in Compose). The API runs
alembic upgrade head on startup when RUN_MIGRATIONS=true.
- First deploy — start Postgres, then API once; confirm logs show migrations applied.
- Upgrades — pull new image, run
docker compose up -d --force-recreate api; entrypoint migrates before serving traffic. - Manual migration —
docker compose run --rm --entrypoint alembic api upgrade head - Backup — snapshot the
postgres-datavolume or usepg_dumpon a schedule; test restore on a staging instance before you need it. - Connection string — use
postgresql+asyncpg://form inDATABASE_URL; pool sizing is handled by SQLAlchemy defaults unless you tune at the proxy layer (PgBouncer recommended for large fleets).
Metadata tombstones remain in Postgres when resources are deleted; files are removed from object storage. Backup policy should cover both DB and bucket for full recovery.
Authentication in production
Disable development authentication everywhere except local laptops:
envDEV_AUTH_ENABLED=false
KEYCLOAK_URL=https://idp.example.org
KEYCLOAK_REALM=fairdatahive
The API validates JWTs via JWKS (PyJWT, RS256, issuer checked). Configure
your IdP (Keycloak or institutional OIDC) to issue tokens with a stable sub claim —
that value becomes owner_sub / publisher_sub on records. The Compose
auth profile (docker compose --profile auth up -d auth) runs Keycloak
in start-dev mode; for production, run Keycloak (or another IdP) as its own
hardened deployment with a dedicated database and TLS.
UI users paste or store access tokens in the catalog header, or you integrate your institution’s OAuth2 proxy in front of the UI and API. Plan token lifetime and refresh according to your security policy; FairDataHive does not implement refresh-token flows itself.
Reverse proxy and TLS
Expose a single public hostname (e.g. catalog.example.org) and route paths to the API
container. Typical path map:
| Path | Backend |
|---|---|
/api/v1/* | API (catalog REST) |
/ui/* | Static UI (or dedicated CDN origin) |
/docs | Project documentation (StaticFiles) |
/swagger, /openapi.json | OpenAPI UI and schema |
/metrics | Prometheus (restrict by network ACL) |
/api/v1/oai | OAI-PMH harvesters (allow institutional IPs) |
Set the public site URL in configuration:
envBASE_URL=https://catalog.example.org
CORS_ORIGINS=https://catalog.example.org
Terminate TLS at the proxy (nginx, Traefik, Caddy, cloud LB). Increase
client_max_body_size (nginx) to at least your
MAX_UPLOAD_SIZE_MB plus overhead for multipart uploads.
Production environment
Example production-oriented .env (adjust hostnames and secrets):
env# Core
DATABASE_URL=postgresql+asyncpg://fairdatahive:***@postgres:5432/fairdatahive
REDIS_URL=redis://redis:6379/0
BASE_URL=https://catalog.example.org
RUN_MIGRATIONS=true
# Object storage
S3_ENDPOINT=http://rustfs:9000
S3_PUBLIC_ENDPOINT=https://objects.example.org
S3_ACCESS_KEY=***
S3_SECRET_KEY=***
S3_BUCKET=fairdatahive-catalog
S3_REGION=us-east-1
S3_ADDRESSING_STYLE=path
PROVISION_STORAGE=true
MAX_UPLOAD_SIZE_MB=500
PRESIGNED_URL_EXPIRY_SECONDS=3600
# Auth — required in production
DEV_AUTH_ENABLED=false
KEYCLOAK_URL=https://idp.example.org
KEYCLOAK_REALM=fairdatahive
CORS_ORIGINS=https://catalog.example.org
# Search / FAIR
EMBEDDING_MODEL=multi-qa-mpnet-base-dot-v1
FAIR_SCORE_PUBLISH_THRESHOLD=0.4
SEMANTIC_SEARCH_THRESHOLD=0.30
# Optional
SMTP_HOST=smtp.example.org
SMTP_USER=***
SMTP_PASSWORD=***
EMAIL_FROM=noreply@catalog.example.org
DATACITE_REPO_ID=***
DATACITE_PASSWORD=***
DATACITE_DOI_PREFIX=10.12345
DATACITE_API_URL=https://api.datacite.org
Never deploy with default RustFS credentials (rustfsadmin) or
DEV_AUTH_ENABLED=true on a network reachable by others. Rotate Postgres and
Redis passwords; restrict admin consoles (RustFS console, Keycloak) to management VPNs.
Operations: backup, upgrade, monitoring
Backup strategy
- PostgreSQL — nightly logical dump or volume snapshot; retain 30+ days for catalog compliance.
- Object storage — versioning/replication on the bucket; lifecycle rules for cost.
- Configuration — version-control
.envtemplates without secrets; store secrets in a vault.
Upgrades
- Snapshot DB and note current image tag.
- Build/pull new API image; run migrations on staging first.
docker compose up -d --force-recreate api(or rolling update in K8s).- Smoke-test: health, search, upload, download presign, OAI
Identify. - Rebuild UI if API paths or env changed:
npm run buildthen recreate API.
Monitoring
The API exposes Prometheus metrics at /metrics (instrumentator). Scrape from your
monitoring network only. Combine with:
GET /api/v1/health— database and Redis connectivity- Container logs — structured via application logging; watch migration and presign errors
- Disk usage on Postgres and object storage volumes
- Redis memory — eviction is acceptable for cache but indicates heavy search load
Semantic search loads the sentence-transformers model on first use; expect a CPU/RAM spike
after cold start — plan readiness probes with generous start_period if you
autoscale API pods.
Security checklist
| Area | Action |
|---|---|
| Authentication | DEV_AUTH_ENABLED=false; institutional OIDC; short-lived tokens |
| Transport | TLS 1.2+ everywhere; HSTS on public catalog host |
| Object storage | Unique keys; bucket policy least privilege; correct public endpoint |
| Database | Strong password; no public port; encrypted volume at rest if required by policy |
| Network | Redis and Postgres internal-only; restrict /metrics and the RustFS console |
| Uploads | Enforce MAX_UPLOAD_SIZE_MB; scan at proxy or storage layer if policy requires AV |
| Private resources | Review access-request workflow; enable SMTP for auditability of grants |
| Dependencies | Rebuild images on security patches; pin base image digests in regulated environments |
For incident response, catalog metadata and audit logs live in Postgres; download events are recorded per distribution. Pair FairDataHive logs with reverse-proxy access logs for a full trace of who retrieved which presigned URL.
React UI
The frontend is a standalone Vite + React 18 + TypeScript application located in
frontend/. It communicates entirely through the REST API and requires no
separate backend of its own.
How to run the UI
| Mode | Command | URL |
|---|---|---|
| Development (recommended) | npm run dev | http://localhost:5173 — hot reload, API proxy |
| Production-like | npm run build then recreate API | http://localhost:8002/ui/ |
| Preview build | npm run preview | http://localhost:4173 — static preview only |
Development server
bash# Terminal 1 — backend
cd fairdatahive && docker compose up -d
# Terminal 2 — frontend
cd fairdatahive/frontend
cp .env.example .env # VITE_DEV_AUTH=true to match API dev auth
npm install
npm run dev # guarded by scripts/dev-server.mjs → http://127.0.0.1:5173
vite.config.ts sets strictPort: true on port 5173 and binds
127.0.0.1. The npm run dev script stops any orphan process on
5174 before starting (legacy second Vite instances).
Production build
bashcd fairdatahive/frontend
npm run build # outputs to frontend/dist/
cd ..
docker compose build api && docker compose up -d --force-recreate api
When frontend/dist/ exists, the API mounts it at /ui and redirects
/ to http://localhost:8002/ui/
without a separate Node process.
Stop / restart dev server
bashcd fairdatahive/frontend
npm run dev:stop # frees 5173 and any orphan 5174
npm run dev # single Vite on http://127.0.0.1:5173 only
Frontend environment variables
| Variable | Default | Description |
|---|---|---|
VITE_API_BASE | empty | Full API base URL. Leave empty to use the Vite proxy (/api → localhost:8002). |
VITE_DEV_AUTH | true | When true, seeds the token field with dev on first load. |
VITE_API_PROXY | http://localhost:8002 | Proxy target for vite.config.ts when overriding the default API port. |
UI Usage Guide
Header controls
The top bar contains navigation links, a scope selector, and an authentication token field.
- Scope selector — switches between the public catalog (all published resources) and a named project scope. When a project ID is entered, browse, search, and create operations are scoped to that project.
- Token field — paste a JWT access token here for Keycloak-authenticated requests. With
DEV_AUTH_ENABLED=true, this field can be left empty or set to any value.
Search
The Search page supports three modes selectable from the dropdown next to the search box. Only published, non-private resources in the current scope are searchable. See Search engine for implementation detail.
- Auto (default) — keyword search first; if fewer than 3 hits, semantic results are merged in.
- Keyword — best for titles, themes, and keyword tags (including short tags like
aor comma-separateda,b,c,d). Uses PostgreSQL full-text search plus a token fallback when FTS misses (e.g. single-letter English stop words). - Semantic — best for natural-language questions (e.g. “air pollution readings from city sensors”). Matches by meaning via embeddings stored at publish time; requires PostgreSQL + pgvector.
Facet filters for theme, license, and format appear once a query is submitted.
Creating a resource
The Create page registers a new DCAT dataset in the current scope. You need at least one distribution to publish; metadata and files can be combined in a single save.
-
Navigate to Create in the top navigation.
-
Fill in the metadata form.
- Title — display name in the catalog and search results.
- Description (min 50 characters) — what the data is and how to use it.
- Keywords (min 2, comma-separated) — discovery tags for search and facets.
- Theme — research domain from the catalog vocabulary.
- License — SPDX-style licence; required before publish.
- Private resource — when checked, downloads require owner approval via access requests.
Additional fields (provenance, spatial/temporal coverage, DOI, language) are available via the API when creating or patching a resource.
-
Add at least one distribution. Each option adds a separate
dcat:Distribution; you may use one or several.- File upload — data stored in FairDataHive (S3/RustFS). Files upload as you select them; ZIP archives are unpacked and each member can be linked. Download on the resource page uses a presigned URL.
-
External URL — a link to data hosted elsewhere (Zenodo, Figshare, a lab
file server, a landing page). Recorded as
access_urlonly. Use for static files or human-oriented pages, not for describing a programmatic API. -
API / data service URL — the root of a REST or other machine-facing
service. Recorded as both
access_url(where to reach the data) and a linkeddcat:DataServicewithendpoint_url(for RDF export and harvesters). The Create form sets both to the same URL. Use when consumers call an API rather than download a fixed file.
External URL vs API — same hostname, different intent: external = “get the file or page here”; API = “access via a data service”. Example external:
https://zenodo.org/…/files/data.csv. Example API:https://api.example.com/v1/measurements. -
Save as draft or publish immediately. Enabling Publish immediately runs the publish guard after saving (licence, at least one distribution, FAIR score ≥ threshold). If the guard fails, the record stays a draft and failing checks are shown on the resource page.
Resource detail page
The resource page shows:
- Full metadata and version history (sidebar)
- FAIR score breakdown with suggestions (sidebar)
- Distributions with Download buttons (generates a presigned URL)
- Actions: Publish (for drafts), Landing page link, RDF (Turtle) link
- Access request button (for private resources the user cannot download)
API — Catalog
All API routes are prefixed with /api/v1.
?scope=.API — Resources
ResourceVersionRead.auth?scope=&page=&size=.text/turtle or application/ld+json.Create resource — request body
json{
"scope": "public", // "public" or a project ID string
"title": "Urban air quality measurements",
"description": "Hourly PM2.5 and NO2 readings from city sensor stations…",
"keywords": ["air quality", "PM2.5", "sensors"],
"theme": "EarthScience",
"license_id": "CC-BY-4.0",
"language": "en",
"provenance": "Collected by a municipal sensor network, 2024…",
"is_private": false,
"datasets": [
{
"distributions": [
{
"dist_type": "upload",
"upload_token": "public/draft-dev-user-abc/abc123-data.csv",
"media_type": "text/csv",
"checksum_sha256": "…",
"byte_size": 45231
}
]
}
]
}
Publish guard — a draft cannot be published unless all three conditions are met: (1) a license_id is set, (2) at least one distribution exists, and (3) the overall FAIR score ≥ FAIR_SCORE_PUBLISH_THRESHOLD (default 0.40).
API — Files
Files are uploaded independently before (or during) resource creation. The returned
upload_token is the S3 object key and must be included in a
DistributionCreate payload when creating or patching a resource.
Upload flow
bash# 1. Upload the file
curl -F "file=@data.csv" \
"http://localhost:8002/api/v1/files/upload?scope=public"
# Response:
# { "upload_token": "public/draft-dev-user-abc/8fa3-data.csv", … }
# 2. Reference the token in the resource payload
# "distributions": [{ "dist_type": "upload", "upload_token": "public/…" }]
ZIP archives are automatically extracted. Each member file is uploaded individually and
returned as extracted_files in the response. You can add each extracted
file as a separate distribution.
API — Search
Query parameters for /search
| Parameter | Values | Default |
|---|---|---|
q | Free-text query string | — |
mode | auto | keyword | semantic | auto |
scope | public or project ID | all |
theme | theme ID from /themes | — |
license_id | SPDX license ID from /licenses | — |
page / size | integers | 1 / 20 |
Search results are cached in Redis (300 s for keyword, 120 s for semantic). The cache is invalidated when a resource in the same scope is published or deleted.
API — Access Control
access_rights.auth
Access requests expire automatically after ACCESS_REQUEST_EXPIRY_DAYS (default 30).
A background task runs every 6 hours to purge expired pending requests.
When a request is accepted or rejected, an in-app notification is created for the requester.
If SMTP is configured, an email is also sent.
API — Interoperability
Identify, ListMetadataFormats, ListSets, ListIdentifiers, ListRecords, GetRecord. Formats: oai_dc, dcat.OAI-PMH usage examples
bash# Identify the repository
curl "http://localhost:8002/api/v1/oai?verb=Identify"
# List all records (Dublin Core)
curl "http://localhost:8002/api/v1/oai?verb=ListRecords&metadataPrefix=oai_dc"
# Harvest a specific scope as a set
curl "http://localhost:8002/api/v1/oai?verb=ListRecords&metadataPrefix=dcat&set=project-alpha"
# Incremental harvest from a date
curl "http://localhost:8002/api/v1/oai?verb=ListRecords&metadataPrefix=oai_dc&from=2025-01-01"
Data Model
The catalog model follows W3C DCAT 3 concepts.
Resource
A logical dataset identified by a stable base_resource_id. Owns all versions. Has a scope and an owner (user sub).
ResourceVersion
One snapshot of metadata and files. States: draft → published → deprecated. Carries all DCAT fields.
Dataset
Logical grouping of distributions within a version. Maps to dcat:Dataset.
Distribution
A single access point — upload (S3), external link
(access_url), or api service (access_url plus
dcat:DataService). See Creating a resource for when
to use each.
Version lifecycle
POST /resources → draft
│
▼ (publish guard passes)
POST /publish → published ←──── current version
│
▼ (owner creates new version)
POST /versions → new draft
│
▼
POST /publish → new published ←── current version
(previous version transitions to deprecated)
DELETE /resources → tombstone (data_deleted=true, metadata retained)
Scopes
Every resource carries a scope string — either "public" or an
arbitrary project identifier. Scopes are not enforced by membership; they function as a
namespace and filter. Any authenticated user may create resources in any scope.
FAIR Scoring
Each resource version receives a FAIR score in the range [0.0, 1.0] computed
as the average of four dimension scores. Each dimension score is the fraction of passing
checks within that dimension.
Findable
- Has a non-empty title
- Description ≥ 50 characters
- At least 2 keywords
- Theme is set
- DOI is registered
Accessible
- At least one distribution
- Access rights defined (for private resources)
- Access or download URL present on all distributions
Interoperable
- License is set
- Language (BCP-47) is set
- IANA media type on all distributions
- Publisher profile is resolvable
Reusable
- Provenance description present (≥ 20 chars)
- Rights statement present
- SHA-256 checksum on all uploaded files
The overall FAIR score must meet or exceed FAIR_SCORE_PUBLISH_THRESHOLD (default 0.40) before a draft can be published. The API returns actionable suggestions for each failing check.
Authentication
FairDataHive uses Bearer JWT authentication. Tokens are validated against
the JWKS endpoint of a configured Keycloak realm. Authorization is resource-centric —
the JWT sub claim is used as the owner and permission anchor; no Keycloak
roles are enforced by the application.
JWT validation flow
- Request arrives with
Authorization: Bearer <token>. - API fetches JWKS from Keycloak (cached for 1 hour).
- Token is verified with
PyJWT(RS256, issuer checked, audience not checked). - Claims are mapped to
CurrentUser(sub, email, name, preferred_username). - The
subclaim is stored asowner_sub/publisher_subon created records.
Endpoint protection summary
| Scope | Requirement |
|---|---|
| Read published resources, search, catalog, RDF, OAI-PMH | None — open access |
| Download from a public distribution | None |
| Download from a private distribution | Authenticated + listed in access_rights or owner |
| Create / patch / publish / delete resources, upload files | Authenticated (any valid user) |
| Mutate another user's resource | Blocked — 403 Forbidden |
Development mode
Setting DEV_AUTH_ENABLED=true bypasses all JWT processing. Every request —
including those without any Authorization header — is treated as
dev-user. Do not use this in production or any shared deployment.
The React UI does not include an OIDC login flow. For Keycloak-authenticated deployments, obtain a token externally (Keycloak login page, client credentials, etc.) and paste it into the token field in the header bar.
Search Engine
Catalog discovery uses two complementary strategies: keyword search
(lexical, indexed text) and semantic search (vector similarity).
The unified endpoint GET /api/v1/search selects or combines them according to
mode.
Keyword search (PostgreSQL FTS + token fallback)
A GIN index over to_tsvector('english', …) covers title,
description, theme, and keywords
(migrations 0002_search_indexes, 0004_expand_fts_index).
The API builds the same combined text for plainto_tsquery and
ts_rank. A two-step query (IDs + scores first, then entity load) avoids ORM
eager-load conflicts.
English FTS drops very short tokens and can miss comma-separated keyword lists. When FTS alone would fail, a token fallback also runs:
- The query is split on commas, semicolons, and whitespace (e.g.
a,b,c,d→ four terms). - All tokens must match (same intent as
plainto_tsqueryAND behaviour). - Tokens longer than two characters: case-insensitive substring match on title, description, theme, and keywords.
- Short tokens (one–two characters): match keyword array elements only (exact element, case-insensitive), so
afinds a keywordabut not the letter “a” insidefairdata.
A hit qualifies if either FTS or the token fallback matches. Use mode=keyword
when searching by explicit tags or abbreviations.
Semantic search (pgvector)
Semantic search finds resources whose meaning is close to the query, not necessarily the same words.
-
Indexing (on publish) — After a resource passes the publish guard, a
background task concatenates title, description, keywords (space-separated), and theme
(newline-separated blocks), then encodes the text with the configured
sentence-transformers model (
EMBEDDING_MODEL, defaultmulti-qa-mpnet-base-dot-v1). The 768-dimensional vector is stored inresource_version.embedding(pgvector). Embeddings are not refreshed on later metadata edits unless the resource is published again. - Query — The user query is encoded with the same model (CPU-bound; first use downloads model weights to the Hugging Face cache and loads the model in-process).
-
Retrieval — PostgreSQL ranks rows by cosine similarity
(
1 − (embedding <=> query_vector)), using an HNSW index (idx_rv_embedding). Only rows with a non-null embedding and similarity ≥SEMANTIC_SEARCH_THRESHOLD(default0.30) are returned, ordered by score descending.
Semantic search requires PostgreSQL with pgvector; it is disabled on SQLite (unit tests). Private published resources are excluded from catalog search the same as for keyword mode.
Auto mode
Keyword search runs first. If the result count is fewer than 3, semantic
search is also run. Unique semantic results are appended to the keyword list (no duplicate
version IDs). The response uses "mode_used": "auto". Vague natural-language
queries with few lexical hits benefit most; precise keyword or tag queries should use
mode=keyword.
Caching
Results are cached in Redis keyed by a SHA-256 hash of the query parameters. TTLs are 300 seconds for keyword results and 120 seconds for semantic results. The cache for a scope is invalidated whenever a resource in that scope is published or deleted. After deploying search changes, restart the API and expect cached responses to expire within the TTL (or flush Redis in development).
Local Development (without Docker)
You need a running PostgreSQL 16 instance with the pgvector extension
enabled, plus Redis and RustFS (or any S3-compatible endpoint).
bash# Create and activate a virtual environment
python3.12 -m venv .venv
source .venv/bin/activate
# Install all dependencies including dev tools
pip install -r requirements.txt
pip install pytest pytest-asyncio httpx aiosqlite
# Set environment variables (override .env or export directly)
export DATABASE_URL="postgresql+asyncpg://fairdatahive:fairdatahive@localhost:5432/fairdatahive"
export DEV_AUTH_ENABLED=true
# Run migrations
alembic upgrade head
# Start the API with hot reload
uvicorn app.main:app --reload --port 8002
# In another terminal — React UI (proxies to :8002)
cd frontend && cp .env.example .env && npm install && npm run dev
Testing
The test suite uses pytest-asyncio with an in-memory SQLite database.
No PostgreSQL, Redis, or object-storage instance is required to run the tests.
bash# Run all tests
TESTING=1 pytest tests/ -q
# Run a specific test file
TESTING=1 pytest tests/test_search.py -v
# Skip integration tests (require real PostgreSQL + pgvector)
TESTING=1 pytest tests/ -m "not integration" -q
| Test file | Coverage area |
|---|---|
test_resources.py | CRUD, publish guard, versioning, tombstones |
test_search.py | Keyword search, suggest, catalog stats, licenses, themes |
test_fair.py | FAIR score computation across all dimensions |
test_rdf.py | Turtle parsing, catalog graph, content negotiation |
test_access.py | Access request lifecycle, private download guard |
test_audit.py | Download logging, notifications, stats endpoint |
test_oai.py | OAI-PMH verbs, deleted record handling |
test_health.py | Health endpoint, static assets, root response |
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
Authentication required on all requests |
DEV_AUTH_ENABLED not active in the running container |
Confirm .env exists and set the variable; rebuild: docker compose build api && docker compose up -d --force-recreate api |
Download link opens http://rustfs:9000/… in browser |
Presigned URL uses Docker hostname | Set S3_PUBLIC_ENDPOINT=http://localhost:9002 in .env and recreate the API container |
Port 9000 already in use |
Another S3 service on the host (e.g. old MinIO stack) | Compose maps RustFS to loopback 9002 (S3) and 9003 (console). Use S3_PUBLIC_ENDPOINT=http://localhost:9002. |
Search returns 500 with a query string |
Known FTS issue (fixed) — image out of date | Rebuild: docker compose build api && docker compose up -d --force-recreate api |
Port 5432 already in use |
Another PostgreSQL on the host | Compose maps to host port 5434 (5434:5432). Connect via localhost:5434. |
Port 8000 or 8001 already in use |
Another API on the host (e.g. legacy stack on 8001) | Compose maps to host port 8002 (8002:8000). Use http://localhost:8002. |
Port 6379 bind error on redis |
Old compose published Redis on the host | Redis is no longer published to the host. Run docker compose down then docker compose up -d. |
Same page on 5173 and 5174 |
Two Vite dev servers (second bound 5174 when 5173 was busy) |
cd frontend && npm run dev:stop then npm run dev. Use only 5173. Compose no longer includes a Docker Vite service. |
Port 5173 is already in use |
Another Vite or process holds 5173 |
cd frontend && npm run dev:stop, or end the other terminal's npm run dev. Check: ss -tlnp | grep 5173 |
Nothing at http://localhost:5173 |
Vite not running | cd frontend && npm run dev (API on 8002), or use http://localhost:8002/ui/ after npm run build |
EACCES on node_modules/.vite |
Root-owned cache (e.g. old Docker Vite container) | From fairdatahive/: docker run --rm -v "$(pwd)/frontend:/app" -w /app node:20-alpine rm -rf /app/node_modules/.vite/deps then npm run dev |
| Keyword search returns no results | Query not in catalog text, or API image out of date | Rebuild API; try full words or theme names. Partial matches use ILIKE fallback when FTS is empty. |
Temporary failure in name resolution for postgres |
.env missing or postgres not running |
Run cp .env.example .env then docker compose up -d postgres and wait for healthy |
| Migrations fail on API startup | Database not ready or schema drift | Wait for healthy Postgres, then: docker compose run --rm --entrypoint alembic api upgrade head. Migrations run automatically via RUN_MIGRATIONS=true in the API entrypoint. |
| Publish fails with FAIR score error | Score below 0.40 threshold | Check GET /resources/{id}/fair-score for failing checks and suggestions |
Useful diagnostic commands
bash# Check dev auth is active
curl http://localhost:8002/api/v1/users/me
# Should return {"sub":"dev-user",…} without any Authorization header
# Check service health
curl http://localhost:8002/api/v1/health
# View API logs
docker compose logs api --tail 50 -f
# Confirm env inside the container
docker compose exec api printenv DEV_AUTH_ENABLED
# Stop stray Vite on 5173 / 5174
cd frontend && npm run dev:stop
# Rebuild API after code or migration changes
docker compose build api && docker compose up -d --force-recreate api
FairDataHive v1.0.0 · Built on FastAPI, PostgreSQL, pgvector, RustFS, Redis · Get started