Filesystem Data Provider

What it does

The filesystem data provider serves folders of Parquet files as data that apps query in SQL. Each folder is a mount: apps open it by name and query its files as tables, and, where its rules allow, add new files to it. One provider serves any number of mounts. See the Overview for how it fits with the other components.

Quick start

Describe the provider in the deployment description, deployment.kdl: the host it runs on and each folder of data files it serves.

fs-data-provider {
    host "providers-1"
    fs-data-mount "warehouse" {
        path "/srv/data/warehouse"
        view "events_2026" file="2026/events.parquet"
        issuer "https://sign-in.example.com/realms/acme" {
            allow claim="groups" array_contains="analysts" rights="r"
        }
    }
}

Every mount needs at least one rule; allow anyone rights="r" lets anyone who reaches the provider read it with no sign-in.

Make its files with the deployment tool, generate or add-provider.

On the host, install the program, then follow the README.md in the host's folder of the bundle, which installs the rest and starts it:

sudo install -D -m 755 fs-data-provider-linux-x86_64 \
    /opt/datumhue/fs-data-provider

Its log names its public key, which matches the line for it in the bundle's pubkeys.txt, and then each announcement.

How it works

Mounts. Each fs-data-mount maps a name to a folder. The provider announces each mount to the aggregator, with a capability for data/<mount>, and workstations find it there. It serves the files in place and keeps no copy.

Tables. A query names a table by its path inside the mount:

• A file, by its whole name, extension included: SELECT * FROM "kpi-2026-q1.parquet". Any extension works, so one mount can hold .parquet and .parq files.
• A folder, with a trailing /, reads every file under it as one table: SELECT * FROM "2026/q1/". Every file there must be Parquet.
• A view, a name the mount's settings give one file: SELECT * FROM "events_2026".

SHOW TABLES lists the views and nothing else; _ls() lists the mount's files and folders, one folder at a time.

Writing. An INSERT into a folder adds a new Parquet file to it. The folder must already hold a Parquet file whose columns the new rows match.

Command line

fs-data-provider <COMMAND>

fs-data-provider serve

fs-data-provider serve [OPTIONS]

Configuration

The settings file is KDL: one fs-data-provider block, then one fs-data-mount block for each folder.

fs-data-provider {
    secret-key-file "/var/lib/datumhue/fs-data-provider/provider.key"
    capability-file "/var/lib/datumhue/fs-data-provider/provider.caps"
    aggregator-url "https://aggregator.example.com"
    relay-url "https://aggregator.example.com"
    query-memory-limit-mb 2048
    query-timeout-secs 60
}

fs-data-mount "warehouse" {
    path "/srv/data/warehouse"
    description "Event warehouse"
    view "events_2026" file="2026/events.parquet"
    issuer "https://sign-in.example.com/realms/acme" {
        allow claim="groups" array_contains="analysts" rights="r"
    }
}

Common provider settings

Every provider's settings block takes these keys beside its own. A relative path anywhere in the settings file resolves against the folder that holds the file, not the folder the provider starts in.

The key and the capability file. The secret key is the provider's identity: the provider creates it at its first start when the file is missing, and logs its public key at every start. The capability file holds the capabilities that let the provider announce what it serves. The provider reads it again before every announcement and rewrites it as capabilities renew, so it refuses to start when it cannot write to the file's folder. See Provider authorization.

The access log. At info, on the log target datumhue_audit, the provider logs each access it allows or refuses, each sign-in it cannot verify, and each session it ends because a new sign-in no longer passes the rules. A mount or channel's open is logged, and the operations it then allows are not, one by one. A refusal that repeats is logged with a count, in repeated, not each time. RUST_LOG=info keeps the access log; RUST_LOG=info,datumhue_audit=off turns it off alone, and a RUST_LOG of warn or error, or one that names only other targets, leaves it out.

Each line has the message access and the fields provider, action, resource, endpoint, iss, sub, outcome, rights, basis, claim, reason, credential and repeated, with - where a field has nothing to say. It names the mount, channel or scope, the connecting program's endpoint id, the signed-in person, the outcome (allow, deny, refused, verify_failed or session_dropped), for a refusal the claim that failed, and in repeated how often a refusal came. It never holds a token, a secret or the value a rule expects.

Filesystem data provider settings

Mounts

Access rules

Access rules say who may use a resource. A rule that reads a sign-in sits in the block of the sign-in service whose tokens it reads, named by its issuer URL exactly as those tokens carry it in iss. datumhue login prints that URL. Each rule names who it admits and the rights it grants:

issuer "https://sign-in.example.com/realms/acme" {
    require claim="email_verified" equals=#true
    allow claim="groups" array_contains="analysts" rights="r"
}

• rights is "r" (read) or "rw" (read and write). A rule admits a request when it matches and its rights cover what the request asks for. Rules add up: any one rule that admits a request is enough.
• require takes the same claim and match, without rights. Every require of a block must hold before any of its rules counts.
• A sign-in counts only in the block of the issuer that signed it. The URL is compared exactly: https://accounts.google.com and https://accounts.google.com/ are two issuers. A sign-in from an issuer with no block gets nothing a block grants.
• equals is typed. equals="true" matches the text true, never the boolean; a number must be quoted and is compared as text.
• claim names a top-level claim of the person's ID token. A value nested in another claim, such as Keycloak's realm_access.roles, never matches: map it to a top-level claim at your sign-in service.
• Prefer the groups or roles your sign-in service manages, or sub, over email. On Google, add require claim="hd" equals="<your domain>" to admit only your Workspace accounts.
• The issuer must be one the deployment's Identity Provider serves.
• An email claim counts only when the same token says email_verified is true, and a phone_number claim only when it says phone_number_verified is true. Microsoft Entra ID does not send email_verified: match oid, sub or groups there.
• A sign-in that does not verify is refused, even where allow anyone admits a request without one.
• A refused request learns which claim failed, never the value a rule expects.

For a mount, rights="r" lets a person query it, and rights="rw" also lets them add files with INSERT.

Every mount needs a rule: a mount without one stops the provider at start. allow anyone rights="r", outside every issuer block, lets anyone who reaches the provider, directly or through a relay of the deployment, query the mount with no sign-in. Keep that for public data; the provider logs a warning for it at start. allow anyone never grants rw: writing always takes a sign-in a rights="rw" rule admits. To admit everyone a sign-in service signs in, put allow signed-in in its block.

Network, state and dependencies

Network surface

The provider listens on no TCP port; workstations reach it through the aggregator. It takes 128 connections at most, a workstation using one for all its apps' mounts, and runs 32 queries at a time.

Persistent state

A query reads a file's columns when it runs, so a replaced file's new columns show in the next query.

External dependencies

• The aggregator, for announcing and for the sign-in services' keys. Within a minute of losing it, the provider refuses sign-ins until it can read the keys again.
• The Identity Provider, when a mount has an issuer block.

Running as a service

systemd

The deployment tool writes a unit like this one for every provider it sets up. To run the provider under systemd by hand, install the program as /opt/datumhue/fs-data-provider, its settings as /etc/datumhue/fs-data-provider.kdl, and give the datumhue user the state folder /var/lib/datumhue/fs-data-provider/.

/etc/systemd/system/datumhue-fs-data-provider.service:

[Unit]
Description=DatumHue Filesystem Data Provider
After=network.target

[Service]
Type=simple
User=datumhue
Group=datumhue
WorkingDirectory=/var/lib/datumhue/fs-data-provider
ExecStart=/opt/datumhue/fs-data-provider serve \
    --config /etc/datumhue/fs-data-provider.kdl
Environment=DATUMHUE_CERT_PATH=/etc/datumhue/ca-cert.pem
EnvironmentFile=/etc/datumhue/keystore.env
Restart=always
RestartSec=10
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/datumhue/fs-data-provider

[Install]
WantedBy=multi-user.target

• The unit keeps /etc/datumhue read-only, and a relative path in the settings file resolves there. Point secret-key-file, capability-file and every other file the provider writes into /var/lib/datumhue/fs-data-provider/.
• keystore.env holds DATUMHUE_KEYSTORE_PASSPHRASE=...; keep it mode 600. Drop the EnvironmentFile= line for a plaintext key.
• Drop the DATUMHUE_CERT_PATH line when the aggregator's certificate comes from a public CA.
• On the aggregator's host, add datumhue-aggregator.service to After=.

sudo systemctl daemon-reload
sudo systemctl enable --now datumhue-fs-data-provider
journalctl -u datumhue-fs-data-provider -f

systemctl stop stops the provider the way Ctrl+C does: it stops announcing and gives its provider seat back, so the next start takes a seat at once.

Add each mount's folder to ReadOnlyPaths=, or to ReadWritePaths= for a mount with a rights="rw" rule.

Docker

To run the provider in a container, build an image from this Dockerfile, in the folder that holds the program:

FROM debian:bookworm-slim
RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*
COPY fs-data-provider-linux-x86_64 /opt/datumhue/fs-data-provider
ENTRYPOINT ["/opt/datumhue/fs-data-provider"]
CMD ["serve", "--config", "/etc/datumhue/fs-data-provider.kdl"]

Run it as the datumhue user that owns the state folder, with the settings folder read-only:

docker build -t datumhue/fs-data-provider .
state=/var/lib/datumhue/fs-data-provider
docker run -d --name datumhue-fs-data-provider --restart always \
    --user "$(id -u datumhue):$(id -g datumhue)" \
    --env-file /etc/datumhue/keystore.env \
    -e DATUMHUE_CERT_PATH=/etc/datumhue/ca-cert.pem \
    -v /etc/datumhue:/etc/datumhue:ro -v "$state:$state" \
    datumhue/fs-data-provider

The docker-compose.yml the deployment tool writes runs images by these names, with the same folders.

Mount each data folder into the container as well, read-only unless a rights="rw" rule lets people write to it: -v /srv/data:/srv/data:ro.

Security notes

• Every mount names who may use it. A mount without rules does not load. allow anyone lets anyone who reaches the provider, directly or through a relay of the deployment, query a mount with no sign-in, and the provider logs a warning for it at start: keep it to public data.
• Queries stay inside the mount. A query reaches the host's files only through the mount: it cannot copy to, or read from, a path outside it. The provider still reads, and on rw mounts writes, with its own user's permissions, so never point a mount at a folder wider than you mean to share.
• Rules grant whole files. A person a rule admits reads every column of every file in the mount. Keep data for different groups in different files or mounts.
• Read-only sessions refuse writes, whatever the query says.
• Access ends with the sign-in. A mount an issuer rule admitted lasts no longer than the person's current ID token, and each refreshed sign-in must pass the rules again, so access you withdraw at your sign-in service ends by the time that token expires.
• Sign-in key changes apply within a minute. A change to a sign-in service's keys, a revocation included, reaches the provider within a minute.
• Settings apply at start. New mounts, rules and views take effect when the provider restarts.
• Protect the key. The deployment tool encrypts it with your passphrase; a key the provider makes at its first start is plaintext unless DATUMHUE_KEYSTORE_PASSPHRASE is set. The provider refuses a key file that others can read.

Troubleshooting

Messages. The provider writes its log to standard output; under systemd, read it with journalctl -u and the unit's name.

The aggregator is refused as untrusted. It serves the certificate it made itself: point DATUMHUE_CERT_PATH at its tls/ca-cert.pem.

provider lease acquisition failed. With QuotaExhausted, every provider seat of the license is taken: stop a provider, or extend the license.

The seat or announcements are refused. capability-file holds no current capability for this key: make one with datumhue-deploy reissue-capabilities.

The provider stops at start, naming relay-url. Set relay-url to the aggregator's public HTTPS URL.

The provider refuses its key file. A key file that others can read is refused: chmod 600 it. DATUMHUE_KEYSTORE_PASSPHRASE must be set for an encrypted key and unset for a plaintext one; the message says which the key is.

The provider cannot write its capability file. Give the provider's user write access to the folder that holds the file; it rewrites the file as capabilities renew.

Apps cannot find a mount. The log shows announcer running once the provider announces. Without it, the capability file lacks a current capability for data/<mount>.

A query says the table does not exist. Without a view, a table's name is the file's whole name, extension included: kpi-2026-q1.parquet, not kpi-2026-q1. List the files with _ls(); SHOW TABLES lists only views.

A view cannot be queried, naming glob=. Query the folder with a trailing / instead, such as SELECT * FROM "2026/q1/", or give the view one file with file=.

The provider stops at start, naming a mount's access rules. A mount needs at least one rule, a rule that reads a sign-in sits in an issuer block, and allow anyone grants rights="r" only. The message says what to write; datumhue login prints the issuer URL a block names.

A request is refused. With no sign-in, or an expired one, the answer is authentication required; when no rule matches, it is access denied, naming the claim that failed, or iss when the rules have no block for the person's sign-in service. Check the person's claims against the mount's rules.

A write is refused. Writing takes a rights="rw" rule that the person's claims match; allow anyone only reads.