Database Data Provider

What it does

The database data provider serves SQL databases to the apps of a deployment. Each database is a mount: apps open it by name and query it in the database's own SQL, and, where its rules allow, change it. One provider serves any number of mounts. Its program, its settings blocks and its zip are named adbc-data-provider, after the database drivers it loads. See the Overview for how it fits with the other components.

Quick start

Install your database's driver on the provider's host: for PostgreSQL, libadbc_driver_postgresql.so and libpq. Then describe the provider in deployment.kdl: the host it runs on and each database it serves.

adbc-data-provider {
    host "providers-1"
    adbc-data-mount "warehouse" {
        library "/usr/lib/x86_64-linux-gnu/libadbc_driver_postgresql.so"
        option "uri" value="postgresql://reader@db.internal:5432/sales"
        credential "operator"
        credential-secret-file "/var/lib/datumhue/adbc-data-provider/db.secret"
        issuer "https://sign-in.example.com/realms/acme" {
            allow claim="groups" array_contains="analysts" rights="r"
        }
    }
}

Make its files with the deployment tool, generate or add-provider, then put the password of reader, a user with only SELECT (not a superuser), in the bundle's providers-1/state/adbc-data-provider/db.secret, mode 600.

On the host, install the program, then follow the README.md in the host's folder of the bundle, which installs the rest and starts it:

sudo install -D -m 755 adbc-data-provider-linux-x86_64 \
    /opt/datumhue/adbc-data-provider

Its log names its public key, which matches the line for it in the bundle's pubkeys.txt, and then each announcement.

How it works

Drivers. The provider reaches a database through an ADBC driver, a shared library you install on its host; the zip holds no driver. Each mount names its driver's file by its full path: the provider looks nowhere else. For PostgreSQL, install libadbc_driver_postgresql.so and the libpq it needs. To serve another kind of database, install its driver and give it a mount of its own.

Mounts. Each adbc-data-mount names a database and how to connect to it. The provider announces each mount to the aggregator, with a capability for data/<mount>, and workstations find it there. It keeps no copy of the data.

Command line

adbc-data-provider <COMMAND>

adbc-data-provider serve

adbc-data-provider serve [OPTIONS]

Configuration

The settings file is KDL: one adbc-data-provider block, then one adbc-data-mount block for each database.

adbc-data-provider {
    secret-key-file "/var/lib/datumhue/adbc-data-provider/provider.key"
    capability-file "/var/lib/datumhue/adbc-data-provider/provider.caps"
    aggregator-url "https://aggregator.example.com"
    relay-url "https://aggregator.example.com"
    query-timeout-secs 60
}

adbc-data-mount "warehouse" {
    description "Sales warehouse"
    library "/usr/lib/x86_64-linux-gnu/libadbc_driver_postgresql.so"
    engine-label "PostgreSQL 16"
    option "uri" value="postgresql://reader@db.internal:5432/sales"
    credential "operator"
    credential-secret-file "/etc/datumhue/warehouse.secret"
    issuer "https://sign-in.example.com/realms/acme" {
        allow claim="groups" array_contains="analysts" rights="rw"
    }
}

Common provider settings

Every provider's settings block takes these keys beside its own. A relative path anywhere in the settings file resolves against the folder that holds the file, not the folder the provider starts in.

The key and the capability file. The secret key is the provider's identity: the provider creates it at its first start when the file is missing, and logs its public key at every start. The capability file holds the capabilities that let the provider announce what it serves. The provider reads it again before every announcement and rewrites it as capabilities renew, so it refuses to start when it cannot write to the file's folder. See Provider authorization.

The access log. At info, on the log target datumhue_audit, the provider logs each access it allows or refuses, each sign-in it cannot verify, and each session it ends because a new sign-in no longer passes the rules. A mount or channel's open is logged, and the operations it then allows are not, one by one. A refusal that repeats is logged with a count, in repeated, not each time. RUST_LOG=info keeps the access log; RUST_LOG=info,datumhue_audit=off turns it off alone, and a RUST_LOG of warn or error, or one that names only other targets, leaves it out.

Each line has the message access and the fields provider, action, resource, endpoint, iss, sub, outcome, rights, basis, claim, reason, credential and repeated, with - where a field has nothing to say. It names the mount, channel or scope, the connecting program's endpoint id, the signed-in person, the outcome (allow, deny, refused, verify_failed or session_dropped), for a refusal the claim that failed, and in repeated how often a refusal came. It never holds a token, a secret or the value a rule expects.

Database data provider settings

Mounts

The database password. Keep it out of the settings file: put it in credential-secret-file, readable only by the provider's user, or let the database trust the provider's host and use credential "none". With "operator", the password goes into the uri option's user. A credential-secret-file that is empty, or one without credential "operator", stops the start.

Access rules

Access rules say who may use a resource. A rule that reads a sign-in sits in the block of the sign-in service whose tokens it reads, named by its issuer URL exactly as those tokens carry it in iss. datumhue login prints that URL. Each rule names who it admits and the rights it grants:

issuer "https://sign-in.example.com/realms/acme" {
    require claim="email_verified" equals=#true
    allow claim="groups" array_contains="analysts" rights="r"
}

• rights is "r" (read) or "rw" (read and write). A rule admits a request when it matches and its rights cover what the request asks for. Rules add up: any one rule that admits a request is enough.
• require takes the same claim and match, without rights. Every require of a block must hold before any of its rules counts.
• A sign-in counts only in the block of the issuer that signed it. The URL is compared exactly: https://accounts.google.com and https://accounts.google.com/ are two issuers. A sign-in from an issuer with no block gets nothing a block grants.
• equals is typed. equals="true" matches the text true, never the boolean; a number must be quoted and is compared as text.
• claim names a top-level claim of the person's ID token. A value nested in another claim, such as Keycloak's realm_access.roles, never matches: map it to a top-level claim at your sign-in service.
• Prefer the groups or roles your sign-in service manages, or sub, over email. On Google, add require claim="hd" equals="<your domain>" to admit only your Workspace accounts.
• The issuer must be one the deployment's Identity Provider serves.
• An email claim counts only when the same token says email_verified is true, and a phone_number claim only when it says phone_number_verified is true. Microsoft Entra ID does not send email_verified: match oid, sub or groups there.
• A sign-in that does not verify is refused, even where allow anyone admits a request without one.
• A refused request learns which claim failed, never the value a rule expects.

For a mount, rights="r" lets a person query the database, and rights="rw" also lets them change it: INSERT, UPDATE, DELETE and any other statement the database lets the connecting user run.

A rights="r" session runs in a read-only transaction, in which the database refuses writes to its tables. PostgreSQL supports this; a database that cannot make a session read-only refuses read-only sessions, so serve it through rights="rw" rules only.

Beyond that, a query can do what the mount's database user can do: use its grants, EXECUTE on a function such as pg_read_file among them; call functions that run as their owner (SECURITY DEFINER); and use extensions such as dblink, which opens a session of its own that is not read-only. Give the user only what the mount should expose, such as SELECT on the tables it serves. To let some people change the database and others only read it, serve it through two mounts: one whose user can write, for the rights="rw" rules, and one whose user only reads.

On PostgreSQL, the provider refuses a database user that is, or can become, a superuser or a role with CREATEROLE, or that is a member of pg_execute_server_program, pg_read_server_files or pg_write_server_files: such a user runs programs and reads or writes files on the database's host from any statement, read-only or not. The provider stops at start for such a user, and refuses a running mount's queries while its user is one.

An embedded database, such as SQLite, runs inside the provider and has no users of its own: a query reaches files as the provider's system user. A session can open any database file that user can read, with ATTACH DATABASE, and create new files wherever it can write, with ATTACH DATABASE or VACUUM INTO. Under the generated service unit it writes only to the provider's state folder and the folders you add to the unit's ReadWritePaths, so keep nothing there, or readable to that user elsewhere, that the mount's users should not reach. SQLite sessions cannot be made read-only, so serve SQLite through rights="rw" rules only.

query-timeout-secs bounds a query only until its first rows. To bound how long a query may run in all, including the time its rows take to reach the workstation, set the user's own timeout, for example ALTER ROLE reader SET statement_timeout = '5min'.

Every mount needs a rule: a mount without one stops the provider at start. allow anyone rights="r", outside every issuer block, lets anyone who reaches the provider, directly or through a relay of the deployment, query the database with no sign-in, as the user the mount connects as; the provider logs a warning for it at start. allow anyone never grants rw. To admit everyone a sign-in service signs in, put allow signed-in in its block.

scope and scope-in-claim, which only publish rules use, are refused on a mount.

Network, state and dependencies

Network surface

The provider listens on no TCP port; workstations reach it through the aggregator. It takes 128 connections at most, a workstation using one for all its apps' mounts, and runs 32 queries at a time.

Persistent state

The data stays in the database; the provider keeps none of it.

External dependencies

• The database, reachable from the provider's host at each mount's connection string.
• The driver, and the libraries it needs, on the provider's host.
• The aggregator, for announcing and for the sign-in services' keys. Within a minute of losing it, the provider refuses sign-ins until it can read the keys again.
• The Identity Provider, when a mount has an issuer block.

Running as a service

systemd

The deployment tool writes a unit like this one for every provider it sets up. To run the provider under systemd by hand, install the program as /opt/datumhue/adbc-data-provider, its settings as /etc/datumhue/adbc-data-provider.kdl, and give the datumhue user the state folder /var/lib/datumhue/adbc-data-provider/.

/etc/systemd/system/datumhue-adbc-data-provider.service:

[Unit]
Description=DatumHue Database Data Provider
After=network.target

[Service]
Type=simple
User=datumhue
Group=datumhue
WorkingDirectory=/var/lib/datumhue/adbc-data-provider
ExecStart=/opt/datumhue/adbc-data-provider serve \
    --config /etc/datumhue/adbc-data-provider.kdl
Environment=DATUMHUE_CERT_PATH=/etc/datumhue/ca-cert.pem
EnvironmentFile=/etc/datumhue/keystore.env
Restart=always
RestartSec=10
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/datumhue/adbc-data-provider

[Install]
WantedBy=multi-user.target

• The unit keeps /etc/datumhue read-only, and a relative path in the settings file resolves there. Point secret-key-file, capability-file and every other file the provider writes into /var/lib/datumhue/adbc-data-provider/.
• keystore.env holds DATUMHUE_KEYSTORE_PASSPHRASE=...; keep it mode 600. Drop the EnvironmentFile= line for a plaintext key.
• Drop the DATUMHUE_CERT_PATH line when the aggregator's certificate comes from a public CA.
• On the aggregator's host, add datumhue-aggregator.service to After=.

sudo systemctl daemon-reload
sudo systemctl enable --now datumhue-adbc-data-provider
journalctl -u datumhue-adbc-data-provider -f

systemctl stop stops the provider the way Ctrl+C does: it stops announcing and gives its provider seat back, so the next start takes a seat at once.

Install the driver on the host before you start the provider, so each library path resolves for the provider's user.

Docker

To run the provider in a container, build an image from this Dockerfile, in the folder that holds the program:

FROM debian:bookworm-slim
RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*
COPY adbc-data-provider-linux-x86_64 /opt/datumhue/adbc-data-provider
ENTRYPOINT ["/opt/datumhue/adbc-data-provider"]
CMD ["serve", "--config", "/etc/datumhue/adbc-data-provider.kdl"]

Run it as the datumhue user that owns the state folder, with the settings folder read-only:

docker build -t datumhue/adbc-data-provider .
state=/var/lib/datumhue/adbc-data-provider
docker run -d --name datumhue-adbc-data-provider --restart always \
    --user "$(id -u datumhue):$(id -g datumhue)" \
    --env-file /etc/datumhue/keystore.env \
    -e DATUMHUE_CERT_PATH=/etc/datumhue/ca-cert.pem \
    -v /etc/datumhue:/etc/datumhue:ro -v "$state:$state" \
    datumhue/adbc-data-provider

The docker-compose.yml the deployment tool writes runs images by these names, with the same folders.

Add the driver's packages to the image's apt-get install line, and set each mount's library to where they install it.

Security notes

• Keep the password out of the settings file, in a credential-secret-file only the provider's user reads. The provider never logs it.
• Every mount names who may use it. A mount without rules does not load. allow anyone lets anyone who reaches the provider, directly or through a relay of the deployment, query the database with the mount's credential and no sign-in, and the provider logs a warning for it at start: keep it to public data.
• Writing takes a rights="rw" rule. Read-only sessions are enforced by the database itself. Beyond that, a query can do what the mount's database user can do, so give it only the grants the mount should expose. On PostgreSQL the provider refuses a user that can run programs or reach files on the database's host.
• An embedded database reaches files as the provider. A SQLite mount's users can read database files the provider's user can read, and create files where it can write: confine the provider with its service unit, as Access rules describes.
• People who query one mount share its database user: each can see and cancel the statements the others run.
• Access ends with the sign-in. A mount an issuer rule admitted lasts no longer than the person's current ID token, and each refreshed sign-in must pass the rules again, so access you withdraw at your sign-in service ends by the time that token expires.
• Sign-in key changes apply within a minute. A change to a sign-in service's keys, a revocation included, reaches the provider within a minute.
• Settings apply at start. New mounts and changed rules take effect when the provider restarts.
• Protect the key. The deployment tool encrypts it with your passphrase; a key the provider makes at its first start is plaintext unless DATUMHUE_KEYSTORE_PASSPHRASE is set. The provider refuses a key file that others can read.

Troubleshooting

Messages. The provider writes its log to standard output; under systemd, read it with journalctl -u and the unit's name.

The aggregator is refused as untrusted. It serves the certificate it made itself: point DATUMHUE_CERT_PATH at its tls/ca-cert.pem.

provider lease acquisition failed. With QuotaExhausted, every provider seat of the license is taken: stop a provider, or extend the license.

The seat or announcements are refused. capability-file holds no current capability for this key: make one with datumhue-deploy reissue-capabilities.

The provider stops at start, naming relay-url. Set relay-url to the aggregator's public HTTPS URL.

The provider refuses its key file. A key file that others can read is refused: chmod 600 it. DATUMHUE_KEYSTORE_PASSPHRASE must be set for an encrypted key and unset for a plaintext one; the message says which the key is.

The provider cannot write its capability file. Give the provider's user write access to the folder that holds the file; it rewrites the file as capabilities renew.

Apps cannot find a mount. The log shows announcer running once the provider announces. Without it, the capability file lacks a current capability for data/<mount>.

load ADBC driver fails, at start or at the first query. The library path is wrong, or the driver or a library it needs, such as libpq, is not installed. Give library the full path.

open database or open connection fails. The connection string is wrong, the database cannot be reached from the provider's host, or it refused the credentials.

The log says database not checked at start. The provider could not check a PostgreSQL mount's database user; the rest of the line says why. It starts anyway, and refuses the mount's queries while the check fails. The check fails when the database cannot be reached or does not answer within query-timeout-secs, when the uri is not a PostgreSQL connection string (a URI or keyword=value pairs), when the database only speaks PostgreSQL's protocol, and behind a connection pooler that drops the connection's options: connect such a mount to the database directly.

The provider stops at start, or refuses a mount's queries, naming its database user. The user is, or can become, a superuser or a role with CREATEROLE, or is a member of pg_execute_server_program, pg_read_server_files or pg_write_server_files. Apps see is refused: its database user is not allowed, and the access log records the refused queries with the reason database_user_not_allowed. Connect the mount as a user with only the grants it should expose, such as SELECT on its tables.

A read-only query fails with read-only setup. The database cannot make a session read-only. Serve it through rights="rw" rules, or use a database that can, such as PostgreSQL.

The provider stops at start, naming a mount's access rules. A mount needs at least one rule, a rule that reads a sign-in sits in an issuer block, and allow anyone grants rights="r" only. The message says what to write; datumhue login prints the issuer URL a block names.

A request is refused. With no sign-in, or an expired one, the answer is authentication required; when no rule matches, it is access denied, naming the claim that failed, or iss when the rules have no block for the person's sign-in service. Check the person's claims against the mount's rules.