Aggregator

What it does

The aggregator is the center of a deployment. Workstations, browsers and providers all connect to it at its public HTTPS URL: it serves the browser workstation, keeps the list of what each provider serves, checks the license and each provider's capabilities, and carries traffic between workstations and providers that cannot reach each other directly. See the Overview for how it fits with the other components.

Quick start

Put the license file DatumHue sent you, customer.dhlicense, in the aggregator's state folder, make the deployment secret beside it and an access key, then start the aggregator with the public URL its host answers at:

chmod +x aggregator-linux-x86_64
sudo mkdir -p /var/lib/datumhue/aggregator
sudo cp customer.dhlicense /var/lib/datumhue/aggregator/
sudo ./aggregator-linux-x86_64 deployment-secret generate \
  /var/lib/datumhue/aggregator/deployment.secret
export DATUMHUE_ACCESS_KEY="$(head -c 32 /dev/urandom | base64 \
  | tr '+/' '-_' | tr -d '=')"
sudo --preserve-env=DATUMHUE_ACCESS_KEY ./aggregator-linux-x86_64 serve \
  --relay-url https://aggregator.example.com \
  --discovery-url https://aggregator.example.com \
  --static-dir ./static \
  --license /var/lib/datumhue/aggregator/customer.dhlicense \
  --deployment-secret-path /var/lib/datumhue/aggregator/deployment.secret \
  --aggregator-key-path /var/lib/datumhue/aggregator/aggregator.key \
  --replication-store-dir /var/lib/datumhue/aggregator/replication \
  --tls-dir /var/lib/datumhue/aggregator/tls

The deployment secret signs every capability of the deployment and the access key admits its people: keep both in your secret store. Workstations set the key as DATUMHUE_ACCESS_KEY, and a browser opens the access link, https://aggregator.example.com/?key= and the key, once. The log says License loaded and verified, the deployment key and the access key id. The first start writes a certificate authority and a server certificate into the tls folder and keeps them: give workstations and providers its ca-cert.pem through DATUMHUE_CERT_PATH, or serve a certificate of your own.

On a host the deployment tool set up, install the program as /opt/datumhue/aggregator and static/ as /opt/datumhue/static, then follow the README.md in the host's folder of the bundle.

Either way, open the access link, the public URL with ?key= and the access key, in a browser to reach the browser workstation, then set up the providers.

How it works

One HTTPS port, 443 by default, carries the browser workstation, the deployment's HTTP API and the relay, which carries traffic between a workstation and a provider that cannot connect directly. Three UDP ports, 7842, 3478 and 7843, serve direct connections; see Network surface.

Announcements. Each provider tells the aggregator what it serves, a mount or a namespace, and repeats it before it lapses; workstations ask the aggregator who serves a name. The aggregator accepts an announcement only with a capability for that name; see Provider authorization.

The license sets how many sessions, aggregators and providers the deployment may run. The aggregator checks it at every start and renews it before it ends.

The deployment secret signs the deployment's capabilities and ties its aggregators together. Every aggregator of the deployment holds the same secret; providers and workstations need no copy.

The access key admits the deployment's people. The workstation app and the command line present it from DATUMHUE_ACCESS_KEY, and a browser carries it in the cookie the access link sets; a provider presents a capability of its own instead. Every aggregator of the deployment needs the same key.

Several aggregators of one deployment, for regions or for availability, share the list of announcements, published keys and revocations among themselves. A provider announcing to one is found through all of them.

The license

The license file, .dhlicense, sets the deployment's validity window and its session, aggregator and provider counts, and says whether the deployment is airgapped. Every aggregator of the deployment loads the same file; providers and workstations need no copy.

At start the log reports License loaded and verified, with a fingerprint of the license's id, its window and counts. A file whose signatures do not verify, or that is not an Enterprise deployment license, stops the start. Started with an expired license, the aggregator serves nothing and waits for a new one: from the renewal service, or for an airgapped license, in the file --license names. aggregator license inspect prints a license file's contents and whether its signatures verify, without starting the server.

Getting a license. Buy an Enterprise subscription on datumhue.com, choosing the session, aggregator and provider counts at checkout. The license arrives by email to your billing contact, and the re-send page linked from the Enterprise page sends it again.

Renewal. Ahead of the license's end, the aggregator asks DatumHue's renewal service for a new license, writes it over the file --license names and uses it at once; connections stay up. It keeps asking until the license ends. Send the aggregator SIGHUP to ask at once, for example after a billing problem is solved; this also works while it waits on an expired license.

A host without internet access. Renew with the offline helper in the aggregator's zip, datumhue-license-renew-linux-x86_64, on any machine that reaches the renewal service. Holding the current license is all the helper needs:

./datumhue-license-renew-linux-x86_64 \
  --current-license customer.dhlicense \
  --out renewed.dhlicense

Copy the renewed file over the one --license names, then send the aggregator SIGHUP or restart it. The helper also renews a license whose subscription lapsed, once the subscription is paid again: upload the license on the resume page linked from the Enterprise page, and the new subscription continues the same license.

An airgapped license lets the deployment run without contacting the renewal service at all; the aggregator logs that it does not. Renew it with the offline helper wherever there is a connection, write it over the license file and restart the aggregator. Providers and workstations need no restart: within about half a minute they take their seats again and reappear.

A new or renewed license leaves every capability in force.

Changing the license size. The resize page linked from the Enterprise page takes the license and the new counts, and applies them once your billing contact confirms with an emailed code. An increase applies at once and a decrease at the next billing period; install the new license file as above.

Adding an aggregator. The license sets how many aggregators may run under it: before adding one past that count, raise it as above. The new host needs the same license file, the deployment's deployment.secret and its access key.

The deployment secret

Every aggregator of the deployment loads the same deployment secret file, deployment.secret, from --deployment-secret-path, readable by the aggregator's user alone. An encrypted file is read with DATUMHUE_KEYSTORE_PASSPHRASE, as the aggregator's key is. At start the log reports deployment key: <fingerprint>, the same on every aggregator of the deployment.

The deployment tool's bundle carries the secret, encrypted with the bundle's passphrase; aggregator deployment-secret generate <PATH> makes one for a deployment set up by hand. It makes the file's folder private to its owner too, so give the secret a folder of its own, such as the aggregator's state folder. Keep a copy in your secret store.

Replacing the secret. datumhue-deploy reissue-capabilities --new-deployment-secret makes a new secret, every provider's capability file under it, and the bundle's state/aggregator/revocations, which keeps the revoked keys revoked. Install the secret, with that file, on every aggregator and restart them together, then install the capability files. Each package provider stops once when its new file arrives, and keeps its catalog when its service starts it again.

Command line

The program has no default action: serve runs the aggregator.

aggregator <COMMAND>

aggregator serve

aggregator serve [OPTIONS]

aggregator tls init-ca

aggregator tls init-ca [OPTIONS] --out-dir <OUT_DIR>

aggregator tls issue-leaf

aggregator tls issue-leaf [OPTIONS] --ca-dir <CA_DIR> --out-dir <OUT_DIR> \
    --hostname <HOSTNAMES>

aggregator tls inspect

aggregator tls inspect <PATH>

aggregator license inspect

aggregator license inspect [OPTIONS] <PATH>

aggregator deployment-secret generate

aggregator deployment-secret generate <PATH>

aggregator lease list

aggregator lease list [OPTIONS]

aggregator lease kick

aggregator lease kick [OPTIONS] [ENDPOINT_ID]...

Configuration

Every setting is a flag of serve, or the environment variable beside it in the table above. A flag wins over its variable, the variable wins over the regions file, and that wins over the default. The access key has a variable and no flag, so it never shows in a process list.

Addresses and URLs

A single aggregator needs --relay-url and --discovery-url, and stops at start without either. Both are usually its one public HTTPS URL. With several aggregators, the regions file sets both for the aggregator --region-id names.

Certificates

The aggregator takes its certificate from the first of these it finds:

1. Your own certificate: --tls-cert-path and --tls-key-path, with --tls-ca-cert-path, a CA certificate, when a private CA signed it. The aggregator serves them as they are and writes no certificate files.
2. Its own certificate authority, the default. The first start writes a CA valid for ten years and a server certificate valid for 90 days into --tls-dir (tls), and later starts reuse them. Give workstations and providers <tls-dir>/ca-cert.pem through DATUMHUE_CERT_PATH.
3. Files made beforehand with aggregator tls init-ca and aggregator tls issue-leaf, on another machine, then copied into --tls-dir. They are the same files the aggregator would write.

The server certificate names the relay URL's host, or the names --tls-hostname gives. At each start the aggregator issues a new one under its CA when the current one has less than 30 days left or names other hosts, so a restart renews it. It logs the time each has left, and warns when the server certificate has less than 14 days and the CA less than 90. aggregator tls inspect prints a certificate's names, validity and fingerprint.

Private keys are written with mode 600 and certificates with mode 644.

Identity key

--aggregator-key-path names the file of the aggregator's identity key, ./aggregator.key unless set. The first start creates it, and every start logs the identity as Aggregator EndpointId. Keep the file on persistent storage: with a new key, providers that were connected wait up to about five minutes before they find the aggregator again.

The first start writes the key plaintext unless DATUMHUE_KEYSTORE_PASSPHRASE is set, and encrypted with it when it is. An encrypted key needs the variable at every start, and a plaintext one needs it unset; otherwise the start stops and says which.

Discovery records

Providers' announcements, their seats and their published keys live in memory unless --replication-store-dir names a folder for them. With the folder, a restarted aggregator serves them at once; without it, they return as providers announce again, within half of each one's announce-ttl, or as another aggregator sends them. The aggregator manages the folder itself, and clears what another version or deployment secret left there. Never move files into it.

Browser workstation files

--static-dir names the folder the aggregator serves at /, the static/ folder of its zip. To serve another build, replace the folder's contents; browsers take the new pages and scripts at their next load. The folder also holds the sign-in pages your sign-in service redirects to, oidc-callback.html and oidc-native-callback.html, each with its script beside it.

Several aggregators

A regions file lists the deployment's aggregators. Each one loads the same file, the same license and the same deployment secret, and names itself with --region-id:

config name="production" {
    aggregator id="us-east" region="us-east-1" {
        base-url "https://aggregator-us-east.example.com"
    }
    aggregator id="eu-west" region="eu-west-1" {
        base-url "https://aggregator-eu-west.example.com"
    }
}

Each aggregator's own listening addresses stay flags, since every aggregator loads the same file:

./aggregator-linux-x86_64 serve \
  --regions-config regions.kdl --config-name production \
  --region-id us-east \
  --tls-cert-path /etc/datumhue/cert.pem \
  --tls-key-path /etc/datumhue/key.pem \
  --license /var/lib/datumhue/aggregator/customer.dhlicense \
  --deployment-secret-path /var/lib/datumhue/aggregator/deployment.secret \
  --aggregator-key-path /var/lib/datumhue/aggregator/aggregator.key

The license's aggregator count must cover them all. Each aggregator reaches the others at their base-url, and only aggregators under the same deployment secret share anything.

Logging

When RUST_LOG is unset, -v turns on debug logging. RUST_LOG sets levels by area, such as RUST_LOG=info,aggregator=debug. The aggregator logs each request at info with its method, path and status, never its query string or headers. It does not log /health, /ping or /relay, nor a request its limits answer with 408, 413 or 503.

At start the aggregator logs the connection limits of port 443 (HTTPS connections: at most …) and the open-file limit they follow.

The log target datumhue_audit, at info, records the sign-in services' signing keys the deployment holds: each service's keys once at start (change=held), then each change to them at any aggregator of the deployment, with the issuer URL, the identity provider that published the keys and the key ids. It also records each workstation lease kick names, as workstation kicked with its endpoint id and whether it held a seat. RUST_LOG=info,datumhue_audit=off turns these lines off, and a RUST_LOG that names only other targets leaves them out.

Access rules

Provider authorization

A provider announces a mount or a namespace only with a capability for it: a signed permission, bound to the provider's key, valid for a limited time and signed under the deployment secret. Without one the aggregator refuses the announcement, so no provider is trusted by default; only your own package provider can announce your datumhue namespace, for example. Workstations check the same capability when they find a provider.

The deployment tool makes capabilities:

• datumhue-deploy generate writes each provider's capability file beside its settings, signed under the bundle's deployment secret.
• datumhue-deploy add-provider writes them for a provider added later.
• datumhue-deploy reissue-capabilities makes a bundle's files again: for capabilities that lapsed while a provider was stopped, or under a new deployment secret.

The capabilities the deployment tool makes last 30 days. A running provider renews its own and rewrites its file. A replaced file applies without a restart, except under a new deployment secret, where a package provider stops once and keeps its catalog when its service starts it again. Each provider logs its public key at start; it matches the line for it in the bundle's pubkeys.txt.

An identity provider needs two capabilities for each sign-in service: one for identity/<issuer>, to announce it by name, and one for the issuer-keys/ namespace of its issuer URL, to publish its keys, which never publishes keys for another service. The capability file names the URL in a comment above that line, and the identity provider logs both at start. A package provider also holds two for its namespace. The deployment tool makes them all.

Published keys last at most a day, and no longer than the capability that published them; a running identity provider publishes them again before then.

Revocation

A revocation withdraws a provider key's right to a namespace at once, before its capability ends. datumhue-deploy revoke signs revocations under the deployment secret and datumhue-deploy lift signs lifts; each submits them to an aggregator, or writes them to a file for datumhue-deploy submit on another host. Any aggregator takes them, and they reach every aggregator of the deployment. A revoked key can neither announce nor renew, and lookups of the namespace leave its provider out, until a lift replaces the revocation: the newest record for a key and namespace wins.

In a sign-in service's issuer-keys/ namespace, a revoked key can no longer publish keys, and the keys it published drop out of every lookup at once; a lift brings back those that have not lapsed. Providers that check sign-ins stop using the keys within a minute.

Within five minutes of a revocation, workstations send the revoked provider no new request for a filesystem, data or egress mount, and take no channel keys or events from it for an ingress channel.

Each aggregator keeps the revocations in the file --revocations-path names and loads them at start. GET /api/admin/revocations lists those in force.

Access link

DATUMHUE_ACCESS_KEY is the deployment's access key, which every aggregator of the deployment needs: one without it does not start. The deployment tool makes it and writes it to the aggregator host's etc/access.env; by hand, make one with head -c 32 /dev/urandom | base64 | tr '+/' '-_' | tr -d '='. A key has 22 to 128 letters, digits and - . _ ~.

The key also closes the browser workstation behind a link: a visitor opens the site's URL with ?key=<value> once, for example https://datumhue.example.com/?key=<value>, which sets a cookie for 30 days and drops the key from the address. Without the cookie a visitor sees a closed page. The HTTP API, the relay and the workstation app's sign-in page, oidc-native-callback.html with its script, need no cookie; a workstation still needs the key to be admitted.

Workstations and providers

An aggregator admits a workstation that presents the deployment's access key, and gives it a seat. It admits a provider that presents a capability for one of its namespaces. Every DatumHue program of the same release can reach the aggregator, so the access key, not the program, keeps others out: share it only with the deployment's people, and replace it when someone leaves. Every aggregator of a deployment needs the same key. Each logs its access key id at start, and warns while another aggregator of the deployment holds a different key: a seat taken at one is then refused at the other. The address port, 7842/udp, carries no traffic between workstations and providers and admits anyone.

To replace the key, make a new one, with datumhue-deploy new-access-key or by hand, restart every aggregator with it together, and hand out the new key. A session that took its seat with the old key ends within seconds, though it can still reach the relay for up to an hour.

A provider holds its seat while one of the capabilities it announces with stands, and loses it within about ten seconds after the last is revoked or ends. A provider whose capability file holds no announce capability does not start.

Seats

aggregator lease list shows the seats taken across the deployment: how many workstation and provider seats against the license's counts, the access key id, and a row for each seat with the holder's endpoint id, the seat, when an aggregator last heard from it and which one. aggregator lease kick <ENDPOINT_ID>... ends workstation sessions, and lease kick --all ends every one. A kicked workstation shows that an administrator ended its session, and that session takes no seat again. Providers keep their seats: a provider's seat ends when its capabilities are revoked. A kick ends a session, not someone's access: starting DatumHue again takes a new seat for whoever holds the access key, so replace the key to keep a person out.

Both commands reach the running aggregator through the admin socket serve --admin-socket names. Run them on the aggregator's host as the account the aggregator runs as: the socket answers no other. The unit the deployment tool writes keeps the socket at /run/datumhue-aggregator/admin.sock:

sudo -u datumhue /opt/datumhue/aggregator lease list \
    --admin-socket /run/datumhue-aggregator/admin.sock

The tool's compose stack keeps it in the aggregator's state folder, and docker compose exec takes the path from the service:

docker compose exec aggregator /opt/datumhue/aggregator lease list

Each aggregator of a deployment answers for the whole deployment, so one is enough.

Network, state and dependencies

Network surface

namespace_count answers over plain HTTPS: poll it after a change until the providers you expect announce. sweep_stats reads {"last_sweep": null} until the first sweep.

A request body must arrive within 5 seconds and fit in 64 KiB, or the aggregator answers 408 Request Timeout or 413 Payload Too Large. A request then has 30 seconds to be answered. Past 512 requests in flight, or 2048 bodies arriving at once, the aggregator answers 503 Service Unavailable.

Port 443 holds at most --connections-per-address connections from one address, and at most 16384 in all, or fewer when the open-file limit leaves less room; the aggregator raises its open-file limit to 65536 at start where the system allows. Through the relay or directly, the aggregator also takes at most 16384 connections at once from workstations, providers and the other aggregators, and turns a further one away as busy.

Every answer of the browser workstation and the HTTP API tells browsers never to show it inside a frame, to run only script files this host serves, to take its content type as declared, and to reach this host name only over HTTPS for a year.

Persistent state

The aggregator writes to all of these but the deployment secret, which it only reads. It renews the license in place, so keep the license in a folder the aggregator can write.

External dependencies

• DatumHue's renewal service, over outbound HTTPS, unless the license is airgapped.
• The other aggregators, at their base-url, with several aggregators.

Running as a service

systemd

The deployment tool writes the aggregator's unit and its environment file. By hand, install the program as /opt/datumhue/aggregator and the browser workstation as /opt/datumhue/static, and give a datumhue user the state folder the quick start made:

sudo useradd --system --home-dir /var/lib/datumhue \
    --shell /usr/sbin/nologin datumhue
sudo chown -R datumhue:datumhue /var/lib/datumhue/aggregator

/etc/systemd/system/datumhue-aggregator.service:

[Unit]
Description=DatumHue Aggregator
After=network.target

[Service]
Type=simple
User=datumhue
Group=datumhue
UMask=0077
WorkingDirectory=/var/lib/datumhue/aggregator
EnvironmentFile=/etc/datumhue/aggregator.env
EnvironmentFile=/etc/datumhue/access.env
ExecStart=/opt/datumhue/aggregator serve
AmbientCapabilities=CAP_NET_BIND_SERVICE
LimitNOFILE=65536
RuntimeDirectory=datumhue-aggregator
RuntimeDirectoryMode=0700
Restart=always
RestartSec=10
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/datumhue/aggregator

[Install]
WantedBy=multi-user.target

/etc/datumhue/aggregator.env:

DATUMHUE_HTTPS_BIND_ADDR=0.0.0.0:443
DATUMHUE_QUIC_BIND_ADDR=0.0.0.0:7842
DATUMHUE_RELAY_URL=https://aggregator.example.com
DATUMHUE_DISCOVERY_URL=https://aggregator.example.com
DATUMHUE_LICENSE_PATH=/var/lib/datumhue/aggregator/customer.dhlicense
DATUMHUE_DEPLOYMENT_SECRET_PATH=/var/lib/datumhue/aggregator/deployment.secret
DATUMHUE_STATIC_DIR=/opt/datumhue/static
DATUMHUE_AGGREGATOR_KEY_PATH=/var/lib/datumhue/aggregator/aggregator.key
DATUMHUE_REPLICATION_STORE_DIR=/var/lib/datumhue/aggregator/replication
DATUMHUE_REVOCATIONS_PATH=/var/lib/datumhue/aggregator/revocations
DATUMHUE_TLS_DIR=/var/lib/datumhue/aggregator/tls
DATUMHUE_ADMIN_SOCKET=/run/datumhue-aggregator/admin.sock
RUST_LOG=info

/etc/datumhue/access.env, mode 600, with the deployment's access key:

DATUMHUE_ACCESS_KEY=<the access key>

sudo systemctl daemon-reload
sudo systemctl enable --now datumhue-aggregator
journalctl -u datumhue-aggregator -f

Every file the aggregator keeps, the license included, lives in /var/lib/datumhue/aggregator, the one folder the unit lets it write besides the admin socket's: RuntimeDirectory= makes /run/datumhue-aggregator at each start, mode 700 and owned by datumhue. AmbientCapabilities= lets the datumhue user listen on port 443.

Docker

FROM debian:bookworm-slim
RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*
COPY aggregator-linux-x86_64 /opt/datumhue/aggregator
COPY static /opt/datumhue/static
ENTRYPOINT ["/opt/datumhue/aggregator"]
CMD ["serve", "--static-dir", "/opt/datumhue/static"]

docker build -t datumhue/aggregator .
docker run -d --name datumhue-aggregator --restart always \
  --user "$(id -u datumhue):$(id -g datumhue)" \
  -p 443:443/tcp -p 7842:7842/udp -p 3478:3478/udp -p 7843:7843/udp \
  --env-file /etc/datumhue/aggregator.env \
  --env-file /etc/datumhue/access.env \
  -e DATUMHUE_ADMIN_SOCKET=/var/lib/datumhue/aggregator/run/admin.sock \
  -v /var/lib/datumhue/aggregator:/var/lib/datumhue/aggregator \
  datumhue/aggregator

The same environment file serves the container: the license, the deployment secret, the identity key, the certificates and the stored records all live in /var/lib/datumhue/aggregator, so they survive a new container. To serve your own certificate, mount its files and name them with DATUMHUE_TLS_CERT_PATH and DATUMHUE_TLS_KEY_PATH, and DATUMHUE_TLS_CA_CERT_PATH for a private CA.

The container has no /run/datumhue-aggregator, so -e puts the admin socket in the state folder instead, in a run folder the aggregator makes mode 700 at start. The lease commands reach it inside the container:

docker exec datumhue-aggregator /opt/datumhue/aggregator lease list

Security notes

• The license, the deployment secret and the access key start an aggregator. Anyone who holds all three can run an aggregator of the deployment. Keep them in your secret store.
• The deployment secret is the deployment's root. Anyone who holds it can authorize and revoke providers for the deployment. Keep it in your secret store, and never copy it to another deployment.
• The access key admits the deployment's people as a group. Anyone who holds it takes a workstation seat. Hand it out privately, keep access.env mode 600, and replace the key when someone should no longer get in.
• The admin socket ends sessions. Whoever can run commands as the aggregator's account can list the seats and end workstation sessions through it. It is a local socket, and answers that account alone.
• Private keys must be mode 600. The aggregator refuses to start when a key it reads, ca-key.pem, leaf-key.pem, the identity key, the deployment secret or --tls-key-path, is readable by others, and names the file.
• Rotating the CA. Delete <tls-dir>/ca-cert.pem and ca-key.pem and restart: the aggregator makes a new CA, and every workstation and provider must be given it. Your own certificates follow your own rotation. A browser that trusted the old CA keeps to HTTPS for the aggregator's host name, and shows a certificate error with no way past it until it trusts the new CA.
• Firewall. Open 443/tcp and 7842, 3478 and 7843/udp inbound, or the ports you set. Outbound, aggregators reach each other's base-url, and reach the renewal service unless the license is airgapped.
• One version. Every aggregator, provider and workstation of a deployment runs the same DatumHue version. Update the aggregators in one window, as the Overview describes: one at a time would split them until all are updated.

Troubleshooting

The aggregator does not start. Run with -v and read the log. A bind address in use, a certificate path that is wrong or unreadable, a key readable by others, a regions file that cannot be read or parsed or has no --config-name block, a --region-id the regions file lacks, a missing license, one whose signatures do not verify or one that is not an Enterprise deployment license, and a missing deployment secret or access key each stop the start. A missing --relay-url or --discovery-url is named.

The identity changes at every restart. The key file does not survive restarts, usually because the program starts in a fresh folder or container each time. Point --aggregator-key-path at persistent storage.

Aggregators do not find each other. Check that every aggregator loads the same license and the same deployment secret (each logs the same deployment key), that each base-url is reachable from the others over HTTPS, that their relay UDP ports are open between them, and that all run the same version.

The license is refused or expired. The log names the reason. An expired license waits for a renewal: deliver one, or wait for the renewal service. A license dated more than a minute ahead of the host's clock is refused: correct the host's clock. For any other reason, ask DatumHue for a new license and check that the file arrived whole.

Announcements are refused. The provider's capability is for another key or namespace, or has ended. Compare the key the provider logs with the bundle's pubkeys.txt, and make its capabilities again with datumhue-deploy reissue-capabilities when they ended or the deployment secret changed.

The aggregator stops: no deployment secret. --deployment-secret-path names no file. Copy the deployment's deployment.secret there, from the bundle's state/aggregator/ or your secret store. For a new deployment, aggregator deployment-secret generate <PATH> makes one.

The aggregator stops: no access key. DATUMHUE_ACCESS_KEY is not set. Give it the deployment's key, from the aggregator host's etc/access.env in the bundle. Without a bundle, make one by hand as Access link shows, and give workstations the same key.

Workstations are refused with 401. They need the deployment's access key: DATUMHUE_ACCESS_KEY from datumhue-client.env for the workstation app and the command line, the access link for a browser. After the key was replaced, they need the new one.

lease commands get no answer. "connecting to the admin socket" means the path is not the one serve --admin-socket names, or the aggregator runs without it. "closed the connection unanswered" means the command ran as another account than the aggregator's: run it as that account, with sudo -u datumhue under the unit.

The aggregator stops: the admin socket's folder. The folder must belong to the aggregator's account, and no other account may write in it; make it mode 700. "another aggregator answers on the admin socket" means two aggregators on one host name the same path: give each its own.

Browsers warn about the certificate. A browser trusts only the certificates its system trusts. Serve a certificate from a public CA, or install <tls-dir>/ca-cert.pem on every computer that opens the deployment. The workstation app reads the CA from DATUMHUE_CERT_PATH and needs no install.

Browsers show a blank page. --static-dir must name the folder that holds index.html and its .js and .wasm files, readable by the aggregator's user; check SELinux labels where SELinux runs.