Ingress Provider

What it does

The ingress provider takes events from other systems, webhooks and notifications, and hands them to the apps you allow. Each kind of event has a channel: a system signs each event with the channel's secret and posts it, the provider checks the signature and the event's shape, and sends it on, encrypted, to the apps granted the channel. One provider serves any number of channels. See the Overview for how it fits with the other components.

Quick start

Describe the provider in deployment.kdl: its host, its listen address, and each channel other systems send events to, with the rules for who may read it.

ingress-provider {
    host "providers-1"
    listen "0.0.0.0:8080"
    ingress-channel "webhook" {
        root-key-file "/var/lib/datumhue/ingress-provider/webhook.rootkey"
        hmac-secret-file "/var/lib/datumhue/ingress-provider/webhook.hmac"
        payload-schema-file "/var/lib/datumhue/ingress-provider/webhook.json"
        issuer "https://sign-in.example.com/realms/acme" {
            allow claim="groups" array_contains="ops" rights="r"
        }
    }
}

Make its files with the deployment tool, generate or add-provider, then put each channel's files in the bundle's providers-1/state/ingress-provider/, mode 600: a root key of 64 hex characters (head -c 32 /dev/urandom | xxd -p -c 32), the sender's signing secret and the JSON Schema of its events.

On the host, install the program, then follow the README.md in the host's folder of the bundle, which installs the rest and starts it:

sudo install -D -m 755 ingress-provider-linux-x86_64 \
    /opt/datumhue/ingress-provider

Its log names its public key, which matches the line for it in the bundle's pubkeys.txt, and then each announcement.

How it works

Posting an event. A system sends each event as JSON, signed with HMAC-SHA256 over the exact body:

POST /<channel>
X-Hub-Signature-256: sha256=<hex HMAC-SHA256 of the body>
Content-Type: application/json

<the event>

The provider listens on plain HTTP at listen, behind your load balancer, which ends TLS. The signature protects each event the whole way. The provider keeps no events, so several of them behind one load balancer can each take any event.

Reading a channel. The provider hands the keys that open a channel's events only to people its rules admit. The keys change every rotation-period.

Apps get channels by name. An app reads a channel only when it is granted it by name: --ingress-mount NAME:CHANNEL when the workstation starts it, ingress-mount=NAME:CHANNEL in a link to the browser workstation, or from a parent app. No grant covers every channel. datumhue ingress list shows the channels a deployment has.

Command line

ingress-provider <COMMAND>

ingress-provider serve

ingress-provider serve [OPTIONS]

Configuration

The settings file is KDL: one ingress-provider block, then one ingress-channel block for each channel.

ingress-provider {
    secret-key-file "/var/lib/datumhue/ingress-provider/provider.key"
    capability-file "/var/lib/datumhue/ingress-provider/provider.caps"
    aggregator-url "https://aggregator.example.com"
    relay-url "https://aggregator.example.com"
    listen "0.0.0.0:8080"
}

ingress-channel "github" {
    description "GitHub webhooks"
    root-key-file "/etc/datumhue/github.rootkey"
    hmac-secret-file "/etc/datumhue/github.hmac"
    payload-schema-file "/etc/datumhue/github.schema.json"
    idempotency-header "X-GitHub-Delivery"
    issuer "https://sign-in.example.com/realms/acme" {
        allow claim="groups" array_contains="ops" rights="r"
    }
}

Common provider settings

Every provider's settings block takes these keys beside its own. A relative path anywhere in the settings file resolves against the folder that holds the file, not the folder the provider starts in.

The key and the capability file. The secret key is the provider's identity: the provider creates it at its first start when the file is missing, and logs its public key at every start. The capability file holds the capabilities that let the provider announce what it serves. The provider reads it again before every announcement and rewrites it as capabilities renew, so it refuses to start when it cannot write to the file's folder. See Provider authorization.

The access log. At info, on the log target datumhue_audit, the provider logs each access it allows or refuses, each sign-in it cannot verify, and each session it ends because a new sign-in no longer passes the rules. A mount or channel's open is logged, and the operations it then allows are not, one by one. A refusal that repeats is logged with a count, in repeated, not each time. RUST_LOG=info keeps the access log; RUST_LOG=info,datumhue_audit=off turns it off alone, and a RUST_LOG of warn or error, or one that names only other targets, leaves it out.

Each line has the message access and the fields provider, action, resource, endpoint, iss, sub, outcome, rights, basis, claim, reason, credential and repeated, with - where a field has nothing to say. It names the mount, channel or scope, the connecting program's endpoint id, the signed-in person, the outcome (allow, deny, refused, verify_failed or session_dropped), for a refusal the claim that failed, and in repeated how often a refusal came. It never holds a token, a secret or the value a rule expects.

Ingress provider settings

Channels

Access rules

Access rules say who may use a resource. A rule that reads a sign-in sits in the block of the sign-in service whose tokens it reads, named by its issuer URL exactly as those tokens carry it in iss. datumhue login prints that URL. Each rule names who it admits and the rights it grants:

issuer "https://sign-in.example.com/realms/acme" {
    require claim="email_verified" equals=#true
    allow claim="groups" array_contains="analysts" rights="r"
}

• rights is "r" (read) or "rw" (read and write). A rule admits a request when it matches and its rights cover what the request asks for. Rules add up: any one rule that admits a request is enough.
• require takes the same claim and match, without rights. Every require of a block must hold before any of its rules counts.
• A sign-in counts only in the block of the issuer that signed it. The URL is compared exactly: https://accounts.google.com and https://accounts.google.com/ are two issuers. A sign-in from an issuer with no block gets nothing a block grants.
• equals is typed. equals="true" matches the text true, never the boolean; a number must be quoted and is compared as text.
• claim names a top-level claim of the person's ID token. A value nested in another claim, such as Keycloak's realm_access.roles, never matches: map it to a top-level claim at your sign-in service.
• Prefer the groups or roles your sign-in service manages, or sub, over email. On Google, add require claim="hd" equals="<your domain>" to admit only your Workspace accounts.
• The issuer must be one the deployment's Identity Provider serves.
• An email claim counts only when the same token says email_verified is true, and a phone_number claim only when it says phone_number_verified is true. Microsoft Entra ID does not send email_verified: match oid, sub or groups there.
• A sign-in that does not verify is refused, even where allow anyone admits a request without one.
• A refused request learns which claim failed, never the value a rule expects.

For a channel, any rule that matches lets the person's apps read it. A channel only delivers, so its rules grant rights="r"; a rule with rights="rw" stops the start.

Every channel needs a rule: a channel without one stops the provider at start. allow anyone rights="r", outside every issuer block, lets anyone who reaches the provider, directly or through a relay of the deployment, read the channel with no sign-in; the provider logs a warning for it at start. Sending to a channel takes its secret, never a rule.

Network, state and dependencies

Network surface

The provider takes 1024 workstation connections at most.

Persistent state

The provider keeps no events: it sends each one on as it arrives.

External dependencies

• Your load balancer, which ends TLS for incoming events and sends them to listen.
• The aggregator, for announcing and for the sign-in services' keys. Within a minute of losing it, the provider refuses sign-ins until it can read the keys again.
• The Identity Provider, when a channel has an issuer block.

Running as a service

systemd

The deployment tool writes a unit like this one for every provider it sets up. To run the provider under systemd by hand, install the program as /opt/datumhue/ingress-provider, its settings as /etc/datumhue/ingress-provider.kdl, and give the datumhue user the state folder /var/lib/datumhue/ingress-provider/.

/etc/systemd/system/datumhue-ingress-provider.service:

[Unit]
Description=DatumHue Ingress Provider
After=network.target

[Service]
Type=simple
User=datumhue
Group=datumhue
WorkingDirectory=/var/lib/datumhue/ingress-provider
ExecStart=/opt/datumhue/ingress-provider serve \
    --config /etc/datumhue/ingress-provider.kdl
Environment=DATUMHUE_CERT_PATH=/etc/datumhue/ca-cert.pem
EnvironmentFile=/etc/datumhue/keystore.env
Restart=always
RestartSec=10
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/datumhue/ingress-provider

[Install]
WantedBy=multi-user.target

• The unit keeps /etc/datumhue read-only, and a relative path in the settings file resolves there. Point secret-key-file, capability-file and every other file the provider writes into /var/lib/datumhue/ingress-provider/.
• keystore.env holds DATUMHUE_KEYSTORE_PASSPHRASE=...; keep it mode 600. Drop the EnvironmentFile= line for a plaintext key.
• Drop the DATUMHUE_CERT_PATH line when the aggregator's certificate comes from a public CA.
• On the aggregator's host, add datumhue-aggregator.service to After=.

sudo systemctl daemon-reload
sudo systemctl enable --now datumhue-ingress-provider
journalctl -u datumhue-ingress-provider -f

systemctl stop stops the provider the way Ctrl+C does: it stops announcing and gives its provider seat back, so the next start takes a seat at once.

Docker

To run the provider in a container, build an image from this Dockerfile, in the folder that holds the program:

FROM debian:bookworm-slim
RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*
COPY ingress-provider-linux-x86_64 /opt/datumhue/ingress-provider
ENTRYPOINT ["/opt/datumhue/ingress-provider"]
CMD ["serve", "--config", "/etc/datumhue/ingress-provider.kdl"]

Run it as the datumhue user that owns the state folder, with the settings folder read-only:

docker build -t datumhue/ingress-provider .
state=/var/lib/datumhue/ingress-provider
docker run -d --name datumhue-ingress-provider --restart always \
    --user "$(id -u datumhue):$(id -g datumhue)" \
    --env-file /etc/datumhue/keystore.env \
    -e DATUMHUE_CERT_PATH=/etc/datumhue/ca-cert.pem \
    -v /etc/datumhue:/etc/datumhue:ro -v "$state:$state" \
    datumhue/ingress-provider

The docker-compose.yml the deployment tool writes runs images by these names, with the same folders.

Publish the listen port to the load balancer: add -p 8080:8080 to docker run.

Security notes

• The secret decides who may send. Only a system that holds a channel's secret can post to it. Give each channel its own, and change it by writing a new hmac-secret-file and restarting.
• The rules decide who may read. A channel without rules does not load. allow anyone opens a channel to anyone who reaches the provider, directly or through a relay of the deployment, and the provider logs a warning for it at start; give each channel the narrowest rule that works.
• Reading ends within two rotation-periods. Each fetch of a channel's keys passes the rules with the person's current sign-in. Someone the rules stop admitting can still read the events of at most two rotation-periods after the last fetch the rules admitted.
• Sign-in key changes apply within a minute. A change to a sign-in service's keys, a revocation included, reaches the provider within a minute.
• Every event matches its schema. An event that does not is refused and never reaches an app.
• Run behind TLS. The provider takes plain HTTP: always put a load balancer that ends TLS in front of it.
• Bodies are never logged. The access log records each channel open, allowed or refused, and, on the log target datumhue_audit::push, each event whose signature matches, with its channel, status and size, and each event with a missing or wrong signature, as refused. RUST_LOG=info,datumhue_audit::push=off turns the event lines off alone.
• Settings apply at start. New channels, schemas, rules and secrets take effect when the provider restarts.
• Protect the key. The deployment tool encrypts it with your passphrase; a key the provider makes at its first start is plaintext unless DATUMHUE_KEYSTORE_PASSPHRASE is set. The provider refuses a key file that others can read.

Troubleshooting

Messages. The provider writes its log to standard output; under systemd, read it with journalctl -u and the unit's name.

The aggregator is refused as untrusted. It serves the certificate it made itself: point DATUMHUE_CERT_PATH at its tls/ca-cert.pem.

provider lease acquisition failed. With QuotaExhausted, every provider seat of the license is taken: stop a provider, or extend the license.

The seat or announcements are refused. capability-file holds no current capability for this key: make one with datumhue-deploy reissue-capabilities.

The provider stops at start, naming relay-url. Set relay-url to the aggregator's public HTTPS URL.

The provider refuses its key file. A key file that others can read is refused: chmod 600 it. DATUMHUE_KEYSTORE_PASSPHRASE must be set for an encrypted key and unset for a plaintext one; the message says which the key is.

The provider cannot write its capability file. Give the provider's user write access to the folder that holds the file; it rewrites the file as capabilities renew.

The provider stops at start, naming a channel's access rules. A channel needs at least one rule, a rule that reads a sign-in sits in an issuer block, and its rules grant rights="r" only. The message says what to write; datumhue login prints the issuer URL a block names.

Apps see no channel. The log shows announcer running once the provider announces. Without it, the capability file lacks a current capability for ingress/<channel>.

Events are refused with 401. The signature is missing, is in another header than signature-header, or does not match. The sender must sign the exact bytes of the body with the secret in hmac-secret-file.

Events are refused with 422. The body is JSON that does not match payload-schema-file.

Events are refused with 400. The body is not JSON, or lacks the channel's idempotency-header.

Events are refused with 404. No channel has that name, or the path is more than one name: post to /<channel>.

An app receives no events. Grant the app the channel by name, with --ingress-mount or a link's ingress-mount=, or from a parent app. The person must also pass the channel's rules: without them, no key and no events.