Egress Provider

What it does

The egress provider lets apps call the web services you name, and no others. Each web service is a mount, pinned to one origin: an app names the mount and a path inside it, never an address, so it reaches only the origins you list. One provider serves any number of mounts. See the Overview for how it fits with the other components.

Quick start

Describe the provider in the deployment description, deployment.kdl: the host it runs on and each web service apps may call.

egress-provider {
    host "providers-1"
    http-mount "weather" {
        origin "https://api.weather.example.com/v1"
        description "Public weather API"
        issuer "https://sign-in.example.com/realms/acme" {
            allow claim="groups" array_contains="staff" rights="r"
        }
    }
}

An app names a mount and a path inside it; it never reaches an address you did not name. rights="r" lets apps send GET and HEAD, and rights="rw" every method.

Make its files with the deployment tool, generate or add-provider.

On the host, install the program, then follow the README.md in the host's folder of the bundle, which installs the rest and starts it:

sudo install -D -m 755 egress-provider-linux-x86_64 \
    /opt/datumhue/egress-provider

Its log names its public key, which matches the line for it in the bundle's pubkeys.txt, and then each announcement.

How it works

Mounts. Each http-mount pins one origin, such as https://api.example.com/v1, names who may use it, and may add a credential to send. The provider announces each mount to the aggregator, with a capability for egress/<mount>.

Requests. An app sends a method, a path inside the mount, headers, a query and a body; the provider sends them on to the origin and returns the answer. The provider sets the host and the framing itself, and refuses a request whose headers would change them, claim a client address, such as X-Forwarded-For, or override the method or path.

Paths stay in the mount. Request paths go below the origin's path: the provider refuses a full URL, another host, and a . or .. segment however it is encoded. It follows a redirect only while it stays within the mount's origin and path.

Apps get mounts by name. An app reaches a mount only when it is granted it by name: --http-mount NAME:MOUNT when the workstation starts it, http-mount=NAME:MOUNT in a link to the browser workstation, or from a parent app. No grant covers every mount.

Credentials belong to the mount. On any mount but a "none" one, the provider drops an Authorization header the app set, so the app never chooses what goes to the origin.

A mount that sends a credential must use an https origin; one with an http origin stops the start.

Command line

egress-provider <COMMAND>

egress-provider serve

egress-provider serve [OPTIONS]

Configuration

The settings file is KDL: one http-proxy block, then one http-mount block for each web service.

http-proxy {
    secret-key-file "/var/lib/datumhue/egress-provider/provider.key"
    capability-file "/var/lib/datumhue/egress-provider/provider.caps"
    aggregator-url "https://aggregator.example.com"
    relay-url "https://aggregator.example.com"
}

http-mount "weather" {
    origin "https://api.weather.example.com/v1"
    description "Public weather API"
    allow anyone rights="r"
}

http-mount "billing" {
    origin "https://billing.internal.example.com/api"
    credential "operator"
    credential-secret-file "/etc/datumhue/billing.secret"
    issuer "https://sign-in.example.com/realms/acme" {
        allow claim="groups" array_contains="finance" rights="rw"
    }
}

Common provider settings

Every provider's settings block takes these keys beside its own. A relative path anywhere in the settings file resolves against the folder that holds the file, not the folder the provider starts in.

The key and the capability file. The secret key is the provider's identity: the provider creates it at its first start when the file is missing, and logs its public key at every start. The capability file holds the capabilities that let the provider announce what it serves. The provider reads it again before every announcement and rewrites it as capabilities renew, so it refuses to start when it cannot write to the file's folder. See Provider authorization.

The access log. At info, on the log target datumhue_audit, the provider logs each access it allows or refuses, each sign-in it cannot verify, and each session it ends because a new sign-in no longer passes the rules. A mount or channel's open is logged, and the operations it then allows are not, one by one. A refusal that repeats is logged with a count, in repeated, not each time. RUST_LOG=info keeps the access log; RUST_LOG=info,datumhue_audit=off turns it off alone, and a RUST_LOG of warn or error, or one that names only other targets, leaves it out.

Each line has the message access and the fields provider, action, resource, endpoint, iss, sub, outcome, rights, basis, claim, reason, credential and repeated, with - where a field has nothing to say. It names the mount, channel or scope, the connecting program's endpoint id, the signed-in person, the outcome (allow, deny, refused, verify_failed or session_dropped), for a refusal the claim that failed, and in repeated how often a refusal came. It never holds a token, a secret or the value a rule expects.

Egress provider settings

Mounts

Access rules

Access rules say who may use a resource. A rule that reads a sign-in sits in the block of the sign-in service whose tokens it reads, named by its issuer URL exactly as those tokens carry it in iss. datumhue login prints that URL. Each rule names who it admits and the rights it grants:

issuer "https://sign-in.example.com/realms/acme" {
    require claim="email_verified" equals=#true
    allow claim="groups" array_contains="analysts" rights="r"
}

• rights is "r" (read) or "rw" (read and write). A rule admits a request when it matches and its rights cover what the request asks for. Rules add up: any one rule that admits a request is enough.
• require takes the same claim and match, without rights. Every require of a block must hold before any of its rules counts.
• A sign-in counts only in the block of the issuer that signed it. The URL is compared exactly: https://accounts.google.com and https://accounts.google.com/ are two issuers. A sign-in from an issuer with no block gets nothing a block grants.
• equals is typed. equals="true" matches the text true, never the boolean; a number must be quoted and is compared as text.
• claim names a top-level claim of the person's ID token. A value nested in another claim, such as Keycloak's realm_access.roles, never matches: map it to a top-level claim at your sign-in service.
• Prefer the groups or roles your sign-in service manages, or sub, over email. On Google, add require claim="hd" equals="<your domain>" to admit only your Workspace accounts.
• The issuer must be one the deployment's Identity Provider serves.
• An email claim counts only when the same token says email_verified is true, and a phone_number claim only when it says phone_number_verified is true. Microsoft Entra ID does not send email_verified: match oid, sub or groups there.
• A sign-in that does not verify is refused, even where allow anyone admits a request without one.
• A refused request learns which claim failed, never the value a rule expects.

For a mount, any rule that matches lets the person use it. rights="r" lets them send GET and HEAD; rights="rw" lets them send POST, PUT, PATCH and DELETE too. A service that reads through POST needs rw.

Every mount needs a rule: a mount without one stops the provider at start. allow anyone rights="r", outside every issuer block, lets anyone who reaches the provider, directly or through a relay of the deployment, send requests through the mount with no sign-in; on an "operator" mount those requests carry your secret. The provider logs a warning for it at start. A "forward-user-token" mount forwards each person's own token, so it takes no allow anyone. To admit everyone a sign-in service signs in, put allow signed-in in its block.

Network, state and dependencies

Network surface

The provider listens on no TCP port; workstations reach it through the aggregator. It takes 256 connections at most. Requests and answers may each be up to 1 GiB. A request times out after 30 seconds, or after the time the app sets, 300 seconds at most.

Addresses the provider refuses. The provider never connects to its own host or to a cloud metadata service: loopback (127.0.0.0/8, ::1), unspecified (0.0.0.0, ::) and link-local (169.254.0.0/16, fe80::/10) addresses, and the metadata services' other addresses. It checks a host name's addresses after resolving it and connects only to those. Private networks stay reachable, so a mount may name a service on your own network; to limit what the provider's host reaches, put it behind your own outbound firewall too.

Through a proxy. With outbound-proxy, the proxy resolves the origins and so decides which addresses requests reach; the provider still refuses an origin written as one of the addresses above, and logs a warning at start. The proxy environment variables, such as HTTPS_PROXY, steer only the provider's own traffic to the aggregator.

Persistent state

External dependencies

• The aggregator, for announcing and for the sign-in services' keys. Within a minute of losing it, the provider refuses sign-ins until it can read the keys again.
• The Identity Provider, when a mount has an issuer block or forwards the person's token.
• The origins, reachable from the provider's host.

Running as a service

systemd

The deployment tool writes a unit like this one for every provider it sets up. To run the provider under systemd by hand, install the program as /opt/datumhue/egress-provider, its settings as /etc/datumhue/egress-provider.kdl, and give the datumhue user the state folder /var/lib/datumhue/egress-provider/.

/etc/systemd/system/datumhue-egress-provider.service:

[Unit]
Description=DatumHue Egress Provider
After=network.target

[Service]
Type=simple
User=datumhue
Group=datumhue
WorkingDirectory=/var/lib/datumhue/egress-provider
ExecStart=/opt/datumhue/egress-provider serve \
    --config /etc/datumhue/egress-provider.kdl
Environment=DATUMHUE_CERT_PATH=/etc/datumhue/ca-cert.pem
EnvironmentFile=/etc/datumhue/keystore.env
Restart=always
RestartSec=10
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/datumhue/egress-provider

[Install]
WantedBy=multi-user.target

• The unit keeps /etc/datumhue read-only, and a relative path in the settings file resolves there. Point secret-key-file, capability-file and every other file the provider writes into /var/lib/datumhue/egress-provider/.
• keystore.env holds DATUMHUE_KEYSTORE_PASSPHRASE=...; keep it mode 600. Drop the EnvironmentFile= line for a plaintext key.
• Drop the DATUMHUE_CERT_PATH line when the aggregator's certificate comes from a public CA.
• On the aggregator's host, add datumhue-aggregator.service to After=.

sudo systemctl daemon-reload
sudo systemctl enable --now datumhue-egress-provider
journalctl -u datumhue-egress-provider -f

systemctl stop stops the provider the way Ctrl+C does: it stops announcing and gives its provider seat back, so the next start takes a seat at once.

The provider trusts the public CAs and those in DATUMHUE_CERT_PATH, for the aggregator and for every origin: put the CA of an internal service there too. One file may hold several CA certificates.

Docker

To run the provider in a container, build an image from this Dockerfile, in the folder that holds the program:

FROM debian:bookworm-slim
RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates \
    && rm -rf /var/lib/apt/lists/*
COPY egress-provider-linux-x86_64 /opt/datumhue/egress-provider
ENTRYPOINT ["/opt/datumhue/egress-provider"]
CMD ["serve", "--config", "/etc/datumhue/egress-provider.kdl"]

Run it as the datumhue user that owns the state folder, with the settings folder read-only:

docker build -t datumhue/egress-provider .
state=/var/lib/datumhue/egress-provider
docker run -d --name datumhue-egress-provider --restart always \
    --user "$(id -u datumhue):$(id -g datumhue)" \
    --env-file /etc/datumhue/keystore.env \
    -e DATUMHUE_CERT_PATH=/etc/datumhue/ca-cert.pem \
    -v /etc/datumhue:/etc/datumhue:ro -v "$state:$state" \
    datumhue/egress-provider

The docker-compose.yml the deployment tool writes runs images by these names, with the same folders.

Security notes

• The origin is the boundary. Apps reach only the origins your mounts name. Give each mount the narrowest origin path that works.
• A forwarded token works at the origin until it expires, within what the sign-in service allows it. Forward tokens only to services that trust your deployment's sign-in service; send others an "operator" secret or nothing. A mount that forwards nothing never receives the person's access token.
• Only the signed-in person's access token is forwarded, as far as your sign-in service lets the provider check it: the ID token's at_hash names it, or your service signed it for the person, by sub or by the claim the identity provider's person-claim names. With ID tokens without at_hash, someone the mount admits could forward an opaque access token they hold, or a signed one your sign-in service's keys do not verify.
• Access ends with the sign-in. A mount opened through an issuer rule, or with a forwarded token, refuses requests once the person's current ID token expires, and each refreshed sign-in must pass the rules again, so access you withdraw at your sign-in service ends by the time that token expires.
• Sign-in key changes apply within a minute. A change to a sign-in service's keys, a revocation included, reaches the provider within a minute.
• Read rights are GET and HEAD. A person whose rules give r cannot send a method that changes the origin, whatever the app asks.
• An operator secret stays on the provider's host. Keep its file readable only by the provider's user.
• allow anyone lends the mount's credential. Anyone who reaches the provider, directly or through a relay of the deployment, sends requests with an "operator" mount's secret and no sign-in, and the provider logs a warning for each such mount at start. Give allow anyone only to mounts whose origin is public.
• Bodies, paths, query strings and credentials are never logged. The access log records each mount open, allowed or refused, and each request refused because the open granted reading only; a request that fails upstream is logged with its mount and method.
• Settings apply at start. New mounts, rules and credentials take effect when the provider restarts.
• Protect the key. The deployment tool encrypts it with your passphrase; a key the provider makes at its first start is plaintext unless DATUMHUE_KEYSTORE_PASSPHRASE is set. The provider refuses a key file that others can read.

Troubleshooting

Messages. The provider writes its log to standard output; under systemd, read it with journalctl -u and the unit's name.

The aggregator is refused as untrusted. It serves the certificate it made itself: point DATUMHUE_CERT_PATH at its tls/ca-cert.pem.

provider lease acquisition failed. With QuotaExhausted, every provider seat of the license is taken: stop a provider, or extend the license.

The seat or announcements are refused. capability-file holds no current capability for this key: make one with datumhue-deploy reissue-capabilities.

The provider stops at start, naming relay-url. Set relay-url to the aggregator's public HTTPS URL.

The provider refuses its key file. A key file that others can read is refused: chmod 600 it. DATUMHUE_KEYSTORE_PASSPHRASE must be set for an encrypted key and unset for a plaintext one; the message says which the key is.

The provider cannot write its capability file. Give the provider's user write access to the folder that holds the file; it rewrites the file as capabilities renew.

Apps see no mount. The log shows announcer running once the provider announces. Without it, the capability file lacks a current capability for egress/<mount>.

An app cannot reach a mount it expects. Grant the app the mount by name, with --http-mount or a link's http-mount=, or from a parent app.

The provider stops at start, naming a mount's access rules. A mount needs at least one rule, a rule that reads a sign-in sits in an issuer block, and a "forward-user-token" mount takes no allow anyone. The message says what to write; datumhue login prints the issuer URL a block names.

A request is refused before it connects. The origin resolved to one of the refused addresses, the path left the mount, or a header named the host, the framing or a forward. With rights="r", a method other than GET and HEAD is refused as access denied.

A redirect is refused. The origin redirected outside the mount's origin or path. When the service moves requests between paths, give the mount the path above both.

Requests ignore a proxy set in the environment. Requests to the origins go directly unless outbound-proxy names the proxy.

The origin refuses a forwarded token. The token comes from your deployment's sign-in service, which the origin must trust. For any other service, use an "operator" secret.

Opening a forwarding mount asks for a sign-in while the person is signed in. The provider did not take the access token as theirs, and the access log says why: access_token_hash_mismatch when the ID token's at_hash names another access token, access_token_names_another when the access token names another person. Set the sign-in service to issue a new ID token with each access token, and either to name people by the same sub in both or to put at_hash in its ID tokens. Microsoft Entra ID does neither for an API registered as an application of its own: set person-claim "oid" on its entry in the identity provider, with profile among its scopes. For a service that can do none of this, use an "operator" secret.

Requests to an origin time out. The provider's host has no route to it: check that host's firewall, DNS and connection to the origin.