swarmexec › Docs › The agents: init, down, doctor
The agents: init, down, doctor_
Provisioning, removing, diagnosing and monitoring the per-node agents.
On this page
init — provision the agents
Deploys the agent as a global Swarm service via the manager API: creates a shared-secret Docker secret, runs the agent in self-signed mode on every node (host port 9443), and writes the matching client config. Run it once per swarm.
| Flag | Default | Description |
|---|---|---|
--image | docker.io/logleio/swarmexec-agent:latest | agent image to deploy |
--secret | random | shared secret to use (default: generate a random one). Note: if the agent secret already exists it is kept as-is (Docker secrets are immutable), so --secret is not applied to the cluster — it is only written to your client config, and init warns about the mismatch risk. To change the cluster secret, remove it first (no service may reference it), then re-run init |
--service-name | swarmexec_agent | name for the agent service |
--port | 9443 | host port the agent publishes |
--force | false | update the service if it already exists. A prune schedule set on the agents is kept |
--allow-legacy-secret | true | accept the raw shared secret from clients predating connection-bound auth; pass =false once every client is upgraded |
--metrics-port | 0 | publish the agent's Prometheus metrics on this host port (0 = off) — see monitoring the agents |
--policy-file | — | authorization rule file for the agents, checked before anything is deployed — see authorization policy |
--save-config | true | write the client config |
--registry-auth | true | pass local registry credentials so nodes can pull a private image |
--wait | true | wait for the agents to come up and report progress |
--rollout-timeout | 90s | how long to wait for agents to start |
--force to roll out a new agent version — see
re-rolling the agents without changing the secret.
down — remove the agents
The inverse of init: removes the global agent service and, by
default, the shared-secret Docker secret. It does not touch your
client config.
| Flag | Default | Description |
|---|---|---|
--service-name | swarmexec_agent | name of the agent service to remove |
--keep-secret | false | do not remove the shared-secret Docker secret |
-y, --yes | false | do not prompt for confirmation |
doctor — diagnose the swarm
Checks the manager connection, whether the agent service is deployed, and probes
every ready node's agent for reachability and version skew. Prints a per-node
table (NODE AGENT VERSION PROTO); exits
non-zero if the manager is unreachable or any agent is unhealthy. A node reported
as too old (init --force) runs an agent older than your client —
see re-rolling the agents without changing the secret.
A node that does not answer is unreachable, and an
ERROR column (error in --json) says why: a
rejected secret, a TLS failure and a refused connection read differently there.
doctor also runs without a complete agent configuration, as on a
machine that has not run init: it reports the manager and whether the
agent is deployed, adds a config: line (config_error) naming
what is missing, skips the per-node probe and exits non-zero.
When a newer swarmexec release exists, an update: line under the
client line says so (latest_release in --json) — information
only, it does not change the exit code; see
ui.check_updates.
| Flag | Default | Description |
|---|---|---|
--connect-timeout | 10s | per-node connect timeout |
--json | false | output JSON instead of a table |
status — the cluster at a glance
Every node with its role, state, availability and engine; every service with its
stack, mode, replicas, the health of its containers and a rolling update in
flight; and a summary line counting the degraded services —
running short, or with an unhealthy container. Replicas come from the manager,
health from the node agents, best effort: without them the services are listed
without health and a note on stderr says so, rather than showing all healthy.
The same report as the MCP tool cluster_overview.
stats — live CPU and memory per container
One reading of every running container of the named services — all services when
none is named: CPU, memory against its limit, and health. The node agents measure;
the manager only knows what was reserved. An agent samples only while
someone asks, so a first reading takes a few seconds: stats waits up
to --wait (12 s) for CPU, which needs two samples, then prints
what it has — warming up for a container still without a reading,
unavailable where the node's agent did not answer, each explained once
on stderr. The same rows as the MCP tool stats.
| Flag | Default | Description |
|---|---|---|
--json | false | output JSON instead of the table (status and stats) |
--wait | 12s | stats: how long to wait for agents still warming up; 0 asks once |
--connect-timeout | 10s | per-node connect timeout |
Monitoring the agents
Every agent can serve Prometheus metrics. They are off by default;
swarmexec init --force --metrics-port 9464 turns them on: the agent
listens on that port and publishes it in host mode on every node, like the agent
port, so Prometheus reaches each node's agent at <node>:9464. The
endpoint is plain HTTP and carries counts only — no names, ids or
payload — but publish it only where your scraper can reach it.
Strict configuration. A set-but-invalid SWARMEXEC_*
environment value (SWARMEXEC_ALLOW_LEGACY_SECRET=flase, a duration without
a unit, a non-numeric port) stops the agent at startup with an error naming the
variable, as an invalid flag does; it no longer falls back to the default silently.
A shared secret under 16 characters is accepted but logged as a warning at startup
(its length, never its value). When the agent cannot inspect a container it refuses
the request after authorizing and auditing it, because a policy rule scoped to
services cannot be applied to a container whose service is unknown.
| Metric | Meaning |
|---|---|
swarmexec_agent_info{version,protocol} | constant 1; compare it across nodes to see version skew |
swarmexec_active_sessions | exec sessions open right now |
swarmexec_sessions_total | exec sessions started |
swarmexec_auth_denials_total | requests the authorizer refused — the one signal that makes an attempt visible |
swarmexec_auth_failures_total{reason} | failed shared-secret authentications (missing, invalid, legacy_refused) — someone guessing the secret; the agent's own loopback health probe is not counted |
swarmexec_limit_refusals_total{limit} | requests refused at a node cap (streams, forward_sidecars) |
swarmexec_active_streams{method} | streams open right now, per RPC (Exec, Logs, PortForward, WatchContainerEvents) |
swarmexec_streams_total{method} | streams admitted past the drain and the stream cap, per RPC — including ones the authorizer then refuses |
swarmexec_bytes_in_total / _out_total | bytes from clients (exec stdin, forwards) / to clients (exec output, logs, forwards) |
The repository ships what goes around it, in agent/deploy/monitoring/:
a scrape config (Prometheus' own Docker Swarm discovery, or static targets), four
alert rules — authorization denials rising, agents on different versions for more
than 30 minutes, cap refusals, an agent not answering — and a Grafana dashboard
(grafana-dashboard.json, import and pick your Prometheus).
Health check
Independently of metrics, every agent has a Docker health check, and Swarm acts on
it: a task that turns unhealthy is replaced. It ships in the agent image,
so the TUI's image update turns it on — no init --force needed. The
probe is the agent binary itself (/agent -healthcheck; the image has no
shell): it dials the agent's own port on loopback, calls Version —
without credentials; the agent refusing them is an answer too — and pings Docker
over the agent's socket, which catches the one failure a restart fixes: a daemon
restarted with live-restore leaves the agent holding a dead socket.
It runs every 10 s with a 10 s timeout, after a 30 s start period, and
needs 6 failures in a row — a minute or two of a dead or wedged agent — before Swarm
replaces it, so a brief daemon restart does not trip it. A rolling agent update now
waits for each node's new agent to pass its first check, about 10 s per node. On a node,
docker inspect --format '{{json .State.Health}}' <agent container>
shows the last results with the probe's reason.
Re-rolling the agents without changing the secret
Sooner or later you have to redeploy the agents on a swarm that is already
provisioned: an agent hangs, doctor shows a node as
too old (init --force), a command fails with “agent is older
than this client (missing RPC) — update it with swarmexec init
--force”, or a node was rebuilt and its agent never came back. The
fix is to re-run init with --force, which tells it to
update the service if it already exists. Without the flag, running
init against an existing service is a usage error telling you to add
--force — nothing is changed.
A release that changes the agent's service spec needs
init --force too, and an image update from the TUI is not
enough for it. v1.30.0 is such a release: the agent gained a read-only
/sys bind mount so it can report the node's own link MTUs, and a
mount lives in the spec, not in the image. Until the re-roll the agents run
fine and simply report no links, so the Networks tab says the underlay's MTU is
unknown instead of checking an overlay's framing.
The “missing RPC” message is what a feature newer than the agent reports:
the container menu's inspect
container (node-local) and kill task, for instance, need an agent
from the release that introduced them (RPCs InspectContainer and
KillContainer). Everything else keeps working against the old agent.
This redeploys the agent on every node and leaves the shared secret
alone. Docker secrets are immutable, so init finds the
existing swarmexec_agent_secret, reports it as
reusing existing, and keeps the value that is already in the
cluster. Every client that worked before the re-roll keeps working — there is
nothing to redistribute.
--secret on a re-roll. If the secret
already exists it cannot be overwritten, so your value is not applied to
the cluster — but it is still written into your client
config. Unless it happens to be the existing secret's real value, the agents will
reject you from that moment on, and the failure surfaces later as what looks like
a broken agent rather than a wrong local config. init warns about
exactly this when it happens; take the warning seriously. To genuinely change the
cluster secret you must remove the secret first (no service may reference it) and
re-run init.
--save-config defaults to true, so a re-roll also
updates ~/.config/swarmexec/config.yaml — but only
contexts.<name>.{agent_secret, insecure, addr_mode, port} of the
Docker context it deployed to. Another cluster's block, the global keys, comments
and unknown keys are left alone, values that merely came from the environment are
never written, and the file is replaced atomically with mode 0600.
A file that does not parse is reported and not touched. Pass
--save-config=false whenever the local client config must stay
untouched — for example when you re-roll the agents from a machine whose config
is already correct, or from CI.
Then verify. doctor reports a status per node; ok on
every node means client, agent and secret line up again.
A swarmexec ui left open during the re-roll needs no restart. While a
node runs an agent older than a feature, the client asks it once and then leaves
that feature alone on that node for up to 15 minutes instead of paying for the
question on every refresh; the redeployed agent comes up on a new connection, and
a new connection is asked again.
The agent's own flags and environment variables
init deploys the agent with everything it needs, so on a swarm
provisioned that way you never pass any of these. They matter when you run the
agent yourself — from the stack file in
agent/deploy,
under systemd, or in a test — and when you want to know what
init actually set. Every flag has an environment variable of the
same name; the flag wins when both are given.
| Flag | Env | Default | Meaning |
|---|---|---|---|
-port | SWARMEXEC_PORT | 9443 | gRPC listen port; -listen overrides it when set |
-listen | SWARMEXEC_LISTEN | derived from -port | gRPC listen address, e.g. :9443 |
-ca-cert | SWARMEXEC_CA_CERT | — | CA that verifies client certificates — see mutual TLS |
-server-cert | SWARMEXEC_SERVER_CERT | — | server certificate |
-server-key | SWARMEXEC_SERVER_KEY | — | server private key |
-self-signed | SWARMEXEC_SELF_SIGNED | false | generate the server certificate at startup; then needs -ca-cert or a shared secret. This is what init deploys |
-cert-sans | SWARMEXEC_CERT_SANS | — | extra SANs for the self-signed certificate, e.g. DNS:swarmexec-agent,IP:10.0.0.5 |
-agent-secret | SWARMEXEC_AGENT_SECRET | — | the shared secret itself; prefer the file form, so it is not in the service spec |
-agent-secret-file | SWARMEXEC_AGENT_SECRET_FILE | — | read the shared secret from a file, e.g. a Docker secret at /run/secrets/… |
-allow-legacy-secret | SWARMEXEC_ALLOW_LEGACY_SECRET | true | also accept the raw shared secret from clients predating connection-bound auth; each agent logs a warning once when it does. Turn it off once every client is rolled forward |
-policy-file | SWARMEXEC_POLICY_FILE | — | authorization rule file; empty allows every authenticated request — see authorization policy |
-docker-host | SWARMEXEC_DOCKER_HOST | unix:///var/run/docker.sock | the Docker endpoint the agent proxies into |
-forward-image | SWARMEXEC_FORWARD_IMAGE | the agent's own image | image used for port-forward sidecars |
-sysfs-path | SWARMEXEC_SYSFS_PATH | /host/sys | where the host's sysfs is bind-mounted read-only, for the node's link MTUs. Empty disables it. Without the mount the agent reports no links and the client calls the underlay unknown — the honest answer, since a swarm service has its own network namespace and would otherwise report its own veth's 1500 |
-drain-timeout | SWARMEXEC_DRAIN_TIMEOUT | 5s | graceful-shutdown drain window |
-idle-timeout | SWARMEXEC_IDLE_TIMEOUT | 30m | per-session idle timeout; 0 disables it |
-max-session | SWARMEXEC_MAX_SESSION | 12h | per-session maximum duration; 0 disables it |
-max-streams | SWARMEXEC_MAX_STREAMS | 256 | concurrent streams (exec, logs, port-forward, event watches). 0 means this built-in default, negative means unlimited |
-max-forward-sidecars | SWARMEXEC_MAX_FORWARD_SIDECARS | 64 | live port-forward sidecar containers — one per forward, not per connection. 0 means this built-in default, negative means unlimited |
-log-level | SWARMEXEC_LOG_LEVEL | info | debug / info / warn / error |
-log-format | SWARMEXEC_LOG_FORMAT | json | json or text |
-audit-dest | SWARMEXEC_AUDIT_DEST | stdout | stdout, stderr or a file path; the audit log never contains stdin/stdout payloads |
-metrics-addr | SWARMEXEC_METRICS_ADDR | — (off) | Prometheus listen address, e.g. :9100 — see monitoring the agents |
-healthcheck | — | — | run the container health probe instead of the agent and exit 0/1 |
-version | — | — | print the build and protocol version and exit |
SWARMEXEC_ALLOW_LEGACY_SECRET=flase, a
SWARMEXEC_DRAIN_TIMEOUT=5 without a unit, a non-numeric port or
limit — stops the agent at startup with an error naming the variable, exactly as
an invalid flag would. It does not fall back to the default silently, because a
silent fallback is how a cap you believed in turns out never to have been set.
Booleans accept 1/true/yes/on and 0/false/no/off.
A shared secret shorter than 16 characters is accepted but logged as a
WARN at startup — its length only, never the value.
init generates a 32-byte one.
The four limits bound what a single node can be made to do; they are blast
radius, not rationing. A port-forward creates a container on the
node, one per target port, so without a cap a client that only ever spoke the
protocol correctly could exhaust the node's PIDs and memory and take down every
Swarm workload sharing that machine. Over a cap the agent answers
ResourceExhausted and names the flag to raise: it refuses rather than
queues, because a queue turns a resource limit into an unbounded wait that the
person at the other end cannot tell apart from a hang. The session timeouts end
abandoned sessions — half an hour of silence in both directions is not
someone thinking. Logs and port-forwards deliberately have no
maximum lifetime; following a log for hours is the normal way to use it.