swarmexec

swarmexec › Docs › The agents: init, down, doctor

The agents: init, down, doctor_

Provisioning, removing, diagnosing and monitoring the per-node agents.

init — provision the agents

Deploys the agent as a global Swarm service via the manager API: creates a shared-secret Docker secret, runs the agent in self-signed mode on every node (host port 9443), and writes the matching client config. Run it once per swarm.

swarmexec init
FlagDefaultDescription
--imagedocker.io/logleio/swarmexec-agent:latestagent image to deploy
--secretrandomshared secret to use (default: generate a random one). Note: if the agent secret already exists it is kept as-is (Docker secrets are immutable), so --secret is not applied to the cluster — it is only written to your client config, and init warns about the mismatch risk. To change the cluster secret, remove it first (no service may reference it), then re-run init
--service-nameswarmexec_agentname for the agent service
--port9443host port the agent publishes
--forcefalseupdate the service if it already exists. A prune schedule set on the agents is kept
--allow-legacy-secrettrueaccept the raw shared secret from clients predating connection-bound auth; pass =false once every client is upgraded
--metrics-port0publish the agent's Prometheus metrics on this host port (0 = off) — see monitoring the agents
--policy-file—authorization rule file for the agents, checked before anything is deployed — see authorization policy
--save-configtruewrite the client config
--registry-authtruepass local registry credentials so nodes can pull a private image
--waittruewait for the agents to come up and report progress
--rollout-timeout90show long to wait for agents to start
The default agent image is public on Docker Hub, so no registry login is needed. Re-run with --force to roll out a new agent version — see re-rolling the agents without changing the secret.

down — remove the agents

The inverse of init: removes the global agent service and, by default, the shared-secret Docker secret. It does not touch your client config.

swarmexec down
FlagDefaultDescription
--service-nameswarmexec_agentname of the agent service to remove
--keep-secretfalsedo not remove the shared-secret Docker secret
-y, --yesfalsedo not prompt for confirmation

doctor — diagnose the swarm

Checks the manager connection, whether the agent service is deployed, and probes every ready node's agent for reachability and version skew. Prints a per-node table (NODE  AGENT  VERSION  PROTO); exits non-zero if the manager is unreachable or any agent is unhealthy. A node reported as too old (init --force) runs an agent older than your client — see re-rolling the agents without changing the secret. A node that does not answer is unreachable, and an ERROR column (error in --json) says why: a rejected secret, a TLS failure and a refused connection read differently there. doctor also runs without a complete agent configuration, as on a machine that has not run init: it reports the manager and whether the agent is deployed, adds a config: line (config_error) naming what is missing, skips the per-node probe and exits non-zero. When a newer swarmexec release exists, an update: line under the client line says so (latest_release in --json) — information only, it does not change the exit code; see ui.check_updates.

swarmexec doctor
FlagDefaultDescription
--connect-timeout10sper-node connect timeout
--jsonfalseoutput JSON instead of a table

status — the cluster at a glance

Every node with its role, state, availability and engine; every service with its stack, mode, replicas, the health of its containers and a rolling update in flight; and a summary line counting the degraded services — running short, or with an unhealthy container. Replicas come from the manager, health from the node agents, best effort: without them the services are listed without health and a note on stderr says so, rather than showing all healthy. The same report as the MCP tool cluster_overview.

swarmexec status swarmexec status --json | jq '.services[] | select(.replicas != "1/1")'

stats — live CPU and memory per container

One reading of every running container of the named services — all services when none is named: CPU, memory against its limit, and health. The node agents measure; the manager only knows what was reserved. An agent samples only while someone asks, so a first reading takes a few seconds: stats waits up to --wait (12 s) for CPU, which needs two samples, then prints what it has — warming up for a container still without a reading, unavailable where the node's agent did not answer, each explained once on stderr. The same rows as the MCP tool stats.

swarmexec stats swarmexec stats web api swarmexec stats --json web
FlagDefaultDescription
--jsonfalseoutput JSON instead of the table (status and stats)
--wait12sstats: how long to wait for agents still warming up; 0 asks once
--connect-timeout10sper-node connect timeout

Monitoring the agents

Every agent can serve Prometheus metrics. They are off by default; swarmexec init --force --metrics-port 9464 turns them on: the agent listens on that port and publishes it in host mode on every node, like the agent port, so Prometheus reaches each node's agent at <node>:9464. The endpoint is plain HTTP and carries counts only — no names, ids or payload — but publish it only where your scraper can reach it.

Strict configuration. A set-but-invalid SWARMEXEC_* environment value (SWARMEXEC_ALLOW_LEGACY_SECRET=flase, a duration without a unit, a non-numeric port) stops the agent at startup with an error naming the variable, as an invalid flag does; it no longer falls back to the default silently. A shared secret under 16 characters is accepted but logged as a warning at startup (its length, never its value). When the agent cannot inspect a container it refuses the request after authorizing and auditing it, because a policy rule scoped to services cannot be applied to a container whose service is unknown.

MetricMeaning
swarmexec_agent_info{version,protocol}constant 1; compare it across nodes to see version skew
swarmexec_active_sessionsexec sessions open right now
swarmexec_sessions_totalexec sessions started
swarmexec_auth_denials_totalrequests the authorizer refused — the one signal that makes an attempt visible
swarmexec_auth_failures_total{reason}failed shared-secret authentications (missing, invalid, legacy_refused) — someone guessing the secret; the agent's own loopback health probe is not counted
swarmexec_limit_refusals_total{limit}requests refused at a node cap (streams, forward_sidecars)
swarmexec_active_streams{method}streams open right now, per RPC (Exec, Logs, PortForward, WatchContainerEvents)
swarmexec_streams_total{method}streams admitted past the drain and the stream cap, per RPC — including ones the authorizer then refuses
swarmexec_bytes_in_total / _out_totalbytes from clients (exec stdin, forwards) / to clients (exec output, logs, forwards)

The repository ships what goes around it, in agent/deploy/monitoring/: a scrape config (Prometheus' own Docker Swarm discovery, or static targets), four alert rules — authorization denials rising, agents on different versions for more than 30 minutes, cap refusals, an agent not answering — and a Grafana dashboard (grafana-dashboard.json, import and pick your Prometheus).

Health check

Independently of metrics, every agent has a Docker health check, and Swarm acts on it: a task that turns unhealthy is replaced. It ships in the agent image, so the TUI's image update turns it on — no init --force needed. The probe is the agent binary itself (/agent -healthcheck; the image has no shell): it dials the agent's own port on loopback, calls Version — without credentials; the agent refusing them is an answer too — and pings Docker over the agent's socket, which catches the one failure a restart fixes: a daemon restarted with live-restore leaves the agent holding a dead socket.

It runs every 10 s with a 10 s timeout, after a 30 s start period, and needs 6 failures in a row — a minute or two of a dead or wedged agent — before Swarm replaces it, so a brief daemon restart does not trip it. A rolling agent update now waits for each node's new agent to pass its first check, about 10 s per node. On a node, docker inspect --format '{{json .State.Health}}' <agent container> shows the last results with the probe's reason.

Re-rolling the agents without changing the secret

Sooner or later you have to redeploy the agents on a swarm that is already provisioned: an agent hangs, doctor shows a node as too old (init --force), a command fails with “agent is older than this client (missing RPC) — update it with swarmexec init --force”, or a node was rebuilt and its agent never came back. The fix is to re-run init with --force, which tells it to update the service if it already exists. Without the flag, running init against an existing service is a usage error telling you to add --force — nothing is changed.

A release that changes the agent's service spec needs init --force too, and an image update from the TUI is not enough for it. v1.30.0 is such a release: the agent gained a read-only /sys bind mount so it can report the node's own link MTUs, and a mount lives in the spec, not in the image. Until the re-roll the agents run fine and simply report no links, so the Networks tab says the underlay's MTU is unknown instead of checking an overlay's framing.

The “missing RPC” message is what a feature newer than the agent reports: the container menu's inspect container (node-local) and kill task, for instance, need an agent from the release that introduced them (RPCs InspectContainer and KillContainer). Everything else keeps working against the old agent.

$ swarmexec init --force

This redeploys the agent on every node and leaves the shared secret alone. Docker secrets are immutable, so init finds the existing swarmexec_agent_secret, reports it as reusing existing, and keeps the value that is already in the cluster. Every client that worked before the re-roll keeps working — there is nothing to redistribute.

Do not pass --secret on a re-roll. If the secret already exists it cannot be overwritten, so your value is not applied to the cluster — but it is still written into your client config. Unless it happens to be the existing secret's real value, the agents will reject you from that moment on, and the failure surfaces later as what looks like a broken agent rather than a wrong local config. init warns about exactly this when it happens; take the warning seriously. To genuinely change the cluster secret you must remove the secret first (no service may reference it) and re-run init.

--save-config defaults to true, so a re-roll also updates ~/.config/swarmexec/config.yaml — but only contexts.<name>.{agent_secret, insecure, addr_mode, port} of the Docker context it deployed to. Another cluster's block, the global keys, comments and unknown keys are left alone, values that merely came from the environment are never written, and the file is replaced atomically with mode 0600. A file that does not parse is reported and not touched. Pass --save-config=false whenever the local client config must stay untouched — for example when you re-roll the agents from a machine whose config is already correct, or from CI.

$ swarmexec init --force --save-config=false

Then verify. doctor reports a status per node; ok on every node means client, agent and secret line up again.

A swarmexec ui left open during the re-roll needs no restart. While a node runs an agent older than a feature, the client asks it once and then leaves that feature alone on that node for up to 15 minutes instead of paying for the question on every refresh; the redeployed agent comes up on a new connection, and a new connection is asked again.

$ swarmexec doctor

The agent's own flags and environment variables

init deploys the agent with everything it needs, so on a swarm provisioned that way you never pass any of these. They matter when you run the agent yourself — from the stack file in agent/deploy, under systemd, or in a test — and when you want to know what init actually set. Every flag has an environment variable of the same name; the flag wins when both are given.

$ docker run --rm logleio/swarmexec-agent:latest --version
FlagEnvDefaultMeaning
-portSWARMEXEC_PORT9443gRPC listen port; -listen overrides it when set
-listenSWARMEXEC_LISTENderived from -portgRPC listen address, e.g. :9443
-ca-certSWARMEXEC_CA_CERT—CA that verifies client certificates — see mutual TLS
-server-certSWARMEXEC_SERVER_CERT—server certificate
-server-keySWARMEXEC_SERVER_KEY—server private key
-self-signedSWARMEXEC_SELF_SIGNEDfalsegenerate the server certificate at startup; then needs -ca-cert or a shared secret. This is what init deploys
-cert-sansSWARMEXEC_CERT_SANS—extra SANs for the self-signed certificate, e.g. DNS:swarmexec-agent,IP:10.0.0.5
-agent-secretSWARMEXEC_AGENT_SECRET—the shared secret itself; prefer the file form, so it is not in the service spec
-agent-secret-fileSWARMEXEC_AGENT_SECRET_FILE—read the shared secret from a file, e.g. a Docker secret at /run/secrets/…
-allow-legacy-secretSWARMEXEC_ALLOW_LEGACY_SECRETtruealso accept the raw shared secret from clients predating connection-bound auth; each agent logs a warning once when it does. Turn it off once every client is rolled forward
-policy-fileSWARMEXEC_POLICY_FILE—authorization rule file; empty allows every authenticated request — see authorization policy
-docker-hostSWARMEXEC_DOCKER_HOSTunix:///var/run/docker.sockthe Docker endpoint the agent proxies into
-forward-imageSWARMEXEC_FORWARD_IMAGEthe agent's own imageimage used for port-forward sidecars
-sysfs-pathSWARMEXEC_SYSFS_PATH/host/syswhere the host's sysfs is bind-mounted read-only, for the node's link MTUs. Empty disables it. Without the mount the agent reports no links and the client calls the underlay unknown — the honest answer, since a swarm service has its own network namespace and would otherwise report its own veth's 1500
-drain-timeoutSWARMEXEC_DRAIN_TIMEOUT5sgraceful-shutdown drain window
-idle-timeoutSWARMEXEC_IDLE_TIMEOUT30mper-session idle timeout; 0 disables it
-max-sessionSWARMEXEC_MAX_SESSION12hper-session maximum duration; 0 disables it
-max-streamsSWARMEXEC_MAX_STREAMS256concurrent streams (exec, logs, port-forward, event watches). 0 means this built-in default, negative means unlimited
-max-forward-sidecarsSWARMEXEC_MAX_FORWARD_SIDECARS64live port-forward sidecar containers — one per forward, not per connection. 0 means this built-in default, negative means unlimited
-log-levelSWARMEXEC_LOG_LEVELinfodebug / info / warn / error
-log-formatSWARMEXEC_LOG_FORMATjsonjson or text
-audit-destSWARMEXEC_AUDIT_DESTstdoutstdout, stderr or a file path; the audit log never contains stdin/stdout payloads
-metrics-addrSWARMEXEC_METRICS_ADDR— (off)Prometheus listen address, e.g. :9100 — see monitoring the agents
-healthcheck——run the container health probe instead of the agent and exit 0/1
-version——print the build and protocol version and exit
An environment variable that is set but invalid — SWARMEXEC_ALLOW_LEGACY_SECRET=flase, a SWARMEXEC_DRAIN_TIMEOUT=5 without a unit, a non-numeric port or limit — stops the agent at startup with an error naming the variable, exactly as an invalid flag would. It does not fall back to the default silently, because a silent fallback is how a cap you believed in turns out never to have been set. Booleans accept 1/true/yes/on and 0/false/no/off.

A shared secret shorter than 16 characters is accepted but logged as a WARN at startup — its length only, never the value. init generates a 32-byte one.

The four limits bound what a single node can be made to do; they are blast radius, not rationing. A port-forward creates a container on the node, one per target port, so without a cap a client that only ever spoke the protocol correctly could exhaust the node's PIDs and memory and take down every Swarm workload sharing that machine. Over a cap the agent answers ResourceExhausted and names the flag to raise: it refuses rather than queues, because a queue turns a resource limit into an unbounded wait that the person at the other end cannot tell apart from a hang. The session timeouts end abandoned sessions — half an hour of silence in both directions is not someone thinking. Logs and port-forwards deliberately have no maximum lifetime; following a log for hours is the normal way to use it.