swarmexec

swarmexec › Docs › TUI: the Nodes tab

TUI: the Nodes tab_

Every node, what it holds, and what it could give back.

Nodes tab

Lists the swarm's nodes with general info and a few aggregations — NODE (hostname; managers in aqua, the leader marked ★), ROLE (manager / worker), AVAIL (active / pause / drain, colour-coded), STATE (ready / down), CPU% and MEM% (what the node's containers use, as a share of the node's cores and memory — measured by its agent on the same pass that puts usage badges on the tree, so containers only, not the kernel or anything outside docker; yellow from 80 %, red from 90 %, – when the agent did not answer and … while it has its first CPU sample only), TASKS (running tasks scheduled on the node), VOLS (volumes the node holds — node-local, so it fills in a moment after the rest; – when the node's agent did not answer, rather than a 0 that would read as “no volumes”), ENGINE version and LABELS (count). The ? overlay explains the ★ too.

Every action on this tab has a command-line twin under swarmexec node: node ls for the list, node prune, node update --availability / --label-add / --label-rm, and node prune-schedule.

KeyAction
Enter / inode details — a read-only overlay with hostname, id, role/leader, availability, state, address, engine, platform, CPUs, memory, running-task count, the reserved resources, in-use and images on this node blocks (see below), volume count, the full label list, and the reclaimable and prune schedule blocks. Inside it M measures the reclaimable space now, P opens the prune dialog and S the schedule editor
ledit the selected node's labels — a staged list editor (a add / e edit / d delete / Ctrl-S or w apply / Esc cancel) over key=value entries. Applying does one NodeUpdate — unlike a service, a node update applies immediately (no rolling update). Node labels are commonly used as placement-constraint targets (node.labels.<k>). If the node changed after the tab last loaded it, the apply is refused (“node … was changed by someone else … nothing was written”) and your staged labels stay on screen; refresh the tab and apply again
aset the node's availability — a menu of active / pause / drain that marks the state the node is currently in. active schedules tasks here normally; pause keeps the running tasks and only stops new placements; drain moves every task off the node. Active and Pause apply straight away; Drain goes through a confirm naming how many running tasks swarm will stop and reschedule elsewhere (services whose tasks can't be placed anywhere else go unschedulable). Like a label change it is one NodeUpdate (read-modify-write on the node spec) and applies immediately — nodes have no rolling update. Remappable as node_availability
Pprune the node (capital P, drawn red). Opens a dialog with one checkbox per category, the expected savings, then a confirm and the result per category — see pruning a node. Against an agent older than this release it opens the old two-entry image menu instead. Remappable as node_prune (the old name node_prune_images still works)
Sthe prune schedule for all nodes — see prune schedule. Remappable as node_prune_schedule

Reserved resources. The node detail carries a reserved by tasks (scheduler's view) block: for CPU and memory, a bar plus reserved / capacity and how much is free. The bar is green, turns yellow from 75% and red at 100% or above; over-commitment is called out in words, and “free” never goes negative. A node that reports no capacity says so instead of drawing a bar. The block costs no extra API call — the nodes list already fetches the task list.

These are reservations, not usage. The figures are the sum of the tasks' declared Reservations — exactly the arithmetic the swarm scheduler itself does when deciding whether a task fits on a node, and the usual reason a service sits unschedulable. Live CPU/RAM consumption is node-local and is not what this shows — it has a block of its own, right below. Tasks that declare no reservation are counted and reported separately: they book nothing here yet can still consume the whole node, so a low percentage does not mean the node is idle — and in most clusters few services set reservations at all. A task holds its reservation from the moment it is assigned until it reaches a terminal state, so tasks that are preparing or starting count, while shutdown / failed / complete / rejected ones do not (swarm keeps those in the task list as history).

In use by containers. Beneath that sits a second block — in use by containers (measured on the node) — drawn with the same proportional bars: what the node's containers are actually using right now, CPU in cores and memory, each against the node's own capacity. The two blocks are kept deliberately apart because they answer different questions and are routinely far apart: a node can be fully booked and idle, or barely booked and on fire. The figures come from that node's agent, the same source as the usage markers in the service tree — while the CPU delta isn't ready yet the line reads measuring…, and a node whose agent is unreachable or too old gets no block at all rather than a row of zeros that would read as “idle”.

Containers only. This counts the node's containers — the kernel, the docker daemon and anything running outside docker are not in the figure. It is not the node's load average, and the block says so itself.

Images on this node. Last in the detail comes an images on this node block: how much the node's layer store holds and in how many images, and how much of that is reclaimable in untagged leftovers (a node with nothing untagged says so rather than offering a zero). When there is any, one further line notes how much more is tagged but unused — phrased as a cost, removing it means pulling those images again, not as an offer. The block exists because images are the one node-local resource the cluster side cannot see at all: the manager API has no view of images whatsoever, unlike volumes, which it at least knows are mounted. Volumes could already be managed cluster-wide from swarmexec; a node quietly filling up with old layers could not even be seen. It loads on its own, in parallel with the node list, so it appears a moment after the rest of the overlay.

Reclaimable space and pruning a node. The node detail ends with a reclaimable (measured on the node) block: per category, how many objects a prune would remove and how much it would free, and when that was measured (measured 23 min ago). The agent measures in the background, shortly after it starts and then hourly, and after every prune; m in the detail asks for a fresh measurement (the agent allows one every 30 seconds). P opens the prune dialog: one checkbox per category with its figure, the expected total underneath, then a confirm that lists every ticked category, and finally the result — freed bytes and count per category.

CategoryWhat a prune removesPolicy action
stopped containerscontainers that stopped more than 24 h ago (t in the dialog changes the age). Counted from when they stopped, not when they were created, so the logs of a task that just crashed stay readable. The detail shows how many are kept for nowcontainer.prune
untagged imagesthe layers left behind by rebuilds, which nothing can start from. Safe: it cannot stop a service and it cannot force a pullimage.prune
every unused imagealso removes tagged images that no container is running right now — every service scaled to zero and every task between restarts will have to pull its image againimage.prune.all
build cacheall build cache no build is usingbuildcache.prune
unused local networksnetworks of scope local that no container references, stopped ones included. Never bridge, host, none, docker_gwbridge, ingress or any overlaynetwork.prune
volumesevery volume no container uses, named volumes included — their data is deletedvolume.prune

The dialog starts with containers, untagged images, build cache and networks ticked. a ticks everything except volumes; volumes are a separate, red checkbox that is never ticked for you. The two image entries exclude each other, because every unused image already includes the untagged ones.

Do not read “every unused image” as “the same, but more”. It removes images that belong to something which is merely not running at this instant. A service scaled to zero, a task restarting, a stack you stopped yesterday — all of them lose their image and must pull it again to come back, and if the registry is unreachable they will not come back up. The confirm says so whenever it is ticked, and it says in red that volumes take their data with them whenever those are ticked.
Why networks are not pruned the docker way. On a manager, Docker's own network prune also removes every unused swarm overlay network, cluster-wide, and cannot be limited by scope. The agent therefore never calls it: it lists the node's networks and removes the unused local ones one by one.

One permission per category. The agent authorizes each category as its own action (table above; reading the figures is reclaim.read), so a policy can allow the cheap sweeps without the destructive ones. Every category of every prune is written to the agent's audit log (event=prune) with how much it freed and how many objects went.

Where the numbers come from. They are read from the same source docker system df reports from, and match it exactly — verified on a real node at 56 images, 32.70 GB on disk, 22.99 GB reclaimable. In particular swarmexec does not add up the individual image sizes: an image's size includes every layer it is built from, and layers are shared, so that sum over-reports badly — during development it claimed 32.7 GB where the daemon said 22.2 GB. One caveat worth stating: the untagged-only figure is a lower bound — a layer shared by two untagged images belongs to neither one's unique size — so an untagged prune may free a little more than advertised, never less.

Same precondition as the usage and health numbers above: it needs agents from this release or newer. A node whose agent is older shows agent too old for prune stats in the reclaimable block, and P falls back to the two-entry image menu (untagged leftovers / every unused image) such an agent still understands. Update the agent image — the agent service spec did not change, so init --force is not needed. A node whose agent does not answer at all shows the first cause on one line — agent not reachable on 10.0.1.5:9443: connection refused — swarmexec doctor — and the full error goes to the client log (`).

Prune schedule (all nodes). S on the Nodes tab opens an editor for a schedule every agent runs on its own node: weekday toggles, the start time, runs per day (the day is split evenly from the start: 2 runs from 00:00 are 00:00 and 12:00, 3 are 00:00, 08:00 and 16:00), the time zone, the categories — images as every unused, untagged only or off, and volumes as an option that is off by default — how long stopped containers are kept, and an optional label filter. A live preview shows the resulting run times and the next runs. Ctrl-S (or w) applies after a warning.

Saving restarts the agents. The schedule is stored as container labels on the agent service — the one place an agent on a worker node can read, by inspecting its own container. Saving is therefore a service update: the agents restart one by one, and open shells, log streams and port-forwards on a restarting node drop.

Each agent reads the labels when it starts; labels that do not parse are logged and switch the schedule off, they never stop the agent. A scheduled run is authorized like a request from the identity scheduler (so a policy rule such as deny: [volume.prune] also covers the schedule) and audited with trigger=schedule. The node detail shows the schedule as that node's agent sees it, the next run, and the last run with what it freed per category and any error. swarmexec init --force keeps the labels; the CLI equivalent is node prune-schedule.

Container labelDefaultMeaning
swarmexec.prune.enabledno labels: offtrue runs the schedule
swarmexec.prune.schedule.daysevery daymon,tue,…,sun, ranges like mon-fri
swarmexec.prune.schedule.start00:00first run of the day
swarmexec.prune.schedule.times1runs per day (1–24)
swarmexec.prune.schedule.tzUTCIANA time zone
swarmexec.prune.categoriescontainers,images,buildcache,networksalso dangling-images and volumes
swarmexec.prune.containers.until24hkeep stopped containers this long after they stop
swarmexec.prune.filter.label—only objects with these labels (key or key=value, comma-separated); build cache is then skipped