swarmexec

swarmexec › Docs › TUI: the service tree — health, usage, stacks

TUI: the service tree — health, usage, stacks_

What a row's colour, count and markers tell you.

Service rows

Every service is shown (even one scaled to zero), each row rendered like a docker service ls line — name, mode, running/desired count, image and published ports — and coloured by how it is actually doing: the running/desired count sets the base colour (aqua = all tasks up, orange = partial, red = down, grey = scaled to zero), and a failing healthcheck overrides it (see below). A ▸/▾ marker shows whether a service is collapsed or expanded; its running containers nest underneath. A service that is mid rolling-update carries a coloured badge on its tree row — ⟳ updating or ↺ rolling back — and the same status appears in the service inspect overlay; it clears once the update completes.

Jobs

A job is counted by what it finished, not by what is running. A replicated-job or global-job is not meant to stay up, so its row shows completed/total instead of running/desired: a job that ran through reads 3/3 in aqua where it used to read 0/1 in red — the display of an outage, for a service that did exactly what it was asked to do. Swarm reports a replicated job's desired count as its MaxConcurrent, which is why the live count says nothing useful once the job is done. The stack row above it counts the job the same way, so a nightly backup no longer drags its whole stack into orange between runs.

Healthcheck colours

Coloured by healthcheck, not only by replica count. A service row's colour used to come from running vs desired alone — so a service whose every container was failing its healthcheck still rendered as a calm aqua 3/3. The count was true and the row was misleading. Swarm's task state does not help here either: a task reads running while its container fails every probe, so the verdict has to come from the node. A failing probe is now treated as degradation of the same kind as a missing replica:

The markers. A service row carries ✖ N unhealthy in red, or ◌ N starting in yellow while the probes have not passed yet; a container leaf carries ✖ unhealthy / ◌ starting. Unhealthy outranks starting — it is the one that needs acting on. A stack row rolls the same marker up over everything inside it (see grouped by stack). A healthy container gets no marker, and neither does one without a healthcheck, so the marker keeps meaning something. These sit alongside the cpu/mem resource markers below, and health comes first in the row: it is the thing contradicting the count standing next to it.

Where the verdict comes from — and what it costs. Nothing extra: it rides along with the resource readings below, same agent, same call, no additional API request. The container list the agent already fetches on each sampling pass carries the verdict in its status line (Up 3 days (healthy)) — the same string docker ps prints. The parser is deliberately strict: anything it does not recognise becomes no verdict rather than a guess, so if the daemon ever rewords that line swarmexec stops claiming to know instead of reporting a failing container as healthy. Same precondition as the usage numbers, too — it needs agents from this release or newer, so a cluster that has not been re-rolled shows no health markers at all until you roll them out with swarmexec init --force.

Live CPU & memory

Live CPU & memory markers. Next to those badges a row can also carry what it is actually using right now — until now swarmexec could only show what the scheduler had booked (see the Nodes tab). A container or a service whose usage crosses 70% gets an orange marker and 90% a red one, at the end of the row: the resource named in words — cpu or mem — followed by the percentage (cpu 94%, mem 91%). Both appear when both are hot. Words rather than symbols, and at the same width: a marker whose meaning has to be looked up is not doing its job, and letters cannot fail to render in a terminal. Below 70% nothing is drawn at all, so the markers stay a signal rather than wallpaper. A service row takes the worst of its replicas, not an average — an average hides the one container that is about to die, which is exactly the one worth seeing. (Absolute memory on a service row is the sum across replicas; the percentage is the peak.)

What the percentages are of. Each one is measured against what that container may actually use: its own limit when it has one, the node's capacity when it does not. That distinction is the whole point — 91% of a 256 MB limit means an OOM kill is close, 91% of a 64 GB node is a different conversation. The inspect overlay's stats view prints both sides of that division per container and names the basis under the table.

Where the numbers come from — and why they arrive late. Usage is not in the manager API: container stats are per node. They come from a node-local agent RPC (Stats); each agent samples its own containers in the background and answers from memory, so the client just polls it on the refresh cycle it already has. Two consequences worth knowing. A CPU percentage is a delta between two readings, so it does not exist until the agent has taken two (a few seconds) — while it isn't ready the views show …, never 0%; memory needs no delta and appears from the first reading. And an agent only samples while someone is actually looking: it stops about a minute after the last request, and forgets its readings rather than serving stale ones. So the very first numbers after you open the UI take a moment to fill in. That is deliberate — an agent nobody is watching should cost nothing.
No markers anywhere? Check the agent version. Usage is supplementary, and it degrades quietly: a node whose agent is unreachable — or older than this release and therefore unaware of the Stats RPC — simply contributes no readings. The markers stay off, the node block stays absent, and everything else is untouched; the client also backs off from such a node rather than probing it every refresh. On a cluster that has not been re-rolled yet there are therefore no usage numbers and no health markers at all until the agents are updated — roll them out with swarmexec init --force. This is by far the most likely reason you see nothing.

Grouped by stack

Grouped by stack. docker stack deploy labels every service it creates with com.docker.stack.namespace. swarmexec reads that label off the service spec the manager already returned — Swarm has no stack object, and this is the only link — and nests the tree three levels deep: stack → service → container. A stack row shows the stack name, how many services are under it and the summed running/desired task count ((3 svc · 7/8)), coloured by exactly the same rule as a service row — the summed running/desired count, with the same healthcheck override on top of it. Roll-up counters follow it when they are non-zero: ⟳ n — services in that stack currently mid rolling-update — 🛡 n — services with an actionable security finding, the same shield the service rows carry — and then the health marker, ✖ N unhealthy or ◌ N starting, summed over every container of every service in the stack. So a folded stack still tells you whether anything inside it needs attention. Health rolling up matters most here: the stack row is the level an operator scans first, so a misleading “everything is up” is worse there than one tier down, not better.

Services created with docker service create carry no stack label; they are collected under (no stack), which always sorts last so it never pushes real stacks down the list. Nothing is hidden — every service still appears exactly once. Stacks sort by name and start expanded, so the grouped tree shows the same services the flat one did; folds you make survive the auto-refresh.

Grouping is on by default but only takes effect when at least one service in the cluster actually carries a stack label — on a cluster without stacks the tree looks exactly as it always did, with no pointless (no stack) parent. Press s to switch between the grouped tree and the flat service list (the footer lists it as s group); with no stack labels anywhere it says so rather than redrawing an identical tree. The key is remappable as stack_group in keys.yaml.

Searching (/) is unchanged: the filter still matches service, container and node. A stack the filter empties disappears from the tree, and each stack row's counts describe what actually ended up under it — so the numbers follow the filter instead of advertising services you filtered away.

:latest version check

Swarm pins :latest to a digest at deploy time, so a service tagged :latest actually runs a fixed image. swarmexec resolves that against the registry (using your local docker credentials) and annotates the service row right after the image URI: the real version in parentheses — read from the image's org.opencontainers.image.version label — and an up-arrow ↑ when the registry's current :latest is a newer digest than the one the service is pinned to (e.g. nginx:latest (1.4.0) ↑). Best-effort and cached: registry errors just leave the row unannotated. The inspect overlay's IMAGE section shows the same version and, when a newer image exists, a selectable newer version available row — move onto it and press u (or Enter) to update the service — see setting a version.