swarmexec › Docs › TUI: the service tree — health, usage, stacks
TUI: the service tree — health, usage, stacks_
What a row's colour, count and markers tell you.
On this page
Service rows
Every service is shown (even one scaled to zero), each row rendered like a
docker service ls line — name, mode, running/desired count, image
and published ports — and coloured by how it is actually doing: the
running/desired count sets the base colour (aqua = all tasks up, orange =
partial, red = down, grey = scaled to zero), and a failing
healthcheck overrides it (see below). A ▸/▾
marker shows whether a service is collapsed or expanded; its running containers
nest underneath. A service that is mid rolling-update carries a
coloured badge on its tree row — ⟳ updating or
↺ rolling back — and the same status appears
in the service inspect overlay; it clears once the update completes.
Jobs
A job is counted by what it finished, not by what is running.
A replicated-job or global-job is not meant to stay up, so
its row shows completed/total instead of running/desired: a job that
ran through reads 3/3 in aqua where it used to read
0/1 in red — the display of an outage, for a service that did
exactly what it was asked to do. Swarm reports a replicated job's desired
count as its MaxConcurrent, which is why the live count says nothing
useful once the job is done. The stack row above it counts the job the same way, so a
nightly backup no longer drags its whole stack into orange between runs.
Healthcheck colours
Coloured by healthcheck, not only by replica count. A service
row's colour used to come from running vs desired alone — so a service whose
every container was failing its healthcheck still rendered as a calm aqua
3/3. The count was true and the row was misleading. Swarm's task
state does not help here either: a task reads running while its
container fails every probe, so the verdict has to come from the node. A failing
probe is now treated as degradation of the same kind as a missing replica:
- every checked container unhealthy → the row goes red, which is as bad as none running;
- some of them unhealthy → orange, like a partial rollout;
- all healthy → the colour the replica count already gave the row, unchanged;
- containers that declare no healthcheck → the row is not recoloured at all. Nothing is known about them, and guessing would be the same mistake in the other direction.
The markers. A service row carries
✖ N unhealthy in red, or
◌ N starting in yellow while the probes
have not passed yet; a container leaf carries
✖ unhealthy /
◌ starting. Unhealthy outranks starting — it
is the one that needs acting on. A stack row rolls the same marker up over everything
inside it (see grouped by stack). A healthy container gets
no marker, and neither does one without a
healthcheck, so the marker keeps meaning something. These sit alongside the
cpu/mem resource markers below, and health comes
first in the row: it is the thing contradicting the count standing
next to it.
Up 3 days (healthy)) — the same string docker ps prints.
The parser is deliberately strict: anything it does not recognise
becomes no verdict rather than a guess, so if the daemon ever rewords that
line swarmexec stops claiming to know instead of reporting a failing container as
healthy. Same precondition as the usage numbers, too — it needs agents from this
release or newer, so a cluster that has not been re-rolled shows no health markers
at all until you roll them out with
swarmexec init --force.
Live CPU & memory
Live CPU & memory markers. Next to those badges a row can also
carry what it is actually using right now — until now swarmexec could only
show what the scheduler had booked (see the Nodes tab). A container
or a service whose usage crosses 70% gets an orange marker and
90% a red one, at the end of the row: the resource named in
words — cpu or mem — followed by the
percentage (cpu 94%,
mem 91%). Both appear when both are hot.
Words rather than symbols, and at the same width: a marker whose meaning has to be
looked up is not doing its job, and letters cannot fail to render in a terminal.
Below 70% nothing is drawn at all, so the markers stay a signal rather than
wallpaper. A service row takes the worst of its
replicas, not an average — an average hides the one container that is
about to die, which is exactly the one worth seeing. (Absolute memory on a service
row is the sum across replicas; the percentage is the peak.)
What the percentages are of. Each one is measured against what that container may actually use: its own limit when it has one, the node's capacity when it does not. That distinction is the whole point — 91% of a 256 MB limit means an OOM kill is close, 91% of a 64 GB node is a different conversation. The inspect overlay's stats view prints both sides of that division per container and names the basis under the table.
Stats); each agent samples its own
containers in the background and answers from memory, so the client just polls it
on the refresh cycle it already has. Two consequences worth knowing. A
CPU percentage is a delta between two readings, so it does not
exist until the agent has taken two (a few seconds) — while it isn't ready the
views show …, never 0%; memory needs no delta and appears
from the first reading. And an agent only samples while someone is actually
looking: it stops about a minute after the last request, and forgets its
readings rather than serving stale ones. So the very first numbers after you open
the UI take a moment to fill in. That is deliberate — an agent nobody is watching
should cost nothing.
Stats
RPC — simply contributes no readings. The markers stay off, the node
block stays absent, and everything else is untouched; the client also backs off
from such a node rather than probing it every refresh. On a cluster that has not
been re-rolled yet there are therefore no usage numbers and no health
markers at all
until the agents are updated — roll them out with
swarmexec init --force. This is by far the most
likely reason you see nothing.
Grouped by stack
Grouped by stack. docker stack deploy labels every
service it creates with com.docker.stack.namespace. swarmexec reads
that label off the service spec the manager already returned — Swarm has no stack
object, and this is the only link — and nests the tree three levels deep:
stack → service → container. A stack row shows the stack
name, how many services are under it and the summed running/desired
task count ((3 svc · 7/8)), coloured by exactly the same
rule as a service row — the summed running/desired count, with the same
healthcheck override on top of it. Roll-up counters follow it when they are
non-zero: ⟳ n — services in that stack currently
mid rolling-update — 🛡 n —
services with an actionable security finding, the same shield the
service rows carry — and then the health marker,
✖ N unhealthy or
◌ N starting, summed over every container
of every service in the stack. So a folded stack still tells you whether anything
inside it needs attention. Health rolling up matters most here: the stack row is the
level an operator scans first, so a misleading “everything is up”
is worse there than one tier down, not better.
Services created with docker service create carry no stack label; they
are collected under (no stack), which always sorts
last so it never pushes real stacks down the list. Nothing is hidden
— every service still appears exactly once. Stacks sort by name and start expanded,
so the grouped tree shows the same services the flat one did; folds you make survive
the auto-refresh.
Grouping is on by default but only takes effect when at least one
service in the cluster actually carries a stack label — on a cluster without stacks
the tree looks exactly as it always did, with no pointless (no stack)
parent. Press s to switch between the grouped tree and the flat service
list (the footer lists it as s group); with no stack labels anywhere it
says so rather than redrawing an identical tree. The key is remappable as
stack_group in keys.yaml.
Search
Searching (/) is unchanged: the filter still matches service, container and node. A stack the filter empties disappears from the tree, and each stack row's counts describe what actually ended up under it — so the numbers follow the filter instead of advertising services you filtered away.
:latest version check
Swarm pins :latest to a digest at
deploy time, so a service tagged :latest actually runs a fixed image.
swarmexec resolves that against the registry (using your local docker credentials) and
annotates the service row right after the image URI: the
real version in parentheses — read from the image's
org.opencontainers.image.version label — and an
up-arrow ↑ when the registry's current :latest is a
newer digest than the one the service is pinned to (e.g.
nginx:latest (1.4.0) ↑).
Best-effort and cached: registry errors just leave the row unannotated. The inspect
overlay's IMAGE section shows the same version and, when a newer image
exists, a selectable newer version available row — move onto it and
press u (or Enter) to update the service — see setting a version.