Node History

Node history records what each node was doing over time (busy, idle, drained, down) and every period it was unavailable, with the reason Slurm gave. It answers questions like "how busy has this node been this month?" and "how often has it been drained, and by whom?" without digging through slurmctld logs. It's new in 3.0.

What It Records

The dashboard checks node states every minute by default and keeps two records.

Time in Each State

Each check adds the time since the last check to the node's current state for that hour. Each node is counted in one state at a time, the first that applies from this list:

Totals are kept per hour, so storage doesn't grow with how often jobs start and end. If the dashboard stops collecting, those hours stay empty rather than being counted as the last known state.

Drain and Down Periods

When a node enters one of these states, the dashboard opens a period for that node:

• DOWN
• DRAIN
• FAIL
• ERROR
• NOT_RESPONDING
• MAINTENANCE
• REBOOT_ISSUED
• INVALID_REG

Each period stores the state (for example DRAIN or DOWN+NOT_RESPONDING), Slurm's reason, the user who set the reason, and when the period started and ended. The start time comes from Slurm's timestamp for the reason when it has one.

A period ends when the node is available again or no longer exists in Slurm. If the state or the reason changes while the node is still unavailable, the old period ends and a new one starts where it left off, so periods never overlap.

Where It Appears

Click a node on the dashboard to open its popup. Above the node's jobs, the Node history, last 30 days panel shows:

• A bar covering the last 30 days, one segment per hour, colored by the node's state in that hour. Colors follow your node card color scheme. The schemes use one color for drained and down, so down is hatched, and draining is striped with the partial color. Maintenance is stone, reserved indigo and powered down gray. A drain, outage or maintenance that covered at least a tenth of an hour colors that hour, so short outages stay visible. Hours with no data are left empty.
• Hovering an hour shows how long the node spent in each state during it.
• A legend with each state's share of the recorded time.
• A list of drain and down periods with the state, the reason and who set it, the start and end times, and the duration. A period that's still open ends at "now".

The panel only appears when node history is on.

Requirements

Node history needs a PostgreSQL database. Set DATABASE_URL and the dashboard creates its node_state_time and node_events tables and starts recording. See Features and the Database for database setup and permissions.

History starts when collection starts. A node that's already unavailable at that point gets a period that starts at Slurm's timestamp for its reason; periods that ended before collection started aren't recorded.

Settings

State totals take one row per node, hour and state. A 1,000-node cluster keeps roughly a million rows for 30 days.

If you run several dashboard instances against one database, only one of them collects at a time; the others take over if it stops.