Event Log

The dashboard can write what it sees to a log as JSON lines, one event per line, so other systems can pick it up: Telegraf into InfluxDB, Vector or Fluent Bit into Loki or Elasticsearch, or anything else that reads JSON logs. It needs no database.

Set EVENT_LOG to stdout or to a file path and restart the dashboard:

# Into the dashboard's own output (the pod log in Kubernetes, the journal under systemd)
EVENT_LOG="stdout"

# Or append to a file
EVENT_LOG="/var/log/slurm-dashboard/events.jsonl"

Node states are checked at least every 30 seconds, so a state change is logged within half a minute. Each dashboard process writes its own log; if you run several, set EVENT_LOG on one of them.

Events

Every event has time (ISO 8601, UTC), event and cluster. States are the dashboard's: idle, mixed, allocated, draining, drained, down, maintenance, reserved, powered_down and other.

node_state

A node changed state. Nothing is logged for the first check after the dashboard starts.

{"time":"2026-10-24T20:16:04.140Z","event":"node_state","cluster":"hpc","node":"cn012","state":"drained","previous":"mixed","slurm_state":"IDLE+DRAIN","reason":"bad DIMM"}

slurm_state is Slurm's own state and flags; reason is the drain or down reason, when there is one.

node_metrics

One per node every interval.

{"time":"2026-10-24T20:15:34.666Z","event":"node_metrics","cluster":"hpc","node":"gn03","state":"mixed","cpus":64,"cpus_alloc":32,"cpu_load":28.4,"memory_mb":515000,"memory_alloc_mb":256000,"memory_free_mb":198000,"gpus":4,"gpus_alloc":2,"power_watts":1240}

cluster_metrics

The cluster's totals every interval, after the node events: nodes, a nodes_<state> count for each state present, cpus, cpus_alloc, memory_mb, memory_alloc_mb, gpus, gpus_alloc, and power_watts when any node had a reading.

{"time":"2026-10-24T20:15:34.666Z","event":"cluster_metrics","cluster":"hpc","nodes":120,"nodes_mixed":71,"nodes_allocated":40,"nodes_idle":6,"nodes_drained":3,"cpus":7680,"cpus_alloc":6912,"memory_mb":61800000,"memory_alloc_mb":48100000,"gpus":32,"gpus_alloc":28,"power_watts":61200}

Telegraf to InfluxDB

With EVENT_LOG set to a file, Telegraf's tail input reads it. Names and states become tags, everything else fields:

[[inputs.tail]]
  files = ["/var/log/slurm-dashboard/events.jsonl"]
  from_beginning = false
  data_format = "json_v2"

  [[inputs.tail.json_v2]]
    measurement_name_path = "event"
    timestamp_path = "time"
    timestamp_format = "rfc3339"

    [[inputs.tail.json_v2.object]]
      path = "@this"
      tags = ["cluster", "node", "state", "previous"]
      excluded_keys = ["time", "event"]

[[outputs.influxdb_v2]]
  urls = ["http://influxdb:8086"]
  token = "$INFLUX_TOKEN"
  organization = "hpc"
  bucket = "slurm"

This gives measurements node_metrics, cluster_metrics and node_state, ready to chart in Grafana.

With EVENT_LOG="stdout", the log also has the dashboard's own messages. Every event line starts with {"time", so keep only those lines, for example with a grep in front of Telegraf's execd input, or a filter in your log shipper.