Upgrading to 3.0

3.0 changes how the dashboard is configured and how history and metrics are collected. Most installs upgrade without changing anything. This page lists what changed, what you might need to do, and what you can now remove.

What Changed

• Settings are read when the server starts. Before 3.0, plugin flags, the cluster name and logo, the support URL, the metrics time zone and the hierarchy labels were baked in when the app was built. Changing one now needs a restart instead of a rebuild. Your .env file still works. The only build-time setting left is NEXT_PUBLIC_BASE_PATH.
• Features turn on when what they need is configured. You no longer have to set a flag for each feature. A flag set to "false" still turns its feature off. The admin page's Plugins tab now shows every feature's state and why.
• The dashboard collects history itself. With a database configured, it saves hourly Rewind snapshots, records each node's states and drain and down periods (new: node history), records the cluster's hourly CPU and GPU capacity (for utilization on the metrics page), and captures GPU history. These replace the cron jobs from earlier versions.
• The metrics page has four tabs. Overview, Explorer, Efficiency and Jobs share one filter bar, and the URL holds the whole view, so links can be shared. See Job Metrics.
• Rewind looks up hours in UTC. Snapshot file names are read as UTC times. See Snapshot File Times if your cron host isn't on UTC.
• SLURM_API_VERSION is optional. When it's unset, the dashboard uses the newest version your slurmrestd serves.
• DATABASE_URL replaces SLURM_JOB_METRICS_DATABASE_URL. The old name still works.

What Works With and Without a Database

See Features and the Database for flags and database setup.

Upgrade Steps

Everyone

1. Update the code or image, build, and restart as usual. See Updating the Dashboard.
2. If you set any settings only in the build environment (for example NEXT_PUBLIC_ENABLE_GPU_UTILIZATION=true npm run build), set them where the server runs instead: in .env or the service's environment.
3. Open Admin → Plugins and check that the features you use are on. Each feature that's off says why.

If You Build the Docker Image

Build arguments for plugin flags, CLUSTER_NAME, CLUSTER_LOGO, NEXT_PUBLIC_BASE_URL, NEXT_PUBLIC_SUPPORT_URL and the hierarchy labels still work: each becomes the container's default. You can now set them at runtime instead (docker run -e, an env file, or your Kubernetes manifest), which lets you use one image for several clusters. NEXT_PUBLIC_BASE_PATH is still a build argument.

If You Use Job Metrics, GPU History or Rewind

• Database permissions: on first start, the dashboard creates the tables it writes (dashboard_schema_migrations, node_events, node_state_time, node_snapshots, cluster_capacity, and accounts, organizations, account_mappings and job_gpu_metrics if they're missing). Its database user needs CREATE permission on the public schema for that first start. If it doesn't have it, job metrics keep working, and Rewind snapshots and node history stay off until you grant it:

  GRANT CREATE ON SCHEMA public TO <dashboard user>;
  

  PostgreSQL 15 and later no longer let every user create tables in public. The simplest setup is a database the dashboard's user owns (CREATE DATABASE slurmdash OWNER slurmdash;), which needs no grant. If the dashboard shares a database with slurm-history-ingestor under a different user, run the grant above.

• Remove the GPU capture cron job (curl -X POST .../api/gpu). The dashboard now captures every minute. The endpoint still works, so leaving the cron job in place does no harm.

• Remove the Rewind snapshot cron job. With a database the dashboard saves snapshots itself. Rewind still reads the snapshot files in data/, so older history stays available. The cron script was what deleted old files, so delete them yourself once you no longer need them. First check Snapshot File Times.

• Run one collector per database. If you run several dashboard instances against one database, only one of them collects at a time; the others take over if it stops.

If You Use the Modules Page

The sample public/modules.json is no longer shipped (it's now infra/modules.example.json), so Modules only turns on when you provide your own file. If git pull stops because your local public/modules.json would be overwritten, move your file aside, pull, and move it back. To keep the file outside the app directory, set MODULES_FILE.

If You Run slurm-history-ingestor

Upgrade to the release that ships with 3.0. It saves its sync position, so a quiet cluster no longer re-scans its whole history on every pass, which could pin a CPU. In API mode it uses the newest API version slurmrestd serves when SLURM_API_VERSION is unset (the old default, v0.0.41, isn't served by Slurm 26.05), and it's published as a container image. The first pass after upgrading still scans from the last finished job, once.

Utilization on the metrics page compares jobs with the capacity the dashboard records, matched by cluster name. The dashboard uses Slurm's own name for the cluster (ClusterName), so set the ingestor's CLUSTER_NAME to the same value.

See Slurm History Ingestor for the image and release binaries.

Things to Check

These don't stop the dashboard from starting, so they're easy to miss.

Snapshot File Times

Rewind in 3.0 asks for each hour in UTC and reads snapshot file names as UTC. The cron script from earlier versions named files with date, which uses the server's local time but still ends the name in Z. Before 3.0 that worked, because Rewind matched the time you picked against the name as written. After upgrading, if the cron host wasn't on UTC, your older Rewind history is off by its UTC offset: on a host seven hours behind UTC, the snapshot for 4 PM shows up at 9 AM.

Snapshots the dashboard saves in the database are stored in UTC and aren't affected. To fix the files, rename them once, from the data/ directory. This needs GNU date, and handles daylight saving time:

cd /path/to/slurm-node-dashboard/data
CRON_TZ="America/New_York"   # the time zone of the host that ran the cron job
mkdir -p utc
for f in *.json.gz; do
  local_time=$(echo "${f%.json.gz}" | sed -E 's/T([0-9]{2})-([0-9]{2})-([0-9]{2})\.([0-9]{3})Z$/ \1:\2:\3.\4/')
  utc=$(date -u -d "TZ=\"$CRON_TZ\" $local_time" +%Y-%m-%dT%H-%M-%S.%3NZ) || continue
  mv -n "$f" "utc/$utc.json.gz"
done
mv -n utc/*.json.gz . && rmdir utc

Run it once only: a second run shifts the files again. If you keep the cron job (for Rewind without a database), change it to date -u at the same time. See Historical Data.

Node Charts Can Appear

NEXT_PUBLIC_ENABLE_PROMETHEUS_PLUGIN is now ignored, so a dashboard that had it set to "false" but also had PROMETHEUS_URL set now shows node charts and Prometheus details on large cards. 3.0 also matches nodes to Prometheus by their full name or scrape target, so nodes that never had charts may now match. To keep charts off, unset PROMETHEUS_URL.

Collection Polls slurmrestd Every Minute

With a database, node history fetches all nodes from slurmrestd every minute (NODE_HISTORY_INTERVAL_SECONDS), in one dashboard process per database. Before 3.0, the dashboard only called slurmrestd when someone had a page open. On a very large cluster or a busy slurmrestd, raise the interval.

Admin Sign-In Secret

The admin sign-in needs a secret. The template now calls it BETTER_AUTH_SECRET; an AUTH_SECRET from an older .env still works. Without either, admin sign-in fails. While the app builds, Next.js may log BetterAuthError: You are using the default secret several times; that's the build running without your environment, and the running server uses the secret you set.

Running in Kubernetes or Behind a Firewall

• Database egress: with a NetworkPolicy or a firewall in front of the dashboard, allow it to reach PostgreSQL. If the database is unreachable, pages still load, the history features turn off, and Admin → Plugins says why.
• One collector: run one replica, or several against one database; a database lock picks one to collect. With a ReadWriteOnce volume for data/, use the Recreate strategy.
• Read-only root filesystem: works, with writable volumes at /tmp and /app/.next/cache, and at /app/data if you keep snapshot files there.

Removed

• InfluxDB and maintenance notification flags (NEXT_PUBLIC_ENABLE_INFLUXDB_PLUGIN, NEXT_PUBLIC_MAINT_NOTIFICATIONS_ENABLED): neither did anything beyond showing a status in the admin panel.
• NEXT_PUBLIC_SLURM_API_VERSION: the admin panel now shows the version slurmrestd actually answered with.
• NEXT_PUBLIC_ENABLE_PROMETHEUS_PLUGIN never controlled Prometheus and is now ignored. Node charts are on whenever PROMETHEUS_URL is set, as before.

New Settings