HPCPerfStats

The package formerly known as TACC Stats

A toolkit for monitoring resource usage on HPC systems at multiple levels of resolution.

---

Overview

The hpcperfstats package is split into two parts:

Component	Build system	Role
monitor	Autotools	Online data collection and transmission in production
hpcperfstats	Python setuptools	Data curation and analysis (off-cluster)

Documentation (docs/)

Document	What it is for
MONITOR_VARIABLES.md	Canonical reference for monitor-reported variables: names, types, units, and semantics. Use this instead of any legacy “attributes definition” doc.
DEPLOY_CONCURRENCY_AND_NUMA.md	Thread/process limits vs PostgreSQL, total_cores / effective_cores, and pool sizing for web and pipeline.
design-document.md	As-built system design: architecture, data flow, components, contracts, and operations context.
measurements/	Recorded ops measurements (for example node-daemon CPU overhead on Stampede3 SPR).
using-the-website-as-a-researcher.md	How to read the Django/React job UI—plots, metrics, and diagnostic themes—for HPC users and researchers.
TESTING.md	Test commands, CI, compose-backed workflows, Playwright/Vitest, and host vs container pytest notes.
upgrade.md	Existing stacks only: image rebuilds, Redis/INI/volume layout, RabbitMQ recreate, PG18 dual-run.
OPERATOR_HOST_DATA_DEV_UNIQUENESS.md	host_data 5-col uniqueness: Stage 1 decompress (+ 02 PK normalize) and Stage 2 one-shot migrate for 0032 / compress_after 8d.

Maintaining MONITOR_VARIABLES.md: the catalog is generated and augmented by maintainer scripts in the same folder: regenerate_monitor_variables_catalog.py, augment_monitor_variables_diagnostics.py.

REST API note: GET /api/jobs/{jid}/{type_name}/ (type detail) returns a Bokeh tplot_item (json_item payload) plus stats_data / schema. Legacy tscript / tdiv fields were removed; clients should embed only tplot_item via Bokeh embed_item.

API contract changes (2026-06):

• GET /api/jobs/ with filters that match zero jobs now returns HTTP 200 with nj: 0, an empty job_list, and a filter_summary object. Do not treat HTTP 404 as “no matches” for successful searches (404 is reserved for database/unavailable errors on this endpoint).
• GET /api/search/ was removed; job-ID and host routing are handled in the React SPA. API clients should use GET /api/jobs/?jid=… (or open /machine/job/{jid}/ in a browser) instead of the old search redirect endpoint.

Building and installing the hpcperfstatsd-3.0-1.el9.x86_64.rpm package (via monitor/hpcperfstats.spec) installs a systemd service hpcperfstats. Production sampling is typically configured for multi-minute intervals (often on the order of minutes), with samples at job start and end. On a Stampede3 Sapphire Rapids node (2026-08-14), a ten-minute pidstat observation of hpcperfstatsd showed sample-window peaks of at most 3.0% of one core and a 0.19% average over the full window; see docs/measurements/monitor_overhead_stampede3_spr_2026-08-14.md. The daemon hpcperfstatsd sends data to a RabbitMQ server over the administrative network. RabbitMQ must be installed and running on the server to receive data.

The hpcperfstats container orchestration sets up a Django/PostgreSQL ingest and archival stack plus a RabbitMQ server to receive data from the monitor on the nodes.

---

Installation

These steps are a fresh install on a new host. For an existing Compose stack (rebuilds, Redis/INI/volume/RabbitMQ changes, PostgreSQL 18 dual-run), use docs/upgrade.md instead.

Monitor subpackage

The monitor now uses a static-bundle build flow for packaging. The canonical path builds pinned static archives for libev, rabbitmq-c, and (on x86) LIKWID, then compiles hpcperfstatsd with --enable-all-static.

1. Install RPM build prerequisites (Rocky/EL-like systems):

   sudo dnf install \
     gcc gcc-c++ make autoconf automake libtool cmake pkgconfig \
     systemd-rpm-macros gzip tar curl perl gawk pciutils rdma-core-devel \
     rpm-build

   On aarch64, install one of:

   sudo dnf install datacenter-gpu-manager-4-devel
   # or
   sudo dnf install libdcgm-devel

2. Prepare rpmbuild directories and source tarball (from HPCPerfStats/monitor):

   ./scripts/prepare_rpmbuild_dirs.sh

   This script:

   • creates monitor/rpmbuild/{SPECS,SOURCES,BUILD,RPMS,SRPMS,BUILDROOT}
   • runs scripts/build_static_bundle.sh --deps-only into monitor/rpmbuild/static-prefix
   • runs autoreconf -fi, ./configure, and make dist
   • copies hpcperfstats-<version>.tar.gz to rpmbuild/SOURCES

3. Build the RPM:

   Use the rpmbuild command printed by scripts/prepare_rpmbuild_dirs.sh. A typical script output is:

   rpmbuild -ba --define "_topdir $(pwd)/rpmbuild" rpmbuild/SPECS/hpcperfstats.spec

4. Optional build options:

   • Reuse existing static deps when already staged:

     SKIP_DEPS=1 ./scripts/prepare_rpmbuild_dirs.sh

   • Build dependencies + monitor binary directly (without rpmbuild staging):

     ./scripts/build_static_bundle.sh

   • Build only pinned dependency archives:

     ./scripts/build_static_bundle.sh --deps-only

   • Release-optimized monitor build:

     ./scripts/build_static_bundle.sh --release
     # equivalent to: HPC_BUNDLE_RELEASE_BUILD=1 ./scripts/build_static_bundle.sh

   • Pass extra configure args through bundle build (example):

     ./scripts/build_static_bundle.sh --disable-lustre

5. Configuration — after install, edit /etc/hpcperfstats/hpcperfstats.conf:

   Field	Description
   server	Hostname or IP of the RabbitMQ server
   queue	System/cluster name being monitored
   port	RabbitMQ port (default 5672)
   freq	Sampling interval in seconds

   Example:

   server localhost
   queue default
   port 5672
   freq 600

   Reload a running daemon with: kill -HUP <pid> (or restart the service).

6. Service control:

   sudo systemctl start hpcperfstats
   sudo systemctl stop hpcperfstats
   sudo systemctl restart hpcperfstats

---

Job scheduler configuration

Job start/end: Notify hpcperfstats by writing to /var/run/stats_jobid on each node:

• Job start: echo the job ID into the file
• Job end: echo - into the file

Do this from your scheduler’s prolog and epilog.

Accounting ingest (SLURM sacct): Use hpcperfstats-sacct-gen from the hpcperfstats-tools package. By default this command runs sacct for a date range and POSTs the results to the HPCPerfStats API ingest endpoint. Each successful POST also creates or overwrites a daily accounting file at {acct_path}/YYYY-MM-DD.txt (same pipe-delimited body Slurm returned) only when the body has at least one job row. Empty or header-only days are not written. Alternatively, use -f DIR to write those YYYY-MM-DD.txt files locally (same format/naming) without calling the API; -f is mutually exclusive with --api-key, and DIR must already exist. The scheduled sync_acct.py job can reingest these files from disk. The API rejects payloads with fewer lines than the existing file for that date (HTTP 409) so a partial sacct export cannot replace a fuller one. Header-only or empty POSTs skip the write without a 409.

1. Install the tools (Python):

   # Client CLIs live in-tree under hpcperfstats-tools/ (separate distribution).
   # From the HPCPerfStats git checkout:
   python3 -m pip install ./hpcperfstats-tools
   # Editable during development:
   # python3 -m pip install -e ./hpcperfstats-tools

2. Configure the API base URL:

   Set HPCPERFSTATS_TOOLS_INI to an INI file that contains [API] base_url (see hpcperfstats-tools/hpcperfstats-tools.ini.example in the repo for a template).

3. Run the ingest (requires a staff-capable API key):

   # Ingest today only (default date range is today .. today, inclusive)
   hpcperfstats-sacct-gen --api-key YOUR_KEY
   
   # Ingest an explicit date range (both ends inclusive)
   hpcperfstats-sacct-gen 2024-01-01 2024-01-08 --api-key YOUR_KEY
   
   # Write daily YYYY-MM-DD.txt files locally instead of POSTing (DIR must exist;
   # mutually exclusive with --api-key)
   hpcperfstats-sacct-gen -f /path/to/accounting 2024-01-01 2024-01-08

Run-location and permissions requirements:

• Run hpcperfstats-sacct-gen on a host where Slurm’s sacct binary exists and works (typically a Slurm login node).
• Run it as a user that has the correct Slurm permissions to query the relevant jobs/accounts via sacct.
• API mode: the API key you pass with --api-key must be staff-capable for the ingest endpoint; the web container must have the shared data volume mounted at /hpcperfstats/ (same as pipeline) so the API can write under acct_path (default /hpcperfstats/accounting).
• File mode (-f DIR): DIR must already exist; no API URL or key is required. Place or sync the resulting .txt files where sync_acct.py expects them (acct_path).

---

hpcperfstats subpackage (container stack)

This is a container orchestration with Django/PostgreSQL, ingest/archival tools, and RabbitMQ. The steps below assume a Rocky Linux host.

1. Install rootless Podman, Compose, and Node 24:

   sudo dnf module reset -y nodejs
   sudo dnf module enable -y nodejs:24
   sudo dnf install -y git podman podman-compose nodejs npm \
     shadow-utils fuse-overlayfs slirp4netns

   Run containers as the unprivileged deployment account—never with sudo podman. Rootless subordinate IDs must cover image uid 901860; allocate a non-overlapping range of at least 1,048,576 IDs in both /etc/subuid and /etc/subgid, then run podman system migrate as that account. Example for user sharrell (choose a site-safe unused start):

   sudo usermod --add-subuids 1000000-2048575 \
     --add-subgids 1000000-2048575 sharrell
   podman system migrate

   Redis / Linux kernel: Redis warns when vm.overcommit_memory is disabled; background saves use fork(), and the kernel can reject that fork even with free RAM. Enable it on the Linux Podman host:

   sudo sysctl -w vm.overcommit_memory=1

   Persist across reboots:

   echo 'vm.overcommit_memory = 1' | sudo tee /etc/sysctl.d/99-redis-overcommit.conf
   sudo sysctl --system

   Alternatively, add vm.overcommit_memory = 1 to /etc/sysctl.conf and reboot (or run the sysctl -w command once).

   Compose pins Redis Open Source 8.10 (redis:8.10.1-alpine3.23) with maxmemory 16gb, volatile-lru (Django cache keys keep TTL and remain evictable), --io-threads 4 / --io-threads-do-reads yes, Redis 8.10 compact hashes (--hash-min-template-entries 1, default 0 disables auto-conversion), and a Unix domain socket at /run/redis/redis.sock shared by redis, web, and pipeline via the redis_runtime named volume (not a host bind). Do not put Redis on web / pipeline depends_on (any condition:, including service_started / service_healthy) — podman-compose can create hpcperfstats_redis_1 and never start it, so logs redis stays empty. Redis still has a TCP+socket PING healthcheck for ps. Startup wait uses [CACHE] redis_location, remaps Compose hostname redis (including baked redis://redis:6379/1) to unix:///run/redis/redis.sock?db=1, and falls back to that socket URL when the INI key is missing. TCP 6379 stays up for redis-cli and external Redis URLs. Do not use allkeys-* for Django/listend cache keys. Redis has no persistence volume (appendonly no). Size the host (or Colima) so Redis can use that cap alongside Postgres shm_size / shared_buffers. Redis image bumps on an existing host: docs/upgrade.md.

2. Enable container restart after reboot:

   sudo loginctl enable-linger sharrell
   systemctl --user enable --now podman-restart.service

   Keep every non-DNF download, cache, tool, temporary file, image layer, writable layer, and volume under /data. Create the roots once:

   sudo mkdir -p /data/user/sharrell/{cache,tmp,tools}
   sudo mkdir -p /data/podman/sharrell/{storage,images,volumes,cache,tmp}
   sudo chown -R sharrell:sharrell /data/user/sharrell /data/podman/sharrell

   Set login exports for XDG_CACHE_HOME=/data/user/sharrell/cache, TMPDIR=/data/user/sharrell/tmp, PIP_CACHE_DIR=/data/user/sharrell/cache/pip, npm_config_cache=/data/user/sharrell/cache/npm, npm_config_prefix=/data/user/sharrell/tools/npm, and PLAYWRIGHT_BROWSERS_PATH=/data/user/sharrell/cache/ms-playwright. Configure rootless storage.conf with graphroot=/data/podman/sharrell/storage and imagestore=/data/podman/sharrell/images; configure containers.conf with volume_path=/data/podman/sharrell/volumes and image_copy_tmp_dir=/data/podman/sharrell/tmp. Small config files may stay under ~/.config; /run/user/$UID remains required ephemeral state. Verify these paths with podman info before the first build.

3. Clone the repo:

   git clone https://github.com/TACC/hpcperfstats.git
   cd hpcperfstats

4. Compose settings (site bind volumes):

   cp docker-compose.settings.yaml.example docker-compose.settings.yaml

   docker-compose.settings.yaml is gitignored — treat docker-compose.settings.yaml.example as the committed operator template. Base docker-compose.yaml includes the settings file automatically; you do not pass a second -f for settings. Named volume definitions (bind device: paths) live only in settings — base compose mounts them by name and must not declare empty volume stubs (podman-compose cannot merge null stubs with settings dicts). After cp, edit site-specific device: paths below; any new bind volumes or optional knobs must be added to .example in the repo (see hpcperfstats/cursor-rules/docker-compose-settings-example-sync.mdc) so the next clone gets them.

   Edit docker-compose.settings.yaml and set at least:

   • volumes → ssh_keys → device: host directory with pipeline SSH keys (permissions suitable for mount as /hpcperfstats/.ssh/)
   • volumes → proxy_ssl_source → device: host directory containing fullchain.pem and privkey.pem (flat PEM dir; default example uses /opt/certs). For Let's Encrypt, set device: /etc/letsencrypt and uncomment the optional services.proxy.environment block in the settings example with HPCPERFSTATS_SSL_CERTS_REL=live/your.hostname — do not bind only the live/hostname leaf (archive symlinks break).

   TLS PEMs are not baked into the image. The proxy service mounts proxy_ssl_source read-only at /mnt/ssl-source; proxy_entrypoint.sh materializes real PEM files into /etc/ssl/hpcperfstats before nginx starts (no .env, no manual resolve step). After cert renew, docker compose restart proxy (no image rebuild).

   Create host directories for every bind device: before up (defaults from the example):

   sudo mkdir -p /data/hpcperfstats_data/site_data
   sudo mkdir -p /data/hpcperfstats_data/site_data/accounting
   sudo mkdir -p /data/hpcperfstats_data/site_data/archive
   sudo mkdir -p /data/hpcperfstats_data/site_data/daily_archive
   sudo mkdir -p /data/hpcperfstats_data/site_data/logs/current
   sudo mkdir -p /data/hpcperfstats_data/site_data/logs/log_archive
   sudo mkdir -p /data/hpcperfstats_data/rabbitmq
   sudo mkdir -p /data/hpcperfstats_site/staticfiles
   sudo mkdir -p /data/hpcperfstats_site/media
   sudo mkdir -p /data/hpcperfstats_db/pg15
   # Also ensure the ssh_keys and proxy_ssl_source device paths you set above exist.

   The hpcperfstatsdata bind maps to /hpcperfstats/ in the pipeline and web containers (for example /hpcperfstats/accounting, /hpcperfstats/archive, /hpcperfstats/daily_archive, and /hpcperfstats/logs/ for cluster syslog).

   Daily monitor archive compression: sync_timedb seals each day’s YYYY-MM-DD.tar to YYYY-MM-DD.tar.zst with zstd. Defaults: archive_zstd_threads=0 (-T0, niced seal/restore), ingest_zstd_threads=4 (-T4, un-niced ingest/populate streams), archive_seal_parallel_workers=4 (concurrent daily seals), nice/ionice deprioritization so seal yields to web/db on shared hosts — see docs/DEPLOY_CONCURRENCY_AND_NUMA.md (Archive zstd priority). Other [PIPELINE] keys: archive_zstd_level, archive_zstd_nice, archive_zstd_ionice_class, archive_zstd_ionice_level in hpcperfstats.ini.example. When inspecting archives by hand, use for example zstd -d -o YYYY-MM-DD.tar YYYY-MM-DD.tar.zst (legacy .tar.gz uses zstd -d --format=gzip). Before raising archive_zstd_level above 9 on production data, benchmark a representative daily tar on the pipeline host: zstd -b3 -e12 -T0 -S -- ./YYYY-MM-DD.tar.

   Daily archive member cache: sync_timedb keeps complete member maps in process memory and persists them under {archive_dir}/.sync_timedb_archive_members/. Dedicated [thread:populate-pool] workers stream sealed/tar archives into that store. Django [CACHE] redis_location remains for the web/listend cache only. Ingest duplicate-check zstd runs at normal priority; janitor seal paths keep archive nice/ionice. Member maps are invalidated after tar append, dedupe, seal, and archive finalize — wipe day sidecars with scripts/invalidate_archive_members.py (never .sync_timedb_job_store.json). See sync_archive_members_populate_* / sync_archive_members_wait_poll_seconds in hpcperfstats.ini.example and docs/DEPLOY_CONCURRENCY_AND_NUMA.md.

   Ingest-first durability vs archive failure: with sync_enable_ingest_first_durability_mode=yes (default), DB-ingested raw may be checkpointed as processed when archive append retries are exhausted (ingest_first_archive_abandoned_raw in logs; entry also in .sync_timedb_dead_letter.json). Raw is not deleted until a later pass verifies tar+sealed archive membership. Recovery: fix archive/tar issues, clear or replay dead-letter entries, and rescan — do not delete raw manually unless you have confirmed DB and archive parity.

   Startup snapshot wait: on large trees with backlog, first pending rescan may wait up to sync_startup_snapshot_wait_seconds (default 300, min 120) for the janitor startup heavy pass to publish StartupArchiveScanCoordinator snapshot before single-flight fallback build. Grep sync_timedb: pending rescan begin, startup archive scan ready, janitor: discover_ready_day_close. Boot DAY_CLOSE discover runs on [sync_timedb:thread:archive-janitor] only — ingest is not gated on day-close completion. Inspect async manifest .sync_timedb_async_day_close.json for in-flight rows. These startup paths run only when the pipeline command includes backlog (see below).

   Startup maintenance (backlog only): janitor reason=startup heavy snapshot + boot handoff for ingest catch-up, then startup ingest gate cleared; ingest may begin. CLI backlog is ingest-only for day-close (current / date-range own seal/verify/delete). Runs when sync_timedb.py is invoked with backlog. Date-window runs skip startup maintenance and begin ingest immediately.

   sync_timedb.py date arguments: with no dates, ingest uses the last five calendar days through now. A single YYYY-MM-DD limits ingest to that day only. Two dates set an explicit start/end range. Prefix once to exit after one idle rescan (for example once 2024-01-15 or once backlog).

   Startup archive scan (single-flight): on large trees with backlog, janitor startup maintenance publishes one build_archive_maintenance_snapshot via StartupArchiveScanCoordinator — tune sync_startup_snapshot_wait_seconds (default 300, min 120) if logs show long waits before startup archive scan ready. Details: docs/DEPLOY_CONCURRENCY_AND_NUMA.md § canonical startup archive scan.

   Cluster syslog (optional; not auto-started): the pipeline service still publishes TCP and UDP port 514 on the Docker host for sites that enable ingest manually. Compute or login nodes should forward syslog to <docker-host>:514 (rsyslog examples: TCP @@host:514, UDP @host:514). syslog-ng and seal_syslog_daily are not supervisord programs — the pipeline supervisor runs as hpcperfstats (uid 901860) and cannot bind privileged port 514. Live files, when syslog is enabled, go under /hpcperfstats/logs/current/ as $HOST.$R_YEAR$R_MONTH$R_DAY.log. seal_syslog_daily packs the previous day’s files into /hpcperfstats/logs/log_archive/YYYY-MM-DD-syslog.tar.gz. Do not add compose user: "901860:901860" on pipeline — the root supervisor_startup.sh must still chown the data bind and copy ssh keys before supervisord drops privileges.

   Manual cluster syslog enable (as root inside pipeline): uncomment or run the two commented lines in supervisor_startup.sh, or by hand:

   docker compose exec pipeline sh -lc '
     mkdir -p /var/lib/hpcperfstats-syslog &&
     /usr/local/bin/python3 -m hpcperfstats.render_syslog_ng_generated &&
     /usr/sbin/syslog-ng -F --no-caps -f /home/hpcperfstats/services-conf/syslog-ng.conf
   '

   Render must run before starting syslog-ng — boot no longer refreshes /var/lib/hpcperfstats-syslog/generated.conf. Run python3 -m hpcperfstats.seal_syslog_daily as hpcperfstats on a schedule if you need daily seals (not supervised).

   [SYSLOG] in hpcperfstats.ini: set allow_from to a comma- or line-separated list of IPv4 CIDRs that may send remote syslog (for example 10.0.0.0/8, 192.168.50.0/24). If allow_from is blank or [SYSLOG] is omitted, all IPv4 sources are accepted (backward compatible). Changing allow_from requires re-running render_syslog_ng_generated (then restart syslog-ng) so /var/lib/hpcperfstats-syslog/generated.conf matches INI — a pipeline recreate alone does not refresh that fragment. listen_tcp / listen_udp (default yes) toggle listeners.

   Operational notes: when syslog-ng is running, it emits periodic internal stats (stats(freq(3600)) in services-conf/syslog-ng.conf); operators can run syslog-ng-ctl stats (as root) inside pipeline for counters. Monitor disk use on the data volume (logs/log_archive grows with cluster size and retention). Pipeline process control uses podman-compose -p hpcperfstats (logs / ps / stop / exec) — supervisorctl is not configured. Troubleshooting: if packets reach the host but nothing is logged, confirm syslog-ng is actually running, check firewall rules, that traffic targets the published 514 on the host running pipeline, allow_from includes the sender’s IPv4 address, and (for filenames) that forwarders preserve a sensible hostname/FQDN.

5. Application config:

   cp hpcperfstats.ini.example hpcperfstats.ini

   In hpcperfstats.ini under [DEFAULT] (install-required and site-wide):

   • machine — cluster name
   • host_name_ext — FQDN of the cluster
   • server — FQDN of the host running the containers
   • restricted_queue_keywords - queues you want to filter out and prevent jobs in them from being displayed
   • staff_email_domain - the email domain of the institution/organization so authorized staff can see all jobs
   • timezone - your machine's local timezone
   • total_cores - CPU budget for app parallelism (omit to use code default 40; see docs/DEPLOY_CONCURRENCY_AND_NUMA.md)
   • secret_key - a random string
   • PostgreSQL connection: engine_name, dbname, username, password, host, port (Compose uses host=db in the image-built ini)

   Optional tuning lives in other sections (see hpcperfstats.ini.example):

   • [PORTAL] — Gunicorn/Django web stack only: gunicorn_workers, parallel_db_prefetch_max, api_small_executor_max_workers, db_conn_max_age, db_statement_timeout_ms, db_idle_in_transaction_timeout_ms, cors_origin_scheme, and development-only separate_test_login (default no)
   • [PIPELINE] — ingest, archive, and metrics: required paths acct_path, archive_dir, daily_archive_dir; optional sync_*, metrics_*, archive_*, metrics_pool_processes, and related keys
   • [RMQ], [OAUTH2], optional [CACHE], [SYSLOG], [XALT] — integration sections unchanged

   For a fresh Docker install you typically edit [DEFAULT] as above; [RMQ] and PostgreSQL defaults in the example are already wired for Compose. Do not change [RMQ] hostnames unless your RabbitMQ layout differs. For cluster syslog, add or edit the optional [SYSLOG] section (see hpcperfstats.ini.example and the compose step above).

   hpcperfstats.ini.example layout: each setting has a one-line # comment directly above it; optional tuning keys appear commented with defaults matching conf_parser. Pipeline/archive behavior (zstd seal, DB-before-append gate sync_archive_require_db_ingest under [PIPELINE], syslog allowlist) is described in the bullets above and in docs/DEPLOY_CONCURRENCY_AND_NUMA.md.

   PostgreSQL container (docker-compose.yaml db service): max_connections is 500 so overlapping Gunicorn workers, threaded API routes (job_plots, home_options, job_detail aux tasks), and pipeline pools rarely hit too many clients. Parallel helpers: max_worker_processes=32, max_parallel_workers=24, max_parallel_workers_per_gather=4, max_parallel_maintenance_workers=2. Memory spikes are still controlled by a lower work_mem, smaller maintenance/autovacuum work mem, temp_buffers, and a slightly lower shared_buffers—see inline comments there. Summary plot aggregate prefetch uses at most two inner threads (see summaryplot.compute_summary_aggregate_prefetch_pool_size) so nested thread pools do not stack against the shared API executor. If legitimate bulk jobs slow down, prefer raising work_mem only during batch windows or increasing the db container mem_limit rather than unconstrained per-query memory.

   For memory-constrained deployments, start with the conservative baseline values documented in hpcperfstats.ini.example, then scale up gradually after observing stable DB checkpoints and container RSS headroom.

   pipeline memory cap (docker-compose.yaml): on hosts with ~192 GiB RAM and no swap, set mem_limit: 128g and memswap_limit: 128g on the pipeline service (defaults in base compose; override in docker-compose.settings.yaml if needed) so ingest spikes cgroup-OOM inside the container before starving db/web. stop_grace_period defaults to 30s for web and pipeline (HPCPERFSTATS_WEB_STOP_GRACE / HPCPERFSTATS_PIPELINE_STOP_GRACE); short cutovers may SIGKILL before full sync_timedb drain. Pair with [PIPELINE] RSS knobs documented in docs/DEPLOY_CONCURRENCY_AND_NUMA.md § OOM. Recreating after limit changes: docs/upgrade.md.

   Python interpreters (image): web/gunicorn and helpers use GIL /opt/python3.14 via /usr/local/bin/python3 / gunicorn (built on debian:trixie, not Hub python:*). Pipeline daemons listend, sync_timedb, and update_metrics are baked onto free-threaded /opt/python3.14t/bin/python (no INI toggle). Image jemalloc is force-linked and preloaded (LD_PRELOAD + /etc/ld.so.preload) so CPython and manylinux wheels share it; gunicorn keeps MALLOC_CONF=background_thread:false. Stdlib zlib and other image-built natives link zlib-ng under /opt/zlib-ng (ZLIB_COMPAT; direct rpath link, not apt zlib1g). Image zstd CLI and CPython _zstd / compression.zstd use zstd 1.5.7 under /opt/zstd (CLI gzip/zlib support linked to zlib-ng; symlinked into /usr/local/bin and /usr/bin; not apt zstd). docker compose exec pipeline python3 stays GIL for operator one-liners; greppable startup lines python_abi executable=… Py_GIL_DISABLED=… prove the live daemon ABI. sync_timedb ingest/append/populate run as in-process threads; durable queues are the job-store sidecar. Production images install Intel MKL from PyPI and source-compile numpy/numexpr/pandas against it for both ABIs (not a host pip step); full image rebuilds take longer than wheel-only installs.

   RabbitMQ memory cap (docker-compose.yaml rabbitmq service): default vm_memory_high_watermark is 40% of detected host RAM, which can OOM the box under thousands of monitor publishers. Compose sets mem_limit: 96g / memswap_limit: 96g and mounts services-conf/rabbitmq_vm_memory.conf (vm_memory_high_watermark.absolute = 80GiB) so publishers block ~16 GiB below the cgroup hard wall (avoids Erlang binary_alloc at the limit). Compose also sets ERL_FLAGS=+MBas aobf +MBlmbcs 512 +MHlmbcs 512 (address-order best-fit binary allocator and 512 KiB largest multiblock carriers) to reduce Erlang fragmentation under many publishers. Console logging is warning (not info) with connection/channel at error; crash dumps are off (ERL_CRASH_DUMP_SECONDS=0, ulimits.core: 0). stop_grace_period: 10m gives the broker time for orderly shutdown (quorum/mnesia) before SIGKILL. On hosts with less than 96 GiB RAM, lower both mem_limit/memswap_limit and the absolute watermark together (then recreate as in docs/upgrade.md). Recovery / preserve-extract: docs/OPERATOR_RABBITMQ_RECOVERY.md.

6. Supervisord and rsync:

   Tracked services-conf/supervisord.conf is baked into the image — do not copy a supervisord .example. The rsync_data program always runs rsync_data_wrapper.sh, which prefers rsync_data.sh if present, otherwise rsync_data.sh.example.

   Both scripts ship with a top-of-file guard (sleep 43200, echo "rsync not yet configured", exit) so default deploys idle for 12 hours and never SSH/rsync to remote hosts. To enable site rsync: edit rsync_data.sh (the wrapper-preferred file), remove those three guard lines, put your rsync commands in the while true loop, and put sleep 43200 at the end of the loop so a configured site does not tight-loop. Ensure SSH keys are configured in compose as documented for the pipeline service.

7. Web server (nginx):

   Committed services-conf/nginx.conf uses fixed in-container TLS paths (/etc/ssl/hpcperfstats/fullchain.pem and privkey.pem). Cert PEMs are materialized at container start from the proxy_ssl_source settings volume (mounted at /mnt/ssl-source) via resolve_proxy_ssl_certs_dir.py in proxy_entrypoint.sh. There is no Compose ssl_certs volume, no build-time host / bind, no additional_contexts / manual resolve step, and no production .env. Do not edit TLS paths in nginx — set proxy_ssl_source.device in docker-compose.settings.yaml (and for Let's Encrypt, optional HPCPERFSTATS_SSL_CERTS_REL in settings).

   After Let's Encrypt renew (or changing the TLS source path): restart proxy only — docker compose restart proxy. Never point production proxy_ssl_source.device at tests/fixtures/proxy-ssl. Image rebuilds that leave proxy running: docs/upgrade.md.

   Compose bind-mounts ./services-conf/nginx-main.conf to /etc/nginx/nginx.conf (process/http tunables: worker_processes auto, affinity, sendfile/tcp_*, open_file_cache) and ./services-conf/nginx.conf to /etc/nginx/http.d/default.conf on proxy, and bind-mounts the shared snippets (nginx-static-files.conf, nginx-django-proxy-common.inc, nginx-compress-proxy.inc, nginx-compress-static.inc, nginx-edge-security-headers.inc, nginx-csp-no-active.inc, nginx-csp-django-html.inc) as the only runtime source for those snippets (they are not baked into proxy.Dockerfile).

   The proxy image is source-built from services-conf/proxy.Dockerfile: pinned nginx 1.31.5, jemalloc, zlib-ng (ZLIB_COMPAT), OpenSSL 3.5.x, ngx_brotli, and zstd 1.5.7 (static libzstd.a + GetPageSpeed zstd-nginx-module), with -march=native -mtune=native on every source-built lib (including OpenSSL --with-openssl-opt). Build the proxy image on the production host (native flags are not portable across CPU generations). Do not rely on Alpine edge apk nginx packages. The image **COPY**s nginx-main.conf and cps nginx.conf into default.conf at build time for a non-Compose baseline, generates hps-proxy-allowed-hosts.inc from [DEFAULT] server=, and ships proxy_entrypoint.sh plus the TLS/resolver helpers. Compose still replaces nginx.conf / default.conf with the host mounts. Runtime proxy_entrypoint.sh materializes TLS PEMs from /mnt/ssl-source, regenerates the OCSP resolver include from container /etc/resolv.conf, waits for SPA HTML under /srv/static/frontend/{machine,pub}/index.html, and may write private diagnostic CSP includes under /etc/nginx/ (never under /srv/static). SPA shells carry their own hash CSP via HTML <meta http-equiv="Content-Security-Policy"> (written at frontend export / SPA heal); nginx /machine/ and /pub/ locations must not send a competing hash CSP header (open_file_cache off on those locations so SPA heal is not stale). HTTP GETs for *.inc (and direct *.br / *.gz / *.zst sidecar URLs) under /static/ return 404. Nginx compresses Gunicorn and SPA HTML on the fly (zstd, then Brotli, then gzip) and serves precompressed .br/.gz siblings for hashed /static/frontend/_next/ files (1y cache); unhashed /static/ stays 30d. Nginx is the public authority for HSTS, framing, COOP, Permissions-Policy, Referrer-Policy, and CSP (hash-based for SPA shells; no-active for JSON/redirects). Certificates without an AIA OCSP URL will not staple; that must not take the site offline. Hostnames come from [DEFAULT] server= in hpcperfstats.ini (preferred in the build context, else hpcperfstats.ini.example): parse_hpcperfstats_proxy_hosts.py emits /etc/nginx/hps-proxy-allowed-hosts.inc, which the main config includes for server_name. Requests whose Host header does not match receive 404 on port 80; on port 443, unknown names get TLS handshake rejection (ssl_reject_handshake). Restart proxy after changing server=, TLS cert PEMs, or nginx.conf. No docker-compose.yaml edits are required for TLS paths or hostnames (TLS paths are fixed; certs via proxy_ssl_source settings volume + entrypoint materialization).

   Static/media routing is split into a reusable include mounted at services-conf/nginx-static-files.conf; nginx serves /static/ and /media/ directly, shells the SPA under /machine/ and /pub/, and proxies only an explicit Django URL prefix list (shared proxy_* directives in services-conf/nginx-django-proxy-common.inc); every other path gets 404 from nginx. When you add a new top-level Django route, extend the allowlist in nginx-static-files.conf and keep it aligned with Django’s root urlpatterns.

   Production: browsers must load /static/* through the proxy service (ports 80/443); nginx reads the staticfiles_ram tmpfs volume at /srv/static, published from disk STATIC_ROOT (staticfiles_data) on web startup after collectstatic, SPA heal, and sidecar compress. /media/ is the same pattern: disk media_data stays MEDIA_ROOT staging; nginx reads media_ram at /srv/media (empty media is valid). Hitting web:8000 directly is not a supported way to load hashed SPA assets (Gunicorn does not implement /static/ URL serving). For local parity with that layout, use full compose including proxy, or run manage.py runserver --nostatic and still obtain /static/ via nginx rather than Django’s dev static handler. The proxy container is built from services-conf/proxy.Dockerfile and enables hybrid compression (on-the-fly zstd/Brotli/gzip for proxied and SPA HTML; Brotli/Gzip sidecars for hashed static files).

8. Build and start:

   podman-compose -p hpcperfstats up --build -d

   View logs (docker-compose.yaml uses the json-file logging driver with max-size: 100m and max-file: 3 so stdout stays available to Compose on Docker and Podman and does not flood host syslog/journald):

   podman-compose -p hpcperfstats logs

   Rootless development uses unprivileged host ports without changing production defaults:

   HPCPERFSTATS_HTTP_PORT=8080 HPCPERFSTATS_HTTPS_PORT=8443 \
     HPCPERFSTATS_SYSLOG_PORT=1514 \
     podman-compose -p hpcperfstats-dev up --build -d

   These variables are a development launch override only. Production site configuration remains in hpcperfstats.ini and docker-compose.settings.yaml, never a required .env.

   On first startup, the web container runs Django migrations (manage.py migrate only — schema changes ship as reviewed, committed migration files; production startup never runs makemigrations) and collectstatic --noinput --clear so disk STATIC_ROOT (staging for the tmpfs nginx serves as /static/) is emptied of unused leftovers then populated before Gunicorn starts. Collectstatic omits *.map source maps. After collectstatic, startup verifies SPA shells under STATIC_ROOT/frontend/{machine,pub}/index.html. If the package image lacks the shells, web fail-closes. Volume fingerprint heal after a later image rebuild is documented in docs/upgrade.md. After heal, startup writes Brotli-11 / Gzip-9 sidecars beside compressible static files (hashed Next chunks, Django/DRF admin assets), then always publishes STATIC_ROOT and MEDIA_ROOT onto shared tmpfs (staticfiles_ram / media_ram) that proxy mounts at /srv/static and /srv/media. Existing stacks pick this up on the next web restart; keep the host mkdir for /data/hpcperfstats_site/staticfiles and /data/hpcperfstats_site/media. Direct *.br / *.gz URLs stay 404 at nginx.

   The compose DB service includes explicit PostgreSQL checkpoint/memory tuning (max_connections, shared_buffers, work_mem, maintenance_work_mem, autovacuum_work_mem, checkpoint_*, min_wal_size, max_wal_size, and parallel-worker caps) plus shm_size. Keep these aligned with host RAM and service memory limits; tune upward one notch at a time only after confirming checkpoint stability and no OOM events. The pipeline daemons (listend, sync_timedb, and update_metrics) use in-process threads and ordinary Python objects, so the pipeline service does not reserve a separate shm_size for worker IPC. Do not change db shm_size: "16gb".

---

Useful commands

Task	Command
Build and start container stack	podman-compose -p hpcperfstats up --build -d
Stop and remove containers	podman-compose -p hpcperfstats down
Existing-stack rebuilds / Redis / PG18 / INI	docs/upgrade.md
Restart proxy after cert renew or server= change	docker compose restart proxy
View logs	podman-compose -p hpcperfstats logs
PostgreSQL shell	docker compose exec db psql -h localhost -U hpcperfstats
Pipeline shell (data/processing)	docker compose exec pipeline su hpcperfstats
Get queues and message counts from rabbitmq	docker compose exec rabbitmq rabbitmqctl list_queues name messages consumers
RabbitMQ default queue type	Compose mounts services-conf/rabbitmq_default_queue_type.conf (default_queue_type = quorum). New durable monitor ingest queues are declared quorum. Existing brokers: docs/upgrade.md.
RabbitMQ memory cap	Compose mem_limit / memswap_limit 96g plus services-conf/rabbitmq_vm_memory.conf (vm_memory_high_watermark.absolute = 80GiB headroom). Erlang allocator: ERL_FLAGS=+MBas aobf +MBlmbcs 512 +MHlmbcs 512. Logging: console warning (not info); ERL_CRASH_DUMP_SECONDS=0. Inspect: docker compose exec rabbitmq rabbitmqctl status (Alarms + watermark; do not use rabbitmqctl list_alarms — absent on 4.3.x). Recovery: docs/OPERATOR_RABBITMQ_RECOVERY.md.
RabbitMQ memory scream	Pipeline supervisord rabbitmq-watcher polls management every 5 min; lines start with [rabbitmq-watcher]; literal ERROR at 40 GiB and every +10 GiB. Grep: docker compose logs pipeline 2>&1 | grep '\[rabbitmq-watcher\]'. Recreate pipeline after deploy.
RabbitMQ frame_max	Compose mounts services-conf/rabbitmq_frame_max.conf (frame_max = 131072, RabbitMQ default) so listend AMQP frames match pika LISTEND_AMQP_FRAME_MAX. Recreate rabbitmq and pipeline after changing that file (docs/upgrade.md).
Admin Monitor RabbitMQ stats	Staff Admin Monitor → RabbitMQ statistics uses the management HTTP API on compose-internal http://rabbitmq:15672 (image rabbitmq:*-management-alpine). Port 15672 is not published on the host; loopback_users.guest = false in services-conf/rabbitmq_management.conf allows web→rabbitmq auth.

---

Publications

• Comprehensive Resource Use Monitoring for HPC Systems with TACC Stats
• Understanding application and system performance through system-wide monitoring
• 

---

Developers and maintainers

• Amit Ruhela — aruhela@tacc.utexas.edu
• Stephen Lien Harrell — sharrell@tacc.utexas.edu
• Sangamithra Goutham — sgoutham@tacc.utexas.edu
• Chris Ramos — cramos@tacc.utexas.edu

Developer emeritus

John Hammond · R. Todd Evans · Bill Barth · Albert Lu · Junjie Li · John McCalpin

---

Copyright and license

Copyright (c) 2011 University of Texas at Austin

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.