Totals for the table below
Metric definitions
| Term | Plain meaning | Formula |
|---|---|---|
| Avg Utilization % | Utilized GPU-h as a share of Held GPU-h. Shown with its full split: the Utilized, Under-Utilized, and Non-Utilized shares of Held sum to 100%. Higher Utilized % is better. Lower Non-Utilized % is better. Under-Utilized is not decisively good or bad, but may suggest potential for optimizing GPU efficiency. | Utilized ÷ Held; Utilized + Under-Utilized + Non-Utilized = Held |
| Failed % | Share of Held GPU-h on hard-failed jobs: crashes, SGE setup errors, and mid-run kills including qdel and OOM. Excluded despite SGE flagging them failed: interactive-session closes (QRLOGIN / qlogin / ood-* jobs ending exit 137/143, the session was closed or expired) and wall-kills (runtime reached the requested h_rt, see Wall-Kill %). All of a failed job’s held time counts; one failed array task flags the whole job. | hard-failed Held-h ÷ Held |
| Free GPU-h | Powered on with no job. Unreserved and available. | Σ unassigned samples × 5 min |
| Held % | Share (%) of Uptime GPU-h the GPU(s) were occupied by jobs. | Held ÷ Uptime |
| Held GPU-h | Total time (hours) a GPU was held by a job, and therefore attributable to a user and project. Splits into Utilized GPU-h, Under-Utilized GPU-h, and Non-Utilized GPU-h. | Σ samples with a job × 5 min |
| Jobs # | Distinct GPU jobs behind this row; array sub-tasks collapse to their parent job, dependency jobs count separately | distinct job numbers |
| kWh | Energy from sampled GPU power draw | Σ power × 5 min |
| Max GPUs | Most GPUs a user held at the same instant in the period (not additive); user rows only | peak concurrent GPUs |
| Median Wait | Median queue wait (qsub → start) over this row’s jobs; later array tasks excluded | median of qsub→start |
| Under-Utilized GPU-h | Held time where a process sat on the GPU but kernels were not executing. Each sample contributes its unused share of 5 minutes: a sample reading 30% Utilization adds 3.5 minutes here and 1.5 to Utilized GPU-h. Samples with no process at all count in Non-Utilized GPU-h instead. Typical causes: data loading, I/O, checkpointing, partial kernels. Some is unavoidable. | Held − Utilized − Non-Utilized |
| Non-Utilized GPU-h | Held time with no process on the GPU at all. Each 5-minute sample records which processes hold the card; a sample with a job but no process counts its full 5 minutes here. A process sitting at 0% Utilization does not count here; that time is Under-Utilized GPU-h. The worst waste: a GPU held and doing literally nothing, usually an idle OnDemand desktop/notebook or a CPU-bound job squatting on a GPU node. | Σ no-process samples × 5 min |
| Pool Share % | This row’s Held GPU-h as a share of all powered-on GPU time in the pool, held or free. Every grouping answers the same question, this row’s share of the whole pool, so the top-level rows of any grouping sum to the pool’s Held %. A GPU-Type filter narrows the pool to that type. An H200 hour counts like a V100 hour. Rows below 0.05% show <0.1%, never a false 0% | row Held-h ÷ pool Uptime |
| Trend (12mo / 30d) | Held GPU-h over the trailing 12 months or 30 days; bars scale to each row’s own max | Held GPU-h per period |
| Uptime GPU-h | Total powered-on, monitored time (hours) for GPU(s). Splits into Held GPU-h, Free GPU-h. | Σ powered-on samples × 5 min |
| Utilization % | NVIDIA’s nvidia-smi --query-gpu utilization.gpu: “Percent of time over the past sample period (~1s) during which one or more kernels was executing on the GPU.” On the SCC, this reading is recorded on every GPU, every 5 minutes, for all Uptime GPU-h. Each reading stands for its full 5-minute interval. It is a ceiling on true compute use for a period (e.g. a kernel using a fraction of the GPU reads as 100%). Utilized GPU-h, Under-Utilized GPU-h, and Avg Utilization % all derive from it. | per 5-min sample |
| Utilized GPU-h | The sum of every 5-minute sample of Held GPU-h, each scaled by its Utilization %. A ceiling on the hours actually computing, since Utilization % over-reads true compute use. | Σ samples (util% ÷ 100) × 5 min |
| VRAM Full % | Share of Held GPU-h at ≥90% VRAM | ≥90% VRAM samples ÷ Held |
| VRAM Headroom GB | GPU memory left unused even at peak; large headroom flags a candidate to move down a GPU tier | capacity − peak GB |
| VRAM Mean % | Mean GPU memory in use, as a share of the GPU capacity | mean(mem_used ÷ mem_total) |
| VRAM Min-GPU | Smallest GPU whose memory would fit this peak | smallest GPU ≥ peak GB |
| VRAM Peak % | Highest VRAM use reached, as a share of the GPU capacity | max(mem_used ÷ mem_total) |
| VRAM Peak GB | Highest VRAM used, in GB | max VRAM used |
| Wait >1h % | Share of jobs that waited more than 1 hour to start; array tasks excluded | jobs waited >1h ÷ jobs |
| Wait Distribution | Spread of queue wait across buckets: <1m / 1–10m / 10–60m / 1–6h / >6h | qsub→start histogram |
| Wall-Kill % | Share of Held GPU-h on jobs the scheduler killed at their requested h_rt. A soft failure: the job hit its own time limit, typically checkpointed training that resubmits and loses only the work since its last checkpoint. Accounting cannot tell that apart from a job that never finished, so the share stays visible here. Not counted in Failed %. | wall-killed Held-h ÷ Held |
Sums run over gpustats samples taken every 5 min (1 GPU-hour = 12 samples).