Understanding Machine CPU Usage

Luke Hebert

- 2 minutes read - 367 words

High CPU load is one of the most common root causes of performance issues on a server. If you are already running Prometheus with the Node exporter, you have all the data you need to figure out exactly where your CPU time is going. This post walks through the key metric, how to turn raw counters into meaningful rates, and how to calculate a clean “CPU used %” figure you can alert on or dash off to a dashboard.

The node_cpu_seconds_total metric

On a Node exporter metrics page, you will find output like the following:

# HELP node_cpu Seconds the cpus spent in each mode.
# TYPE node_cpu counter
node_cpu_seconds_total{cpu="0",mode="guest"} 0
node_cpu_seconds_total{cpu="0",mode="idle"} 2.03442237e+06
node_cpu_seconds_total{cpu="0",mode="iowait"} 3522.37
node_cpu_seconds_total{cpu="0",mode="irq"} 0.48
node_cpu_seconds_total{cpu="0",mode="nice"} 515.56
node_cpu_seconds_total{cpu="0",mode="softirq"} 953.06
node_cpu_seconds_total{cpu="0",mode="steal"} 0
node_cpu_seconds_total{cpu="0",mode="system"} 6605.46
node_cpu_seconds_total{cpu="0",mode="user"} 23343.01
node_cpu_seconds_total{cpu="1",mode="guest"} 0
node_cpu_seconds_total{cpu="1",mode="idle"} 2.03471439e+06
node_cpu_seconds_total{cpu="1",mode="iowait"} 3633.5
node_cpu_seconds_total{cpu="1",mode="irq"} 0.58
node_cpu_seconds_total{cpu="1",mode="nice"} 542.05
node_cpu_seconds_total{cpu="1",mode="softirq"} 880.49
node_cpu_seconds_total{cpu="1",mode="steal"} 0
node_cpu_seconds_total{cpu="1",mode="system"} 6581.92
node_cpu_seconds_total{cpu="1",mode="user"} 23171.06

This metric comes from /proc/stat and tells you how many seconds each CPU core has spent in each mode since boot:

  • user: Time spent executing in userland
  • system: Time spent in the kernel
  • iowait: Time spent waiting for I/O to complete
  • idle: Time the CPU had nothing to do
  • irq / softirq: Time servicing hardware and software interrupts
  • guest: CPU time consumed by guest VMs (if you are running virtual machines)
  • steal: Time “stolen” by the hypervisor when you are a VM

These modes are mutually exclusive, which makes interpretation straightforward. A high iowait value points to a disk or network bottleneck, while high user or system values indicate the workload is CPU bound.

Calculating per-second rates

Because these are counters, you need to convert them into per-second values using the rate function in the Prometheus expression browser:

rate(node_cpu_seconds_total{job="node"}[1m])

To get an overall picture across all CPUs on a machine, aggregate by mode and instance:

sum by (mode, instance) (rate(node_cpu_seconds_total{job="node"}[1m]))

Since the values for each CPU always sum to one second per second, the per-second rates are also the ratios of usage. You can use that to calculate the percentage of CPU in use by subtracting idle from 100%:

100 - (avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[1m])) * 100)

CPU Used % across several machines

Reference