Before you can say "my change made it 8% faster" you must know how much
the SAME code varies run-to-run on THIS machine, and why. This lab runs a
fixed, memory-free, CPU-bound workload 30 times and reports the spread.
Three things dominate the spread on a laptop:
1. WHICH core you land on. This CPU is hybrid: P-cores (CPUs 0-7, two
SMT threads per core) and E-cores (CPUs 8-15). The same loop is
~1.9x slower on an E-core. The scheduler moves you between them.
2. FREQUENCY. The governor is `powersave`: the clock ramps up during the
first reps. (Turbo is disabled on this laptop, no_turbo=1, which
removes one more source of variance that most machines have.)
3. INTERFERENCE: interrupts, other processes, the SMT sibling thread.
evidence to leave with
Your own min/p50/p99 unpinned vs P-core vs E-core; cycles/iteration from perf stat matches the dependent-chain latency
first, get the code
run this
$ git clone https://github.com/adit4443ya/debug_labs.git && cd debug_labs
then build it
run this
$ cd perf && ./build.sh
produces perf/bin/01_noise_floorwalk it, one step at a time
run this
$ ./bin/01_noise_floor
what it printed on the machine that wrote this lab
work(10000000) x 30 reps | started on CPU 15 | 5 reps changed CPU mid-run
first 5 reps (us): 68362 68982 48433 28946 33327 <- watch the frequency ramp
label min p50 p90 p99 p99.9 max mean (us)
dependent_chain 28855 28978 48433 68982 68982 68982 33383.5
spread max/min = 2.39x p99/p50 = 2.38x ns/iter at min = 2.89
It started on CPU 15 (an E-core), changed CPU five times, and the same
work took anywhere from 28.9 ms to 69.0 ms. A 2.4x spread with NO code
change. Any "optimization" smaller than that is invisible here.
run this
$ ../tools/pin.sh ./bin/01_noise_floor
what it printed on the machine that wrote this lab
── pin.sh ──────────────────────────────────────────────
hybrid CPU: P-cores=0-7 E-cores=8-15
pinning to CPU 2 (SMT siblings: 2-3) governor=powersave no_turbo=1 smt=on
note: governor 'powersave' lets the clock ramp during the run; the first samples
are slower. For stable numbers use a warm-up rep, or (root) run:
sudo cpupower frequency-set -g performance
───────────────────────────────────────────────────────
work(10000000) x 30 reps | started on CPU 2 | 0 reps changed CPU mid-run
first 5 reps (us): 64587 48473 28921 29022 29322 <- watch the frequency ramp
label min p50 p90 p99 p99.9 max mean (us)
dependent_chain 28767 28940 29052 64587 64587 64587 30767.8
spread max/min = 2.25x p99/p50 = 2.23x ns/iter at min = 2.88
No migrations. p90 is now within 1% of min: the body of the distribution
is tight. The max is still 2.2x — that is rep 1 and 2, the frequency ramp.
Discard them (or warm up first) and the spread is ~1%.
run this
$ ../tools/pin.sh -c 8 ./bin/01_noise_floor
what it printed on the machine that wrote this lab
pinning to CPU 8 (SMT siblings: 8) governor=powersave no_turbo=1 smt=on
work(10000000) x 30 reps | started on CPU 8 | 0 reps changed CPU mid-run
first 5 reps (us): 78263 65302 54474 54026 53944 <- watch the frequency ramp
label min p50 p90 p99 p99.9 max mean (us)
dependent_chain 53823 53905 54385 78263 78263 78263 55144.6
spread max/min = 1.45x p99/p50 = 1.45x ns/iter at min = 5.38
1.87x slower than the P-core, and perfectly stable. If run A happened to
land here and run B on a P-core, B would look like a 1.87x "speedup".
run this
$ ../tools/pin.sh -q perf stat -e cycles:u,instructions:u ./bin/01_noise_floor 10000000 5
what it printed on the machine that wrote this lab
...
Performance counter stats for './bin/01_noise_floor 10000000 5':
<not counted> cpu_atom/cycles/u (0.00%)
300,648,113 cpu_core/cycles/u
<not counted> cpu_atom/instructions/u (0.00%)
275,157,204 cpu_core/instructions/u # 0.92 insn per cycle
0.183228871 seconds time elapsed
Read it: perf lists BOTH PMUs. Pinned to a P-core, the cpu_atom line reads
"<not counted>" (the E-core PMU never saw the process). Run it unpinned and
both lines fill in with partial counts — that is the migration you saw in
STEP 1, made visible.
Now do the arithmetic, because this is the skill: 300.6M cycles / 5 reps /
10M iterations = 6.0 cycles per iteration. The chain is imul (3 cycles) +
add (1) + shift (1) + xor (1) = 6 cycles of dependent latency. The counter
matches the microarchitecture exactly; the loop is latency-bound, and
0.92 IPC means the core spent most of each cycle waiting for the previous
iteration. And 60.1M cycles in 28.8 ms means the core ran at ~2.09 GHz
during the fast reps: that is the clock powersave gave us, no turbo.
the workload
a dependent multiply-xorshift chain (each iteration needs the
previous result), so it measures core latency, not memory or the vector
unit. Deterministic: same answer every run.
how to read the table
min = the least-disturbed run. For a CPU-bound loop this is the most
reproducible statistic and the one to compare A vs B on.
p50 = typical. p99/max = the disturbances (frequency ramp, migration,
interrupts). "spread" = max/min: on a quiet pinned P-core after
warm-up it is ~1%; unpinned it can be >2x.
Rule: if the difference between A and B is smaller than the spread of A
alone, you have not measured anything yet.
THE RULES THIS LAB ESTABLISHES (every later lab assumes them):
R1. Pin (tools/pin.sh). Prefer a P-core. Never compare P-core vs E-core
numbers to each other.
R2. Warm up: discard the first reps, or run long enough that the ramp is
a small fraction of the total.
R3. Repeat and report min/p50/p99, never a single number or a mean.
R4. Same binary, same input, same core for A and B. Interleave A/B runs
(ABAB) so slow drift (thermal) affects both.
R5. Count instructions AND cycles with perf stat: a change that cuts
instructions but not cycles is not a speedup; one that cuts cycles
but not instructions changed the microarchitectural behaviour.
WHAT NEEDS ROOT (you can't do these here without sudo; production
low-latency boxes do all of them):
sudo cpupower frequency-set -g performance # no ramp
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo # no turbo
kernel cmdline isolcpus= nohz_full= rcu_nocbs= # isolate cores from the scheduler
IRQ affinity (/proc/irq/N/smp_affinity) # keep interrupts off the core
take it further
1. Run STEP 2 with the SMT sibling busy: `taskset -c 3 ./bin/01_noise_floor
100000000 1000 &` then rerun on CPU 2. Watch p50 move even though the
other process never touches your data (shared execution units).
2. Run with 1 rep, 10 times in a shell loop. Compare to 10 reps once.
3. Change the chain to two independent chains and observe the time drop
while `instructions` roughly doubles: instruction-level parallelism.
Predict cycles/iteration first (hint: it is not 12).
============================================================
read the source · perf/01_noise_floor.c
#include"../tools/bench.h"/* A dependent chain: iteration i+1 cannot start before iteration i finishes. */staticuint64_twork(uint64_t n){uint64_t x =1;for(uint64_t i =0; i < n; i++){ x = x *6364136223846793005ull+1442695040888963407ull; x ^= x >>29;}return x;}intmain(int argc,char**argv){uint64_t n = argc >1?strtoull(argv[1],0,10):10000000ull;/* ~29 ms on a P-core */int reps = argc >2?atoi(argv[2]):30;uint64_t*s =(uint64_t*)malloc((size_t)reps *sizeof*s);int start_cpu =bench_cpu(), migrated =0;for(int r =0; r < reps; r++){int c =bench_cpu();uint64_t t0 =bench_ns();uint64_t x =work(n);BENCH_KEEP(x);/* never let the loop be deleted */ s[r]=(bench_ns()- t0)/1000;/* microseconds */if(bench_cpu()!= c) migrated++;}printf("work(%llu) x %d reps | started on CPU %d | %d reps changed CPU mid-run\n",(unsignedlonglong)n, reps, start_cpu, migrated);printf("first 5 reps (us): %llu %llu %llu %llu %llu <- watch the frequency ramp\n",(unsignedlonglong)s[0],(unsignedlonglong)s[1],(unsignedlonglong)s[2],(unsignedlonglong)s[3],(unsignedlonglong)s[4]); bench_stats st =bench_compute(s,(uint64_t)reps);bench_print_header("us");bench_print("dependent_chain",&st);printf("spread max/min = %.2fx p99/p50 = %.2fx ns/iter at min = %.2f\n",(double)st.max /(double)st.min,(double)st.p99 /(double)st.p50,(double)st.min *1000.0/(double)n);free(s);return0;}
◇
Finish the lab: write the evidence note
A lab is done when you can say what it showed on your machine — not when you have read it.
Which role are you preparing for?
It decides which problems come first, which drills open, which labs you see and how the Learn surface is organised. Change it any time from the switch in the top bar, or press Escape to look around first.