Reduce a reproducer, then pin it with a test

WHAT THIS TEACHES

Bug reports arrive as forty thousand lines. Nobody debugs that. You reduce it to something a maintainer can read in a minute, then write a test so it cannot come back.

The reducing is mechanical — llvm-reduce does it. The interestingness test is the part people get wrong, and this lab is mostly about getting it wrong twice on purpose, because that is how you learn what it has to assert.

BUILD

Nothing to build. 03_reduce_and_pin.ll is one interesting function buried in five noise functions, standing in for the large file.

STEP 1 — ATTEMPT ONE: THE OBVIOUS TEST, AND WHY IT DESTROYS EVERYTHING

#!/bin/sh
grep -q "@the_one" "$1"          # "is my function still mentioned?"

$ llvm-reduce --test=./interesting-loose.sh 03_reduce_and_pin.ll -o loose.ll
$ cat loose.ll
declare i32 @the_one()

A bare declaration. The reducer did exactly what you asked — it kept the name and threw away everything else, because a declaration satisfies grep. This is the single most common way a reduction wastes an afternoon.

The rule: the test must assert the BUG still reproduces, not that the code still looks familiar.

STEP 2 — ATTEMPT TWO: BETTER, STILL WRONG

#!/bin/sh
opt -passes=gvn -S "$1" -o - 2>/dev/null \
  | sed -n '/@the_one/,/^}/p' | grep -q "add i32 %a, %a"

$ llvm-reduce --test=./interesting-good.sh 03_reduce_and_pin.ll -o good.ll
$ cat good.ll
define i32 @the_one(i32 %a) {
entry:
  %sum = add i32 %a, %a
  ret i32 %sum
}

Now it runs the pass and checks the output — much better, and still wrong. There are no loads left. The reducer found a program whose output has the shape we asked for without GVN doing anything at all: the input already said add i32 %a, %a. We asserted a shape, not a transformation.

STEP 3 — ATTEMPT THREE: ASSERT THAT WORK HAPPENED

#!/bin/sh
before=$(grep -c "load i32" "$1")
[ "$before" -ge 2 ] || exit 1
after=$(opt -passes=gvn -S "$1" -o - 2>/dev/null | grep -c "load i32")
[ "$after" -lt "$before" ]

$ llvm-reduce --test=./interesting-correct.sh 03_reduce_and_pin.ll -o correct.ll
$ cat correct.ll
define i32 @the_one() {
entry:
  %a = load i32, ptr null, align 4
  %b = load i32, ptr null, align 4
  ret i32 %b
}

Four lines, and it still exercises the thing we care about: two loads go in, one comes out. This is what you attach to a bug report.

Note ptr null — the reducer replaced the parameter with the simplest value that keeps the property. That is fine here because nothing dereferences it at run time. When your property is a crash, it is not fine, and the test must run the program rather than inspect its IR.

STEP 4 — PIN IT SO IT CANNOT COME BACK

A reproducer that is not in the test suite will be reintroduced. FileCheck is how LLVM pins a transformation:

; RUN: opt -passes=gvn -S < %s | FileCheck %s

define i32 @the_one(ptr %p) {
; CHECK-LABEL: @the_one
; CHECK: %a = load i32, ptr %p
; CHECK-NOT: load
; CHECK: ret
entry:
  %a = load i32, ptr %p
  %b = load i32, ptr %p
  %sum = add i32 %a, %b
  ret i32 %sum
}

$ opt -passes=gvn -S < 04_filecheck_test.ll | FileCheck 04_filecheck_test.ll && echo PASS
PASS

CHECK-NOT: load between two CHECKs is the load-bearing line: it asserts there is no second load between the first one and the return. A test that only checks the first load would pass even if GVN did nothing.

STEP 5 — PROVE THE TEST CAN FAIL

A test you have never seen fail is not a test.

$ opt -passes=gvn -S < 04_filecheck_test.ll | FileCheck 04_filecheck_test.ll && echo PASS
PASS
$ opt -S < 04_filecheck_test.ll | FileCheck 04_filecheck_test.ll && echo PASS || echo FAIL
FAIL

Without -passes=gvn the second load survives, CHECK-NOT fires, and the test fails. Do this every time you write one.

WHAT TO SAY IN AN INTERVIEW

"I reduce with llvm-reduce against an interestingness script, and the script has to assert the transformation, not the shape of the output — otherwise the reducer happily hands you a program that satisfies your grep and reproduces nothing. Then I pin it with a lit/FileCheck test and confirm the test fails without the fix."

That answer contains the two things people miss: what makes a reduction script correct, and that a test must be seen failing.

TRY THIS YOURSELF

1. Write an interestingness test for a crash — opt … ; [ $? -ne 0 ] — and reduce something that asserts. Notice you must distinguish the crash you want from every other crash, or you will reduce to unrelated malformed IR.
2. Add ; CHECK-SAME and ; CHECK-DAG variants and find out what each buys.
3. Run llvm-reduce with --delta-passes= to limit which reductions it tries, and watch the runtime change on a large input.