DEBUGGING LAB #02 — Crash reproducers, and shrinking a 407-line crash to 3 lines

TRACK: debugging | DIFFICULTY: 3 | NEEDS: mlir-opt, llc, llvm-reduce FILES: interesting_cannot_select.sh

What this teaches

When a compiler fails you need two things before anyone can help, you included: a reproducer (one command, one file, fails every time) and a minimal reproducer (small enough to understand). MLIR and LLVM each generate the first automatically, and llvm-reduce / mlir-reduce produce the second. This is ~/debug_lab/passwork lab 03, applied to MLIR-generated code.

Part A — A pass pipeline that FAILS: --mlir-pass-pipeline-crash-reproducer

Take the invalid transform script from lab 03_linalg/03 (reuse of a consumed handle) and run it inside a longer pipeline:

$ sed 's/fuse_into_containing_op %fill into %l1/fuse_into_containing_op %fill into %relu/' ../03_linalg/03_transform_dialect.mlir > out/bad_schedule.mlir && mlir-opt out/bad_schedule.mlir --pass-pipeline='builtin.module(canonicalize, transform-interpreter, cse)' --mlir-pass-pipeline-crash-reproducer=out/repro.mlir -o /dev/null 2>&1 | grep -E 'error|reproducer' | cut -c1-140
out/bad_schedule.mlir:125:20: error: op uses a handle invalidated by a previously executed transform op
out/bad_schedule.mlir:0:0: error: Failures have been detected while processing an MLIR pass pipeline
out/bad_schedule.mlir:0:0: note: Pipeline failed while executing [`InterpreterPass` on 'builtin.module' operation]: reproducer generated at 

The reproducer is the IR as it was when the failing pass started (after canonicalize had already run), with the exact pipeline and options embedded at the bottom:

$ sed -n '/mlir_reproducer/,/verify_each/p' out/repro.mlir
    mlir_reproducer: {
      pipeline: "builtin.module(canonicalize{cse-between-iterations=false    max-iterations=10 max-num-rewrites=-1 region-simplify=normal test-convergence=false top-down=true}, transform-interpreter{ debug-payload-root-tag= disable-expensive-checks=false entry-point=__transform_main}, cse)",
      disable_threading: false,
      verify_each: true
$ mlir-opt out/repro.mlir --run-reproducer -o /dev/null 2>&1 | grep -m1 error | cut -c1-120
out/repro.mlir:24:43: error: op uses a handle invalidated by a previously executed transform op

--run-reproducer replays it: one file you can attach to a bug report or send to a teammate. Their machine doesn't need your Triton kernel, your Python environment or your flags. Add --mlir-pass-pipeline-local-reproducer to make the file contain only the failing pass and the IR it saw.

Triton-based compilers can do the same: TRITON_REPRODUCER_PATH=/tmp/repro.mlir makes Triton write the reproducer for its own MLIR pipelines.

Part B — The BACKEND crashes on IR we generated

$LOWER_SVE_BAD (lab 05_aarch64/02, failure 2) produces LLVM IR the AArch64 backend can't select:

$ mlir-opt ../05_aarch64/out/sve.t.mlir --pass-pipeline="$LOWER_SVE_BAD" | mlir-translate --mlir-to-llvmir -o out/crash.ll && wc -l < out/crash.ll && llc -mtriple=aarch64-linux-gnu -mattr=+sve -O2 out/crash.ll -o /dev/null 2>&1 | head -1 | cut -c1-110
407
LLVM ERROR: Cannot select: 0x26d41450: ch = store<(store (<vscale x 1 x s8>) into %ir.75)> 0x26cb0998, 0x26d47

407 lines, and a message about a DAG node. Reduce it. First write an interestingness test: a script that exits 0 iff the file still shows the same failure:

$ cat interesting_cannot_select.sh
#!/usr/bin/env bash
# Interestingness test for llvm-reduce: exit 0 iff the input STILL crashes the
# AArch64 backend with the SAME error. Matching the message (not just "any
# failure") stops the reducer from drifting to an unrelated crash or a syntax error.
llc -mtriple=aarch64-linux-gnu -mattr=+sve -O2 "$1" -o /dev/null 2>&1 | grep -q "Cannot select.*vscale x 1 x s8"

Two things that took a failed first attempt to learn (keep them):

1. The first version grepped for Cannot select.*nxv4i1.*load, and llvm-reduce said "Input isn't interesting!". Under clang -O2 the crash is on a load; under bare llc it's on a store. Same root cause, different first symptom. Match the root (vscale x 1 x s8: a 1-byte-per-lane predicate memory access), not the first line you saw.
2. Match a message, not just "non-zero exit". Otherwise the reducer happily "reduces" to a file with a syntax error.

$ llvm-reduce --test=./interesting_cannot_select.sh out/crash.ll -o out/crash.reduced.ll 2>&1 | tail -1 && grep -vE '^(;|$|source_filename|target)' out/crash.reduced.ll
Done reducing! Reduced testcase: out/crash.reduced.ll
define void @matmul(ptr %0) {
  store <vscale x 4 x i1> zeroinitializer, ptr %0, align 1
  ret void
}

407 lines → 3. The whole bug: the AArch64 backend can't store an SVE predicate narrower than nxv16i1 (a full svbool) to memory. That's exactly what arm-sve-legalize-vector-storage fixes: it widens predicate stack slots to vector<[16]xi1> and inserts arm_sve.convert_to_svbool / convert_from_svbool:

$ mlir-opt tests/sve_backend_crash.mlir --pass-pipeline='builtin.module(func.func(arm-sve-legalize-vector-storage))' | grep -E 'alloca|svbool'
    %alloca = memref.alloca() alignment = 2 : memref<vector<[16]xi1>>
    %1 = arm_sve.convert_to_svbool %0 : vector<[4]xi1>
    memref.store %1, %alloca[] : memref<vector<[16]xi1>>
    %2 = memref.load %alloca[] : memref<vector<[16]xi1>>
    %3 = arm_sve.convert_from_svbool %2 : vector<[4]xi1>

Root cause: the pass ran before the pass that created the slots. From a backend crash to a one-line pipeline fix, with each step proven.

mlir-reduce — the MLIR-level equivalent

mlir-reduce IN.mlir --test=SCRIPT --test-arg=... does the same on MLIR (deletes ops/regions while the script says "still interesting"). Use it when the failure is inside an MLIR pass and the input is a huge Triton or torch-mlir module. The reduced .mlir becomes the regression test (lab 03).

Try this yourself

1. Generate a local reproducer for Part A (--mlir-pass-pipeline-local-reproducer) and diff its embedded pipeline against the full one.
2. Break interesting_cannot_select.sh on purpose (grep for a message that never appears) and watch llvm-reduce refuse to start. Then make it match any failure (! llc ...) and see what llvm-reduce produces instead.
3. Write an interestingness test for lab 05_aarch64/02 failure 3 (the segfault). What makes that one harder to reduce than a compile-time crash?