DEBUGGING LAB #03 — Pin every fix with a lit/FileCheck test you have watched fail

TRACK: debugging | DIFFICULTY: 2 | NEEDS: llvm-lit, FileCheck, not FILES: tests/lit.cfg.py, tests/*.mlir

What this teaches

Every MLIR project (upstream, IREE, triton-shared, any vendor backend) tests passes the same way: a .mlir file whose // RUN: lines run the tool, and whose // CHECK: lines are patterns FileCheck matches against the output, in order. llvm-lit finds and runs the files. When you fix something, the PR carries a test like this. A test you haven't seen fail proves nothing, so this lab's tests each come with a way to make them fail.

The three tests (each pins a finding from an earlier lab)

STEP 1 — Run them

$ llvm-lit -v tests 2>&1 | grep -E '^(PASS|FAIL)|Passed|Failed'
PASS: mlir_lab :: fusion_no_temp_buffer.mlir (1 of 3)
PASS: mlir_lab :: sve_backend_crash.mlir (2 of 3)
PASS: mlir_lab :: sve_no_alloca_in_loop.mlir (3 of 3)
  Passed: 3 (100.00%)

(lit.cfg.py puts build-full/bin on PATH and defines %lab as the lab root, so tests can refer to lab inputs.)

STEP 2 — Watch one fail (a regressed pipeline, without eliminate-empty-tensors)

$ mlir-opt ../04_optimization/fusion.mlir --pass-pipeline='builtin.module(linalg-fuse-elementwise-ops,one-shot-bufferize{bufferize-function-boundaries function-boundary-type-conversion=identity-layout-map},canonicalize)' | FileCheck tests/fusion_no_temp_buffer.mlir 2>&1 | head -5
tests/fusion_no_temp_buffer.mlir:7:15: error: CHECK-NOT: excluded string found in input
// CHECK-NOT: memref.alloc
              ^
<stdin>:5:11: note: found here
 %alloc = memref.alloc() alignment = 64 : memref<16777216xf32>

FileCheck quotes the check line that failed and the input line that broke it. That's the message a CI bot will post on a PR that reintroduces the 64 MiB temp buffer.

How to write good CHECK lines

• CHECK-LABEL per function. It splits the output into independent chunks, so a failure in one function can't make checks match in another.
• Check the property, not the whole IR. CHECK-NOT: memref.alloc survives unrelated changes; pasting 40 lines of expected IR breaks on every upstream bump. (mlir/utils/generate-test-checks.py generates full checks. Use it, then delete what isn't the point of the test.)
• Capture and reuse values instead of hardcoding SSA names: outs(%[[OUT:.*]] : ...) then later %[[OUT]]. SSA numbering changes; names from captures don't.
• Give every negative test a positive twin. sve_no_alloca_in_loop.mlir's BROKEN prefix proves the pattern would actually find an alloca if there were one. A CHECK-NOT with a typo in the pattern passes forever.
• Errors are tested with --verify-diagnostics and expected-error (lab 01_basics/03), not FileCheck.

Try this yourself

1. Make sve_no_alloca_in_loop.mlir fail by removing {full-unroll=true} from its first RUN line. Read the FileCheck message.
2. Write a test for lab 02_lowering/01's reassoc finding: after convert-arith-to-llvm, fastmath<reassoc> must become llvm.fadd ... {fastmathFlags = #llvm.fastmath<reassoc>}. (Run the pass first and copy the exact spelling.)
3. Look at 2–3 tests in IREE’s Codegen/LLVMCPU/test/ and identify their RUN tool and CHECK style.