BIGCODE LAB #04: The hot function is in a shared library you did not build — profile the compiler itself

TRACK: bigcode | DIFFICULTY: 3 | NEEDS: perf, the distro clang (libclang-cpp.so + libLLVM.so), the LLVM source tree, rg

What this teaches

The realistic case: the program you are profiling is a thin executable whose time is spent inside big shared libraries (here clang-21 → libclang-cpp.so.21.1 + libLLVM.so.21.1, lab 02). You need to (1) profile the right process, (2) read the right PMU section, (3) attribute samples to library symbols, and (4) map a hot symbol back to a source tree you did not build from. And you get to make, and recognise, the two mistakes everyone makes first.

The workload: compiling heavy.cpp (a small file that instantiates std::regex, std::map and std::unordered_map a few times; ~3.2 s at -O2) and llvm/lib/Support/APInt.cpp from the LLVM tree with its real Release flags (~2.9 s).

MISTAKE 1 — You profiled ccache

$ which clang++
/usr/lib64/ccache/clang++
$ perf record -o cc.data clang++ -O2 -c heavy.cpp -o /dev/null
$ perf report -i cc.data --stdio --sort dso | head
    33.34%  ld-linux-x86-64.so.2
    21.15%  libzstd.so.1.5.7
    17.58%  ccache
    14.07%  libstdc++.so.6.0.34

clang++ on this box is a ccache wrapper, and the file had been compiled before: the "compile" was a cache hit. The profile is the dynamic loader (do_lookup_x, _dl_relocate_object), zstd decompressing the cached object, and ccache's own hashing. The compiler never ran. Signs: the DSO list contains no compiler library, and the run was suspiciously fast. Fix: CCACHE_DISABLE=1 or call the real binary (/usr/bin/clang++), and always check which for wrappers (ccache, distcc, sccache, icecc) before trusting a profile of a build tool.

MISTAKE 2 — You read the wrong PMU section

On this hybrid CPU, perf records events from both PMUs (cpu_core and cpu_atom) and perf report prints one section per PMU, the small one first:

$ perf report -i apint4.data --stdio --sort dso | head -4
# Samples: 69  of event 'cpu_atom/cycles/Pu'
    67.06%  ld-linux-x86-64.so.2
    29.01%  libclang-cpp.so.21.1

69 samples on an E-core (the process started there for a few milliseconds before migrating), and a "profile" that says the loader and Sema dominate. Reading only head of the report gave a conclusion that contradicted the compiler's own -ftime-trace (lab 05). The real data is the next section:

$ perf report -i apint4.data --stdio --sort dso | awk '/Samples:.*cpu_core/{on=1} on' | head -6
# Samples: 11K of event 'cpu_core/cycles/Pu'
    71.32%  libLLVM.so.21.1
    19.74%  libclang-cpp.so.21.1
     8.10%  libc.so.6
     0.67%  libstdc++.so.6.0.34
     0.11%  ld-linux-x86-64.so.2

Rule: look at the sample count before the percentages. 69 samples is noise; 11K is a profile. On hybrid machines pin the workload (tools/pin.sh) or filter the report to one PMU.

STEP 1 — The real profile, per library

$ perf record -F 4000 -o cc4.data /usr/bin/clang++ -O2 -c heavy.cpp -o /dev/null
$ perf report -i cc4.data --stdio --sort dso | awk '/cpu_core/{on=1} on' | head -6
# Samples: 12K of event 'cpu_core/cycles/Pu'
    61.24%  libLLVM.so.21.1
    28.82%  libclang-cpp.so.21.1
     8.87%  libc.so.6
     0.75%  libstdc++.so.6.0.34
     0.23%  ld-linux-x86-64.so.2

Two thirds of a -O2 compile of a template-heavy file is the optimizer and code generator (libLLVM), a third is the front end (libclang-cpp), a tenth is malloc/free. That split alone tells you where a compile-time fix would have to land (lab 05 confirms it with -ftime-trace: Backend 2.2 s, Frontend 1.0 s).

STEP 2 — Symbols inside the libraries

$ perf report -i cc4.data --stdio --no-children --sort dso,sym | awk '/cpu_core/{on=1} on' | head -10
     3.08%  libc.so.6             [.] _int_malloc
     1.53%  libLLVM.so.21.1       [.] llvm::FoldingSetBase::FindNodeOrInsertPos(llvm::FoldingSetNodeID const&, void*&, ...)
     1.03%  libc.so.6             [.] malloc
     0.85%  libLLVM.so.21.1       [.] llvm::AnalysisManager<llvm::Function>::getResultImpl(llvm::AnalysisKey*, llvm::Function&)
     0.84%  libclang-cpp.so.21.1  [.] clang::Sema::LookupName(clang::LookupResult&, clang::Scope*, bool, bool)
     0.81%  libLLVM.so.21.1       [.] llvm::InstCombinerImpl::run()
     0.78%  libclang-cpp.so.21.1  [.] clang::Lexer::Lex(clang::Token&)
     0.77%  libc.so.6             [.] cfree@GLIBC_2.2.5
     0.74%  libLLVM.so.21.1       [.] llvm::InlineFunction(llvm::CallBase&, llvm::InlineFunctionInfo&, ...)
$ perf report -i apint4.data --stdio --no-children --sort dso,sym | awk '/cpu_core/{on=1} on' | head -8     # the APInt.cpp compile
     2.52%  libc.so.6             [.] _int_malloc
     1.39%  libLLVM.so.21.1       [.] llvm::ScalarEvolution::getRangeRef(llvm::SCEV const*, ...)
     1.23%  libLLVM.so.21.1       [.] llvm::ScalarEvolution::getAddExpr(llvm::SmallVectorImpl<llvm::SCEV const*>&, ...)
     1.12%  libLLVM.so.21.1       [.] llvm::FoldingSetBase::FindNodeOrInsertPos(llvm::FoldingSetNodeID const&, void*&, ...)
     1.09%  libc.so.6             [.] malloc
     1.04%  libLLVM.so.21.1       [.] computeKnownBits(llvm::Value const*, llvm::APInt const&, llvm::KnownBits&, ...)
     0.88%  libclang-cpp.so.21.1  [.] clang::Lexer::Lex(clang::Token&)
     0.83%  libLLVM.so.21.1       [.] (anonymous namespace)::MachineCopyPropagation::run(llvm::MachineFunction&)

Two things a real profile of a compiler teaches:

It is flat. The top symbol is 3%; the top ten sum to ~12%. There is no single hot loop; the cost is spread over thousands of small functions (the inliner, InstCombine, SCEV, Sema lookups) plus allocation. Flat profiles need aggregation (by library, by caller, by subsystem) rather than a hunt for one line. --sort dso and --call-graph with --children give the tree view.
The names are there because the distro libraries export ~29k symbols in .dynsym (lab 02). Functions with hidden visibility appear as [unknown] or get charged to the nearest exported symbol; for those you need debug info. This box has DEBUGINFOD_URLS set, so perf report can fetch Fedora's debuginfo on demand (slow the first time: libLLVM's is large), or dnf debuginfo-install llvm-libs (root).

STEP 3 — From a symbol in a library you did not build to the source line

You have the source (the LLVM tree), just not the build that produced libLLVM.so. Names survive: demangled symbol → declaration → callers.

$ TOP='llvm::FoldingSetBase::FindNodeOrInsertPos(...)'
$ cd $LLVM_SRC && rg -n --type cpp -m3 '\bFindNodeOrInsertPos\b\s*\(' llvm/lib llvm/include | head -4
llvm/lib/Analysis/ScalarEvolution.cpp:557:  if (const SCEV *S = UniqueSCEVs.FindNodeOrInsertPos(ID, IP)) return S;
llvm/lib/Analysis/ScalarEvolution.cpp:582:  if (const SCEV *S = UniqueSCEVs.FindNodeOrInsertPos(ID, IP))
llvm/lib/Analysis/ScalarEvolution.cpp:1195:        if (const SCEV *S = UniqueSCEVs.FindNodeOrInsertPos(ID, IP))
llvm/lib/IR/LegacyPassManager.cpp:638:    if (auto *N = UniqueAnalysisUsages.FindNodeOrInsertPos(ID, IP))

FindNodeOrInsertPos is the hash-consing lookup of FoldingSet, used by many uniquing tables (SCEV, attributes, metadata). rg lists the call sites; which of them is hot is an inference until a call graph confirms it, but the profile's neighbours (getRangeRef, getAddExpr) point at ScalarEvolution's UniqueSCEVs for the APInt compile, while for heavy.cpp the run script's rg shows Attributes.cpp first: same symbol, different hot caller. So the "hot function" is really "SCEV expression uniquing during loop analysis of APInt.cpp", which is a statement about the workload, not about FoldingSet. To prove the caller attribution rather than infer it: perf record --call-graph dwarf then perf report --no-children -S <symbol> -G shows who called it; with the distro libraries that needs their debuginfo for unwinding through hidden functions, which is where debuginfod earns its keep.

The version mismatch trap: the library is 21.1.8 and the tree is a 23-git checkout; line numbers drift, names mostly do not. When you must be exact, check out the tag that matches DW_AT_producer or the package version (lab 03).

The procedure (memorize)

which the tool: wrappers (ccache, distcc) mean you may profile a cache hit.
perf record -F 4000 the real binary; on hybrid CPUs pin it or read the cpu_core section; check the sample count.
--sort dso first: which library owns the time? Then --sort dso,sym; expect a flat profile in a compiler-like program and aggregate.
Names come from .dynsym; hidden functions need debuginfo (debuginfod, -debuginfo packages, or your own build with -g).
Map symbol → source with rg in the matching version of the tree; confirm callers with a DWARF call graph.
Decide whether the finding is about the code (a hot loop) or about the workload (the input drives a whole subsystem): compilers, databases and interpreters are usually the second kind.

Try this yourself

Profile opt -O3 on the .ll of heavy.cpp (clang++ -O0 -S -emit-llvm heavy.cpp): the front end disappears and the optimizer's flat profile remains.
perf report --sort dso,sym --children on the same data: read the inclusive tree from main down to PassManager::run.
LD_DEBUG=statistics /usr/bin/clang-21 --version 2>&1 | grep -E "relocations|time": measure the loader cost that mistake 1 exposed; then compare time $LLVM_BUILD/bin/clang --version (static monolith) with time /usr/bin/clang-21 --version.