Audit the instructions in each build tarball against its CPU target

[HaoZeke]

Depends on #311, which adds the per-target references this audit reads.
Until #311 is merged, the diff here also shows its commits; the last five commits are the new part.

#311 checks the compiler's native flags before a build.
This adds the matching check after the build: every instruction in the tarball has to be one the target CPU can run, against the same references.
A build for target T made on a node of a newer type passes every test on that node, and the result crashes with SIGILL on a real T machine.
The audit catches that before the tarball is deployed, however the build was made.

How it works.

• scripts/isa_audit/check_tarball.sh unpacks the tarball and picks the newest GCC reference for the target.
• x86_64: isa_audit.py decodes every executable section of every ELF file with iced-x86.
  An instruction fails when it needs a feature the reference marks -mno-X.
• aarch64: isa_audit_aarch64.py disassembles each file with llvm-objdump twice, once with every feature and once with the reference's -mcpu.
  An instruction that decodes only in the first run fails.
• Generic targets use scripts/isa_audit/references/{x86_64,aarch64}/generic/baseline.txt: what GCC enables for -march=x86-64 and -mcpu=generic, the flags EasyBuild's --optarch=GENERIC passes.
  generate_generic_references.sh writes them from a given GCC.
• dispatch-allow.txt lists code that picks its instruction set at run time.
  Each line has a package and its features, optionally with function globs, and a comment naming the mechanism.
  Those sites print as DISPATCH and do not fail the build.
• A violation prints an ERROR: line, which bot/check-build.sh already reports as a failed build.
• Targets without a reference and hosts without iced-x86 or llvm-objdump are skipped with a message.
  EESSI_SKIP_ISA_AUDIT skips the audit.

Measured on 2025.06 with the gcc-14.3.0 references:

files	against	result
sapphirerapids hwloc 2.12.1	icelake	fail: AVX512-FP16 vmovw in libhwloc.so
sapphirerapids GROMACS 2025.4	icelake	fail: vmovw in all four libgromacs variants
icelake hwloc, icelake GROMACS, zen4 GROMACS	their own references	pass
whole intel/icelake tree, 1,263 package versions	icelake	42 flagged, all runtime dispatch
whole intel/sapphirerapids tree	icelake	402 flagged (32%), mostly AVX512-FP16
libzstd from nvidia/grace and neoverse_v1	neoverse_n1	fail: SVE (whilelo, ld1w)
libzstd from each Arm tree	its own reference	pass

A whole x86_64 tree takes about 25 minutes on 12 cores; one tarball from a PR takes seconds.

Generic targets.
I audited 25 x86_64/generic and 19 aarch64/generic installs from 2025.06 (BLAS, MPI, FFTW, GROMACS, SciPy-bundle, FFmpeg, codecs, compression, GCCcore's runtime libraries).
Most of what they flag is run-time dispatch, now on the list with function globs where the symbols allow it.
Two x86_64 installs fail for real:

• UCX 1.19.0 is built with -mavx throughout: AVX shows up in 15 of its 16 ELF files.
  --enable-optimizations makes UCX's configure compile and run an AVX test program, and add -mavx for the whole package when it passes (checking -mavx... yes in the build log), so the generic install needs AVX.
• SciPy-bundle 2025.07 has SSE3 (movddup, fisttp) in NumPy's baseline code, since NumPy's cpu-baseline=min is SSE3 on x86-64 (Enabled: SSE SSE2 SSE3 in the build log).
  The same code reaches scipy.special through npymath.

Neither is on the allow list, since neither is chosen at run time.
Three instruction patterns pass by rule because they are safe on any CPU of the architecture:

• tzcnt is rep bsf, which a CPU without BMI1 runs as bsf; GCC emits it for __builtin_ctz at -march=x86-64.
• xgetbv faults without OSXSAVE, so detection code runs it only after that CPUID check.
• The -moutline-atomics helpers libgcc puts in each library use LSE only after checking __aarch64_have_lse_atomics; stripped libraries lose the helper names, so the auditor recognises the check itself.

Dispatch list.
The seed list came from the icelake tree and has 21 packages; the generic installs added OpenBLAS, OpenMPI, Boost, zstd, libjpeg-turbo, libwebp, pixman and more lines for GCCcore, x264, FFmpeg, libfabric and SciPy-bundle.
Each new line names the selection mechanism, and most carry function globs, e.g. ompi_op_avx_* for OpenMPI or *_bmi2* for zstd, so the same feature anywhere else in the package still fails.
OpenBLAS, FFmpeg and NumPy stay package-wide: FFmpeg is stripped, and the other two dispatch whole objects whose helpers have no target in their names.
The two FMA4 sites in Rust's rust-analyzer are compiler_builtins' libm fma_with_fma4, picked after a cpuid check, so any package with Rust code that calls f64::mul_add carries them.
The shadow-stack instructions from libgcc's unwinder (rdssp, then incssp only after a non-zero rdssp) pass by rule rather than per package.

Tests.
tests/isa_audit/test_check_tarball.sh compiles a small library, packs it as a tarball and checks it against native references, the generic baselines, allow lines with and without function globs and the outline-atomics helpers, stripped or not.
The rest cover a missing reference, EESSI_SKIP_ISA_AUDIT and a missing tarball.
All 21 pass locally with LLVM 19; the new workflow runs them on ubuntu-24.04 with LLVM 20.

Limitations.

• Static archives (.a) are not read.
  Their code is audited in the package that links them.
• Function globs need symbols; a stripped package can only have a package-wide line.
• The bot hosts need iced-x86 and pyelftools for x86_64 and llvm-objdump for aarch64.
  Without them the audit skips with a message instead of failing.

Questions:

1. Should a violation fail the build from the start, or print a warning until the allow list settles?
2. iced-x86 comes from PyPI.
   Is a pip install on the bot hosts fine, or should it come from an EESSI module?
3. UCX and NumPy above would fail a generic rebuild today.
   Should the fix go in their easyconfigs (no --enable-optimizations for UCX, cpu-baseline=none for NumPy), or should generic accept SSE3?