What's Inside a Reduction?
The author describes adding reduction counting to Benchee and initially expecting reductions to give a stable, architecture-specific performance metric that would enable reliable tests for performance‑sensitive functions. Benchmarks comparing Enum.flat_map/2 with Enum.map/2 |> List.flatten/1 showed a much larger difference in reduction counts than in wall‑clock time, which broke that assumption. Investigating BEAM assembly with the decompile package revealed that many instructions (move, test, get_map_elements, bif, jump, allocate, make_fun2, etc.) are not counted as reductions but still consume time, and that BIFs run in C are also outside the reduction count.
Using small Elixir examples, the post shows two functions that each cost one reduction but have very different numbers of BEAM instructions and runtimes, and then a third function that takes two reductions yet runs faster than the original one‑reduction version. From this, the author concludes that reduction counts do not correlate reliably with runtime and are therefore not suitable as a primary tool for performance testing. Reduction data can still guide very low‑level optimization work (for example, replacing reductions with BIFs in hot paths), but such tuning largely belongs to language or, at most, library authors. The feature will likely be documented and released to see how others might use it, and the exploration underscores how BEAM’s compile‑time and runtime optimizations can produce surprising performance characteristics.
Reading notes#
- The author added a reduction‑counting feature to Benchee and initially ran into confusing benchmark results that turned out to be correct and exposed flawed assumptions about reductions as a performance metric.
- The original hope was that reductions would offer a stable, architecture‑specific measure for precise performance tests, unlike noisy wall‑clock timing, because deterministic functions should use a constant number of reductions.
- A benchmark comparing
Enum.flat_map/2withEnum.map/2 |> List.flatten/1showed the slower version having 2.10× the reductions but only 1.36× the runtime, contradicting the expectation of a tight correlation. - The author had implicitly modeled things like lambda calculus with only functions, assuming most work was function calls of roughly similar cost and that BIF usage would be similarly distributed.
- Using Michał Muskała’s
decompilepackage, the BEAM assembly forEnum.flat_map/2showed many operations that are notcallorcall_ext, such asmove,test,get_map_elements,bif,jump,allocate,make_fun2, andkill. - These non‑call instructions are not counted as reductions but still contribute to wall‑clock time, so functions that do lots of such work can be slower while still consuming only one reduction.
- A simple
Testmodule example definesa_from_map/1anda_from_map2/4, both counted as one reduction per call, to illustrate that they differ significantly in instruction count and runtime. - The BEAM assembly for
a_from_map/1has four main instructions (map check, element extraction, move, return), whilea_from_map2/4has multiple repeatedtestandget_map_elementsoperations plus heap and list construction instructions. - Benchmarking these functions with Benchee shows
a_from_map/1running around 3.26× faster thana_from_map2/4, while both still register a reduction count of 1. - The author ties the ~3.5× runtime difference to the roughly 3.5× increase in the number of BEAM instructions between the two functions rather than to reduction counts.
- From these experiments, the author infers that using reduction counts as the main basis for performance testing can give misleading results and false positives where reduction numbers match but timings differ greatly.
- The code that is most performance‑critical is also likely to rely on BIFs and other non‑reduction operations implemented in C, which further weakens the usefulness of reduction counts in testing such functions.
- Reduction information can still be useful for low‑level optimization of extremely hot paths, for instance by guiding the replacement of reductions with BIFs or other faster constructs.
- This style of optimization is framed as appropriate mostly at the language level and perhaps in a limited way in libraries, rather than typical production application code.
- The author leans toward documenting the behavior and releasing the feature despite its limits, to see if others find valuable uses that have not been considered.
- A postscript example extends the
Testmodule witha_from_map3/1, which delegates to a privatedo_a_from_map3/1, resulting in two reductions for a call. - The BEAM assembly for
a_from_map3/1shows it essentially just performing acall_onlytodo_a_from_map3/1, where the actual map pattern match and extraction occurs. - A benchmark of
a_from_map/1,a_from_map2/4, anda_from_map3/1finds thata_from_map3/1(two reductions) runs faster thana_from_map/1(one reduction) and much faster thana_from_map2/4. - Reduction statistics for that benchmark show
a_from_map3/1at 2 reductions versus 1 for the other two functions, yet its wall‑clock performance is best. - The author closes by noting that the BEAM has many compile‑time and runtime optimizations and stresses that performance work on BEAM should rely on measurement instead of guesses because optimizations can lead to surprising results.
