Benchmarks

How Cinder measures performance, what the numbers mean, and how regressions are caught. The measured results for each run live in REPORT.md (section 4); this page is the method.

Programs #

Program	Scene	Default run	Output
`bench/stress.mojo`	Section 8 stress scene: 50,000 static instances in a grid, 2,000 dynamic boxes falling onto a floor, 256 clustered point lights plus a shadowed sun, headless at 1920x1080, high preset	120 frames after one start-up frame and 10 warm-up frames (pipelines, environment bake)	`.cache/bench/stress.json`
`bench/sample_bench.mojo`	The sample game (`sample/`) on autopilot, headless at 1920x1080, high preset, arena level by default	300 frames after one start-up frame	`.cache/bench/sample.json`
`bench/ecs_bench.mojo`	100,000 entities, two archetypes, `position += velocity * dt` through a two-component query, against flecs through the shim	100 frames	stdout (ADR-0011)

All three are plain Mojo programs; --frames, --width, --height, --instances, --bodies, --lights, --level and --out override the defaults.

bash

tools/in_container.sh pixi run bench-stress
tools/in_container.sh pixi run bench-sample
tools/in_container.sh pixi run bench-all      # both, then tools/bench_check.py

What is measured #

Each frame records three durations with the engine's own clock, from inside App.run_frame:

frame_ms: the whole frame, from the start of input polling to the end of the offscreen readback (headless) or present (windowed). This is what the p99 budget is about.
physics_ms: the Jolt step calls of that frame's fixed steps, summed (zero on frames with no step). Kinematic push, body pull and Game.fixed_update are in frame_ms but not here, so a level whose bodies are asleep reports microseconds.
render_ms: Renderer.render, from the bridge's render scene to command submission.

The JSON carries median, p99, mean, max and count of each, the adapter name reported by wgpu, the time from App construction to the end of the first frame (startup_ms in the stress result, cold_start_ms in the sample result, where it includes loading the level's assets), the scene parameters, the draw-call and instance counts of the last frame and the wgpu error count, so any two results can be told apart.

The GPU time itself is not measured separately: a headless frame ends with a readback that waits for the GPU, so frame_ms includes it. render_ms stops at submission, so the difference between the two is a usable proxy for GPU time on a real adapter.

Budgets (section 8 of the brief) #

Budget	Where it is checked
Sample p99 frame time <= 16.6 ms at 1080p, high preset, on the reference hardware	`bench/sample_bench` p99 `frame_ms`
`bench/stress` >= 60 fps (50k instances, 2,000 bodies, 256 lights)	`bench/stress` median `frame_ms` <= 16.6
Physics step <= 4 ms for 2,000 bodies	`bench/stress` p99 `physics_ms`
Cold start <= 2 s to the first frame of the sample	`cold_start_ms` in both results
Zero allocations per frame in steady state	`tests/perf/test_zero_alloc.mojo` (`pixi run test-zero-alloc`), a hard test rather than a benchmark

The budgets name hardware (Linux with an RTX 3060-class GPU, Apple M2). Numbers from any other adapter are regression baselines for that adapter only; REPORT.md says which adapter produced each table.

Regression checking #

tools/bench_check.py RESULT.json... [--baselines bench/baselines] [--tolerance 0.10] compares each result with the baseline recorded for the same benchmark name and adapter and fails when any tracked metric (median and p99 of frame, physics and render time; cold start) is more than 10% slower. A result from an adapter with no baseline is printed and skipped, never compared against a different GPU. --update records the current numbers as the new baseline; a baseline change is a reviewable diff in bench/baselines/.

Counting allocations #

native/src/alloc_counter.c builds to libcinder_alloccount.so. When LD_PRELOADed it interposes malloc, calloc, realloc, posix_memalign, aligned_alloc and the Mojo runtime's KGEN_CompilerRT_AlignedAlloc, and attributes each call to the main executable, the Mojo runtime or a shared library by its return address. The shim exposes the counters as cn_alloc_counter_* (no-ops when the library is not loaded). tests/perf/test_zero_alloc.mojo runs a representative scene, warms up for 90 frames, then asserts that 60 frames make zero allocations from the executable. App.count_allocations prints per-phase deltas and Renderer.alloc_marks prints per-render-stage deltas to locate a regression; CN_ALLOCCOUNT_DUMP=1 prints the per-module totals at exit.

Interpreting a run on lavapipe #

Inside the toolchain container the only Vulkan device is Mesa's lavapipe, a CPU rasteriser. Frame times there are dominated by shading on the CPU and say nothing about the budgets; they are still stable enough (the same container, the same CPU) to catch CPU-side regressions in the engine, which is how they are used in CI.