Matching benchmarks¶
The matching benchmark exercises the public line-matching and ownership attribution APIs. It uses deterministic fixtures and emits JSON so results can be retained and compared across revisions.
Run the pull-request-sized suite from a source checkout:
The full suite increases the repeated-line, low-similarity, reversed-order, Unicode, and many-batch workloads and adds a 50,000-line sparse-edit case:
Use --case NAME to isolate a regression. The option can be repeated. Use
--seed, --warmups, and --repeats to control reproducibility and sampling.
The defaults are one warm-up and three samples in quick mode, and two warm-ups
and seven samples in full mode. The full suite can take several minutes on a
typical workstation.
Reading results¶
Each measured phase reports every sample plus minimum, median, 95th percentile,
and maximum elapsed time and peak Python allocation. Elapsed-time samples run
without allocation tracing. Peak-allocation samples run separately, with at
most three samples per phase, so tracemalloc does not distort the timing
results. Fixture generation, repository creation, and separately managed phase
prerequisites and cleanup happen outside the timer. End-to-end public APIs still
include any cleanup that is intrinsic to the operation.
Synthetic text is coalesced into bounded 64 KiB byte chunks before measurement, so large cases do not retain a Python object for every line alongside the line-addressable buffers.
The phases are:
buffer_loading: create line-addressable buffers and build their line indexes.git_object_resolution: resolve the expressions needed by batch claims.blob_loading: stream source blobs into bounded line buffers.mapping: construct and traverse the structural line mapping.unit_attribution: enumerate changed units from an already-built comparison.claim_attribution: run file attribution end to end, including Git I/O, matching, and ownership claims.
The many-batch case keeps its file dimensions fixed between modes so it isolates the cost of increasing the number of claims.
The report records the seed, sample counts, project revision, project version,
Python, Git, platform, input dimensions, and content hashes.
tracemalloc_peak_bytes measures Python allocator activity; it is not total
process resident memory. Tracing adds overhead, so compare runs made with the
same settings and on comparable hardware. Reports also identify a dirty tracked
working tree so results made from uncommitted code are not mistaken for their
recorded revision.
The binary-exclusion case records that NUL-containing input is intentionally
excluded from text matching. It has no measured phases.
Comparing revisions¶
Create reports from clean checkouts with the same Python, Git, mode, seed, warm-up count, and repeat count. Then compare their median phase times and peak Python allocations:
python scripts/benchmark_matching.py \
--compare before.json after.json \
--regression-threshold-percent 20 \
--output comparison.json
Comparison is trend-based: matching case, phase, and measurement names are
aligned, and a measurement is marked as a regression when its median exceeds
the selected percentage. Comparison rejects reports with different modes,
seeds, sample settings, measurement methods, or case dimensions. Environment
differences such as Python, Git, or platform versions are retained as warnings
in the comparison report.
Add --fail-on-regression when a stable, dedicated runner should return a
failure status. Pull-request CI deliberately runs only a functional smoke test,
without hardware-dependent performance thresholds.
Refresh a stored baseline only after investigating every flagged phase and confirming that the change is expected. Record the revision and benchmark settings with the baseline; the JSON already contains both. Prefer several full-suite runs on a quiet dedicated machine over treating one workstation run as authoritative.