Benchmarks
Workloads
Eight schools
Results
Sampling of the non-centered eight schools model with 4 parallel chains of 1,000 warmup and 1,000 sampling iterations, followed by a summary of the draws, in CmdStanPHP, CmdStanR, and CmdStanPy. All three run the same compiled CmdStan executable with the same arguments and produce identical draws.
- Workload
- Eight schools, 4 chains × 2,000 iterations
- Output check
-
- Draws identical: yes
- Largest absolute draw difference: 0
-
Posterior mean μ
- CmdStanPHP: 4.42518
- CmdStanR: 4.42518
- CmdStanPy: 4.42518
$model = cmdstan_model(exeFile: $exe);$fit = $model->sample( data: $data, seed: 20261101, chains: 4, parallel_chains: 4, iter_warmup: 1000, iter_sampling: 1000, output_dir: $out, show_messages: false,);$summary = $fit->summary( ['mu', 'tau', 'theta'], measures: [ 'mean', 'sd', 'quantile2', 'rhat', 'ess_bulk', ],);model <- cmdstan_model(exe_file = exe)fit <- model$sample( data = data, seed = 20261101, chains = 4, parallel_chains = 4, iter_warmup = 1000, iter_sampling = 1000, output_dir = out, show_messages = FALSE, refresh = 0)summary <- fit$summary( c("mu", "tau", "theta"), "mean", "sd", ~quantile(.x, probs = c(0.05, 0.95)), "rhat", "ess_bulk")model = cmdstanpy.CmdStanModel(exe_file=exe)fit = model.sample( data=data, seed=20261101, chains=4, parallel_chains=4, iter_warmup=1000, iter_sampling=1000, output_dir=out, show_progress=False,)summary = fit.summary(percentiles=(5, 95))Eight schools, 4 chains × 2,000 iterations: elapsed time, peak anonymous memory, and peak resident set size per implementation, median of 7 runs.
| Implementation | Elapsed time | Fastest – slowest run | Peak anonymous memory | Peak resident set size |
|---|---|---|---|---|
| CmdStanPHP | 133 ms | 133 ms – 134 ms | 11.1 MiB | 34.3 MiB |
| CmdStanR (R) | 446 ms | 445 ms – 447 ms | 145 MiB | 170 MiB |
| CmdStanPy (Python) | 113 ms | 112 ms – 114 ms | 53.2 MiB | 82.1 MiB |
Methods
- Machine
- 2 AMD Zen 5 cores, 8 GB RAM, Linux
- Software
- PHP 8.5.12, CmdStan 2.40.0, CmdStanPHP v0.1.2-16-g7ddfe1c, cmdstanr 0.9.0, posterior 1.7.0, cmdstanpy 1.3.0, R 4.6.2, Python 3.14.10
- Repetitions
- 7 timed runs
- Statistic
- Median over the timed runs
- Time measure
- Elapsed time, in ms
- Timing boundaries
- Each run is a fresh process. Each interface runs once as a warm-up, then the three alternate for 7 rounds in a rotated order. The time runs from the interface call that loads the precompiled model to holding the summary: it covers loading the model, writing the data (held in memory in the runtime's usual structure) to JSON, running 4 chains in parallel as 4 CmdStan processes (parallel_chains 4 everywhere) with seed 20261101, reading the CSV output, and the summary (CmdStanPHP summary() and cmdstanr $summary() with mean, sd, 5% and 95% quantiles, R-hat, and bulk ESS for mu, tau, and theta; CmdStanPy summary(), which runs CmdStan's stansummary for every column). Starting the runtime and loading the interface's libraries are outside the timing. Every process runs under bench/peakmem.c. Memory is the interface process's peak anonymous memory (RssAnon in /proc, sampled every millisecond): its own allocations, without the file pages it maps and without the CmdStan chain processes. Peak resident set size, which counts those pages, is reported separately.
- Parallel chains
- 4
- Compilation
- Outside the timing. The model is compiled once with CmdStan's makefile before any run, and every interface loads that executable (sha256 9db59bddf970) with its exe_file argument.
- Memory metric
- Peak anonymous memory; Peak resident set size
- Recorded
- 2026-11-01, commit 7ddfe1c
Time by phase
Results
The eight schools workload timed in three phases inside each interface: loading the model, sampling, and summarizing. The same 4 chains run by CmdStan's executable without an interface.
- Workload
- Eight schools, 4 chains × 2,000 iterations
- Output check
-
- Draws identical: yes
Eight schools, 4 chains × 2,000 iterations: elapsed time and peak resident set size of child processes per implementation and case, median of 7 runs.
Load the model
| Implementation | Elapsed time | Fastest – slowest run | Peak resident set size of child processes |
|---|---|---|---|
| CmdStanPHP | 12.3 ms | 12.2 ms – 12.3 ms | not measured |
| CmdStanR | 59.2 ms | 59.0 ms – 60.9 ms | not measured |
| CmdStanPy | 0.046 ms | 0.043 ms – 0.047 ms | not measured |
Sample: write data, run 4 chains, wait for them
| Implementation | Elapsed time | Fastest – slowest run | Peak resident set size of child processes |
|---|---|---|---|
| CmdStanPHP23.8 ms more than CmdStan alone | 70.7 ms | 68.7 ms – 77.4 ms | not measured |
| CmdStanR266.8 ms more than CmdStan alone | 314 ms | 310 ms – 322 ms | not measured |
| CmdStanPy9.1 ms more than CmdStan alone | 56.0 ms | 55.2 ms – 57.4 ms | not measured |
| CmdStan aloneThe 4 chain processes only, started together with CmdStanPHP's arguments | 46.9 ms | 45.5 ms – 47.4 ms | 13.5 MiB |
Summary: read the CSV files and summarize
| Implementation | Elapsed time | Fastest – slowest run | Peak resident set size of child processes |
|---|---|---|---|
| CmdStanPHP | 45.2 ms | 44.7 ms – 45.4 ms | not measured |
| CmdStanR | 69.7 ms | 68.7 ms – 71.1 ms | not measured |
| CmdStanPy | 54.6 ms | 54.5 ms – 54.7 ms | not measured |
Total
| Implementation | Elapsed time | Fastest – slowest run | Peak resident set size of child processes |
|---|---|---|---|
| CmdStanPHP81.5 ms more than CmdStan alone | 128 ms | 126 ms – 135 ms | not measured |
| CmdStanR395.0 ms more than CmdStan alone | 442 ms | 439 ms – 453 ms | not measured |
| CmdStanPy63.8 ms more than CmdStan alone, including CmdStan's stansummary executable, which its summary runs | 111 ms | 110 ms – 112 ms | not measured |
Methods
- Implementations
- CmdStan alone: CmdStan's compiled executable, 4 processes started together from Python with the arguments CmdStanPHP passes; CmdStanPHP: CmdStanPHP, plain PHP (no phit); CmdStanR: cmdstanr with posterior; CmdStanPy: CmdStanPy
- Machine
- 2 AMD Zen 5 cores, 8 GB RAM, Linux
- Software
- PHP 8.5.12, CmdStan 2.40.0, CmdStanPHP v0.1.2-35-gccadda9, cmdstanr 0.9.0, posterior 1.7.0, cmdstanpy 1.3.0, R 4.6.2, Python 3.14.10
- Threads
- chains 4
- Repetitions
- 7 timed runs, 1 warm-up run
- Statistic
- Median over the timed runs
- Time measure
- Elapsed time, in ms
- Timing boundaries
- Each run is a fresh process. Each runs once as a warm-up, then the four alternate for 7 rounds in a rotated order. Inside each interface the clock is read before loading the precompiled model, after it, after sample() returns, and after the summary; the phases are the differences, and Total runs from the first read to the last, as in the interface benchmark. Starting the runtime and loading libraries are outside the timing. CmdStan alone writes the data file before its clock starts and stops the clock when the last of the 4 chains exits. The summaries differ by interface as in the interface benchmark: CmdStanPy's runs CmdStan's stansummary.
- Time spent in
- Load and Summary: the interface process. Sample: CmdStan's executable, launched by the interface, plus the interface's own work. CmdStan alone: CmdStan's executable only. CmdStanPy's summary also runs CmdStan's stansummary executable.
- Cache
- Warm
- Compilation
- Outside the timing. The model is compiled once with CmdStan's makefile, and every run loads that executable (sha256 9db59bddf970).
- Memory metric
- Peak resident set size of child processes. Largest peak resident set size among the 4 chain processes, from getrusage(RUSAGE_CHILDREN); CmdStan alone only.
- Recorded
- 2026-11-01, commit ccadda9
Reproduction
From the repository root
python3 verification/interfaces/bench/split.py- Raw record
- site/benchmarks/interfaces-split.json
- Source
- verification/interfaces/bench/split.py
Not measured
No benchmark covers larger models, CSV reading at scale, or compilation time.