CmdStanPHP

Data & Workflow

Statistics, Modeling, & Visualization

Moment matching

Verification

Comparisons with R

On this page

Comparisons with other interfaces

Scope

LOO with moment matching, with and without the split step: pointwise values, Pareto k, effective sample sizes, and estimates. Analytic posteriors with identical draws, and a same-seed fit.

References

Reference
loo 2.10.1.9000 loo_moment_match.default(), and cmdstanr 0.99.0 (commit 0a54123) loo(moment_match = TRUE)
Versions
PHP 8.5.10, CmdStan 2.40.0, phit 0.4.4, R 4.6.1, cmdstanr 0.99.0 (commit 0a54123), loo 2.10.1.9000

Data

Analytic regressions
Conjugate normal regressions whose outlier needs shifts only, shifts and scaling, or a covariance map, with exact log densities (tests/MomentMatchFixture.php).
Regression fit
A Stan regression with an outlier, 4 chains of 500 warmup and 1,000 sampling iterations, seed 20261012.

Method

For the analytic posteriors, loo's loo_moment_match.default() receives the same draws and exact log densities. For the fit, CmdStanR's loo(moment_match = TRUE) runs on a same-seed fit of the same program.

Check
Comparison with a reference implementation
Recorded
2026-11-02, PHP 8.5.10, macOS 27.0.1 (arm64), commit 4ec6712

Tolerances

|x − r| ≤ 1e-12

Applies to: Lp__ draws of the fit

|x − r| ≤ 1e-8 × max(|r|, 1)

Applies to: Pointwise values, diagnostics, and estimates

|x − r| ≤ 1e-8

Applies to: Pareto k of the fit

x: package value; r: reference value. Absolute error: |x − r|. Relative error: |x − r| / max(|r|, 1). Max errors are over the values a row compares.

Results

55 comparisons in 9 cases: 55 pass, 0 fail, 0 skipped.

By case

Results by case
Case Comparisons Largest abs. error Largest rel. error Status
Regression, shifts only, unsplit 7 1.1e-11 2.3e-14 pass
Regression, shifts and scaling, unsplit 7 1.8e-11 3.3e-13 pass
Regression, covariance map, unsplit 7 1.5e-11 2.5e-13 pass
Regression, shifts only, split with the exact inverse 7 1.3e-11 2.7e-14 pass
Regression, shifts and scaling, split with the exact inverse 7 1.8e-11 3.3e-13 pass
Regression, covariance map, split with the exact inverse 7 3.6e-11 2.5e-13 pass
Regression, shifts and scaling, loo's own split 1 8.2e-4 pass
Regression, covariance map, loo's own split 1 4.7e-2 pass
Regression fit, 4 chains × 1,000 draws, vs CmdStanR loo(moment_match = TRUE) 11 1.4e-11 7.8e-13 pass

Every comparison

Every comparison
Case Quantity Criterion Values Max abs. error Max rel. error Status
Regression, shifts only, unsplit pointwise elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 6 1.6e-14 2.1e-15 pass
pointwise mcse_elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 6 1.9e-15 1.9e-15 pass
pointwise p_loo |x − r| ≤ 1e-8 × max(|r|, 1) 6 3.4e-14 3.9e-15 pass
pointwise looic |x − r| ≤ 1e-8 × max(|r|, 1) 6 6.8e-14 3.2e-15 pass
pareto_k |x − r| ≤ 1e-8 × max(|r|, 1) 6 2.3e-14 2.3e-14 pass
n_eff |x − r| ≤ 1e-8 × max(|r|, 1) 6 1.1e-11 1.4e-14 pass
estimates and SEs |x − r| ≤ 1e-8 × max(|r|, 1) 6 9.9e-14 5.6e-15 pass
Regression, shifts and scaling, unsplit pointwise elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 1.0e-12 3.7e-14 pass
pointwise mcse_elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 1.0e-14 1.0e-14 pass
pointwise p_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 1.1e-12 4.5e-14 pass
pointwise looic |x − r| ≤ 1e-8 × max(|r|, 1) 7 2.1e-12 3.7e-14 pass
pareto_k |x − r| ≤ 1e-8 × max(|r|, 1) 7 3.3e-13 3.3e-13 pass
n_eff |x − r| ≤ 1e-8 × max(|r|, 1) 7 1.8e-11 2.9e-14 pass
estimates and SEs |x − r| ≤ 1e-8 × max(|r|, 1) 6 1.9e-12 4.3e-14 pass
Regression, covariance map, unsplit pointwise elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 2.2e-14 8.5e-15 pass
pointwise mcse_elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 9.2e-15 9.2e-15 pass
pointwise p_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 1.7e-14 1.2e-14 pass
pointwise looic |x − r| ≤ 1e-8 × max(|r|, 1) 7 2.5e-14 5.3e-15 pass
pareto_k |x − r| ≤ 1e-8 × max(|r|, 1) 7 2.5e-13 2.5e-13 pass
n_eff |x − r| ≤ 1e-8 × max(|r|, 1) 7 1.5e-11 7.8e-14 pass
estimates and SEs |x − r| ≤ 1e-8 × max(|r|, 1) 6 2.0e-14 2.1e-15 pass
Regression, shifts only, split with the exact inverse pointwise elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 6 4.4e-15 2.1e-15 pass
pointwise mcse_elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 6 6.0e-16 6.0e-16 pass
pointwise p_loo |x − r| ≤ 1e-8 × max(|r|, 1) 6 3.6e-15 3.2e-15 pass
pointwise looic |x − r| ≤ 1e-8 × max(|r|, 1) 6 3.2e-14 2.7e-15 pass
pareto_k |x − r| ≤ 1e-8 × max(|r|, 1) 6 2.3e-14 2.3e-14 pass
n_eff |x − r| ≤ 1e-8 × max(|r|, 1) 6 1.3e-11 2.7e-14 pass
estimates and SEs |x − r| ≤ 1e-8 × max(|r|, 1) 6 6.4e-14 2.3e-15 pass
Regression, shifts and scaling, split with the exact inverse pointwise elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 1.4e-14 8.9e-15 pass
pointwise mcse_elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 1.7e-16 1.7e-16 pass
pointwise p_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 5.0e-14 5.4e-15 pass
pointwise looic |x − r| ≤ 1e-8 × max(|r|, 1) 7 2.8e-14 4.4e-15 pass
pareto_k |x − r| ≤ 1e-8 × max(|r|, 1) 7 3.3e-13 3.3e-13 pass
n_eff |x − r| ≤ 1e-8 × max(|r|, 1) 7 1.8e-11 7.7e-15 pass
estimates and SEs |x − r| ≤ 1e-8 × max(|r|, 1) 6 4.6e-14 2.3e-15 pass
Regression, covariance map, split with the exact inverse pointwise elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 2.2e-14 8.5e-15 pass
pointwise mcse_elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 1.2e-15 1.2e-15 pass
pointwise p_loo |x − r| ≤ 1e-8 × max(|r|, 1) 7 1.7e-14 1.2e-14 pass
pointwise looic |x − r| ≤ 1e-8 × max(|r|, 1) 7 5.5e-14 5.3e-15 pass
pareto_k |x − r| ≤ 1e-8 × max(|r|, 1) 7 2.5e-13 2.5e-13 pass
n_eff |x − r| ≤ 1e-8 × max(|r|, 1) 7 3.6e-11 6.4e-14 pass
estimates and SEs |x − r| ≤ 1e-8 × max(|r|, 1) 6 7.3e-14 6.9e-15 pass
Regression, shifts and scaling, loo's own split pointwise elpd_loo differs by more than 1e-6, as documented 1 8.2e-4 pass
Regression, covariance map, loo's own split pointwise elpd_loo differs by more than 1e-6, as documented 1 4.7e-2 pass
Regression fit, 4 chains × 1,000 draws, vs CmdStanR loo(moment_match = TRUE) lp__ draws |x − r| ≤ 1e-12 4,000 5.0e-14 4.8e-15 pass
unsplit pointwise elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 12 2.0e-13 7.4e-15 pass
unsplit pointwise mcse_elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 11 9.0e-17 9.0e-17 pass
unsplit pointwise p_loo |x − r| ≤ 1e-8 × max(|r|, 1) 12 2.9e-13 1.4e-14 pass
unsplit pointwise looic |x − r| ≤ 1e-8 × max(|r|, 1) 12 3.3e-14 6.7e-15 pass
unsplit pareto_k |x − r| ≤ 1e-8 12 7.8e-13 7.8e-13 pass
unsplit n_eff |x − r| ≤ 1e-8 × max(|r|, 1) 11 1.4e-11 6.9e-15 pass
split pointwise elpd_loo |x − r| ≤ 1e-8 × max(|r|, 1) 12 3.6e-14 7.4e-15 pass
split pointwise p_loo |x − r| ≤ 1e-8 × max(|r|, 1) 12 2.8e-14 1.4e-14 pass
split pointwise looic |x − r| ≤ 1e-8 × max(|r|, 1) 12 3.3e-14 6.7e-15 pass
split pareto_k |x − r| ≤ 1e-8 12 7.8e-13 7.8e-13 pass

Limitations

Exclusions

  • Regression fit, 4 chains × 1,000 draws, vs CmdStanR loo(moment_match = TRUE): unsplit pointwise mcse_elpd_loo (1 value). loo reports NA, so the value is not compared; the test requires a finite value from CmdStanPHP.
  • Regression fit, 4 chains × 1,000 draws, vs CmdStanR loo(moment_match = TRUE): unsplit n_eff (1 value). loo reports NA, so the value is not compared; the test requires a finite value from CmdStanPHP.

Known differences

  • The split step uses the exact inverse of the accumulated affine map. loo 2.10.1 and earlier use an inverse that is wrong once a shift is combined with scaling or a covariance map, so with the split only shift-only observations are compared with those versions.
  • The split relative efficiency is the minimum over both halves of the draws; loo uses the second half twice.
  • Relative efficiencies are computed from exp(log_lik − max). Where loo reports NA because every likelihood value is tiny, CmdStanPHP must report a finite value.

Reproduction

From the repository root 
PHIT_LIBRARY=/path/to/phit.so php verification/compatibility/run.php
Raw record
site/verifications/moment-matching.json
Full results
verification/compatibility/results.json