Sigma Statistics Validation

Baseline Characteristics / Descriptive Statistics

Validation objective

The descriptive statistics module was validated against an independent R reference implementation. The comparison covered missing-value preprocessing and point-decimal numeric input, automatic and overridden variable types, overall and grouped cohorts, missing and invalid numeric values, metric summaries, type-7 percentiles, categorical frequencies and modes, and data-quality warnings. The rendered table only rounds for display; the validation used the underlying full-precision values produced by the Sigma calculation core.

Validation design

A deterministic seed generator created 27 wide datasets at sample sizes n = 1, 2, 3, 4, 5, 10, 30, 100, 1000, with three independent replicates per sample size. These seeds were evaluated in 102 configurations covering no-group and grouped analyses, automatic statistic selection, and Show all statistics for all parameters.

Across the validation suite, the datasets covered normal, skewed, heavy-tailed, tied, constant, missing, and extreme-outlier numeric variables, a mixed numeric/text column, numeric and text binary variables, multi-level categorical variables, a numeric-coded category, a category with missing values, a unique subject identifier, and an all-missing column. Some grouped configurations included missing group assignments, which were retained in Overall and excluded from group-specific results. The smallest datasets necessarily did not realize every feature; for example, the extreme outlier was introduced from n ≥ 2, categorical missingness from n ≥ 4, and invalid mixed numeric values and missing group assignments from n ≥ 5.

The independent R comparison used point-decimal input throughout. Point, comma, and automatic decimal- separator branches were additionally exercised by JavaScript core unit tests; comma and automatic separator handling were therefore outside the independent R comparison.

Independent R reference

Reference results were calculated in R 4.5.2 using base-R formulas independent of the Sigma JavaScript implementation. The reference used mean, var, sd, qt, and quantile(type = 7), together with explicit formulas for SEM, confidence intervals, adjusted Fisher–Pearson skewness, adjusted excess kurtosis, missing percentages, frequencies, and modes. Both implementations emitted the same long-format result schema.

Comparison criteria

Counts, cohort labels, category levels, modes, type decisions, warning messages, and configuration metadata were required to match exactly. Numeric values were accepted when abs(Sigma − R) ≤ 1e−10 + 1e−10 × abs(R). This combined absolute and relative threshold protects near-zero results while remaining appropriate for the deliberately included very large outlier-driven moments and variances.

Validation results

R and Sigma each produced 256,822 keyed long-format result rows. All 256,822/256,822 rows passed, with 0 missing keys, 0 text mismatches, and 0 numeric failures. Agreement by result section was:

analysis metadata: 408/408,
cohort definitions and sizes: 396/396,
type detection and data quality: 10,710/10,710,
numeric statistics: 47,817/47,817,
percentiles: 20,493/20,493,
categorical summaries: 13,068/13,068, and
frequency rows: 163,930/163,930.

The largest absolute numeric difference was 4.8828125 × 10⁻⁴ for the sample variance of a deliberately extreme outlier dataset with variance approximately 5.0 × 10¹¹. Its scaled relative difference was only approximately 9.77 × 10⁻¹⁶. Across all results, the maximum scaled relative difference was approximately 1.19 × 10⁻¹³, well inside the predefined threshold.

Reproducibility

The validation bundle is stored under validation tests/descriptive_statistics. It contains the R seed generator, the independent R reference, the Sigma batch runner, the row-level comparison script, generated inputs and outputs, and a one-command validation runner. Re-running run_validation.sh regenerates all seeds and comparison artifacts from master seed 20260801.

Conclusion

The validation demonstrates exact structural agreement and numerical equivalence between Sigma and R across small-sample edge cases, large samples, missing and invalid values, grouping, automatic and manual type handling, distribution-shape statistics, custom percentiles, and categorical frequency calculations. The descriptive-statistics calculation core is therefore validated for the R-comparison scope and definitions described above. Decimal-comma and automatic separator handling have unit-test, but not independent R-comparison, coverage.

Normality Testing

Validation objective

The production JavaScript calculation core was compared with independent, established implementations of all four procedures in the tool: Shapiro–Wilk, the Lilliefors-corrected Kolmogorov–Smirnov test, Anderson–Darling, and D’Agostino–Pearson omnibus K². The comparison covered test statistics, p-values, estimability status, and the reject / do-not-reject decision at α = 0.05.

This supersedes the earlier tool-like comparison, in which several reference formulas had inadvertently reproduced Sigma’s own formulas. No cloned Sigma implementation was accepted as a reference in the present validation.

Validation design

The fixed validation corpus contains 350 datasets: seven scenarios (normal, skewed, heavy_tail, bimodal, outliers, ties, and degenerate), five sample sizes (8, 20, 100, 1000, and 5000), and ten deterministic replicates per combination. Of these, 340 datasets produced paired finite results for each test. The ten constant n = 8 datasets were non-estimable in both Sigma and the reference software.

Additional automated tests cover strict decimal-point and decimal-comma parsing, rejection of partially numeric and non-finite input, test-specific sample-size limits, constant samples, affine invariance, and the extreme Anderson–Darling branch A²* ≥ 10.

Independent reference methods

References were calculated with R 4.5.2 using stats::shapiro.test, and with nortest 1.0-4 using nortest::lillie.test and nortest::ad.test. D’Agostino–Pearson was independently compared with scipy.stats.normaltest from SciPy 1.17.0. The reference programs read the seed CSV files directly; Sigma results were emitted separately by the same calculation core loaded by the production page.

IEEE-754 arithmetic and elementary math functions can differ slightly across JavaScript, R, and Python. Bitwise identity was therefore not required; differences and inferential decisions were compared at full available precision without first rounding displayed values.

Validation metrics

For each procedure, the comparator measured the number of finite pairs, mean and maximum absolute differences in the statistic and p-value, disagreement in finite / non-finite status, and decision disagreement at α = 0.05.

Numerical results

Shapiro–Wilk (340 pairs): mean / maximum absolute difference 9.62 × 10⁻¹¹ / 3.95 × 10⁻¹⁰ for W, and 6.34 × 10⁻¹⁰ / 1.01 × 10⁻⁸ for the p-value.
Lilliefors (340 pairs): mean / maximum absolute difference 2.78 × 10⁻⁹ / 8.50 × 10⁻⁸ for D, and 8.72 × 10⁻¹⁴ / 3.09 × 10⁻¹² for the p-value.
Anderson–Darling (340 pairs): mean / maximum absolute difference 5.41 × 10⁻⁷ / 1.52 × 10⁻⁵ for adjusted A²*, and 2.89 × 10⁻¹³ / 3.29 × 10⁻¹¹ for the p-value.
D’Agostino–Pearson (340 pairs): mean / maximum absolute difference 0.0240 / 0.4149 for K², and 1.02 × 10⁻¹⁴ / 6.65 × 10⁻¹³ for the p-value. The largest statistic differences occurred only in deliberately near-degenerate samples with K² ≈ 17,821; both implementations returned p-values numerically equal to zero. Among all non-degenerate scenarios, the maximum absolute K² difference was 2.42 × 10⁻⁹.

Decision agreement at α = 0.05

There were zero decision mismatches and zero estimability-pattern mismatches for every procedure:

Shapiro–Wilk: 0 mismatches
Lilliefors-corrected KS: 0 mismatches
Anderson–Darling: 0 mismatches
D’Agostino–Pearson: 0 mismatches

Implementation-specific notes

The test explicitly labelled Lilliefors is the fitted-normal variant of the KS statistic, not the classical test with externally fixed normal parameters. Anderson–Darling displays the adjusted statistic A²*. Shapiro–Wilk is limited to 3 ≤ n ≤ 5000; Lilliefors requires n ≥ 5; Anderson–Darling reports its statistic from n ≥ 3 and its p-value from n ≥ 8; D’Agostino–Pearson requires n ≥ 8.

Reproducibility

The seed corpus and independent comparison scripts are stored under validation tests/normality. The validation runner loads the production normality_core.js, invokes the unmodified R package functions and SciPy reference, and calculates all differences from full-precision outputs.

Conclusion

Within the validated definitions and sample-size ranges, Sigma showed close numerical agreement with the independent R and SciPy references and identical significance decisions across all 340 estimable datasets. The validation supports the corrected calculation core; it does not replace appropriate judgment about independence, discreteness, sample size, or whether a normality test is scientifically informative for a particular analysis.

Chi-square (χ²) Test (2×2) and Fisher’s Exact Test

Validation objective

The Sigma 2×2 contingency table module was validated against R with respect to the statistical procedures implemented in the tool: Pearson χ² test with Yates’ continuity correction, two-sided Fisher’s exact test, and the associated contingency-table quantities required for these analyses. Sigma computes Yates-corrected χ² p-values, two-sided Fisher p-values, expected cell counts, a low-expected-count flag, and effect estimates for binary outcomes in 2×2 tables.

Reference methods

Validation was performed against standard R implementations for 2×2 table analysis. The χ² comparison used Pearson’s chi-square test with Yates’ continuity correction, and the exact-test comparison used two-sided Fisher’s exact test. In addition, the underlying 2×2 cell counts and expected counts were compared directly between Sigma and R.

Validation dataset

The validation dataset comprised 900 paired analysis cases. All 900 cases could be matched and processed between Sigma and R. The corrected production core and R both classified 771 tables as estimable for the Yates-corrected χ² test and the same 129 degenerate tables as non-estimable. Fisher’s exact test remained estimable in all 900 cases.

Validation metrics

Agreement between Sigma and R was assessed at four levels:

agreement in the number of usable observations,
agreement in the observed 2×2 cell counts,
agreement in expected counts and the low-expected-count flag, and
agreement in χ² and Fisher p-values, including the final significance decision at α = 0.05.

Results of the validation

Agreement between Sigma and R was exact for the basic table structure. The number of usable observations matched in 900/900 cases. The observed 2×2 cell counts matched exactly in 900/900 cases. The low-expected-count flag also matched in 900/900 cases. Expected counts were numerically identical up to floating-point precision, with a maximum absolute difference of approximately 5.0 × 10⁻¹².

Chi-square test validation

For the Yates-corrected χ² test, R returned a comparable finite result in 771 cases. In the remaining 129 cases, both R and Sigma reported no χ² result because a row or outcome margin was zero. Across the 771 comparable cases, the χ² statistic showed a maximum absolute difference of approximately 4.8 × 10⁻¹² between Sigma and R. Direct evaluation of the one-degree-of-freedom χ² survival function reduced the maximum absolute p-value difference to approximately 2.4 × 10⁻¹³.

A separate stress grid comprised 11,562 structured, zero-cell, random, and extreme-count 2×2 tables. Sigma and R agreed on all 291 non-estimable χ² constellations. Across the 11,271 estimable tables, the maximum absolute χ² p-value difference was approximately 3.8 × 10⁻¹⁴, with no significance-decision mismatch.

Fisher’s exact test validation

For Fisher’s exact test, all 900 cases were comparable. The maximum absolute difference in the two-sided Fisher p-value between Sigma and R was approximately 1.8 × 10⁻¹¹. In the additional 2×2 stress grid, the maximum absolute Fisher p-value difference was approximately 4.5 × 10⁻¹³. Fisher probabilities are evaluated on the logarithmic scale and summed with a log-sum-exp formulation, providing stable results even for sparse and large-count tables.

Effect estimates and edge cases

Odds ratios, Wald confidence intervals, risk ratios, and risk differences were compared with the corresponding R formulae throughout the stress grid. For finite comparable estimates, the maximum relative difference was below 5.9 × 10⁻¹⁵. The Haldane–Anscombe correction is applied to the odds ratio and its confidence interval when an observed cell is zero but both group and outcome margins remain positive. Effects are explicitly reported as non-estimable when a group or outcome margin is zero. The interface additionally requires an explicit event-category mapping for every selected outcome; reversing that mapping correctly reverses the event counts and effect direction while leaving χ² and Fisher p-values unchanged.

Decision agreement

At the conventional significance level of α = 0.05, Sigma and R yielded identical inferential decisions for all comparable analyses. There were 0 decision mismatches for the Yates-corrected χ² test and 0 decision mismatches for Fisher’s exact test in both validation suites.

Interpretation

The validation demonstrates that Sigma reproduces the same 2×2 contingency tables as R and yields numerically equivalent results for both Yates-corrected χ² testing and two-sided Fisher’s exact testing. Degenerate margins, zero cells, and explicit event coding are handled transparently rather than being silently converted into apparently valid estimates. The remaining numerical differences are negligible floating-point effects.

Conclusion

The corrected Sigma Chi-square / Fisher module showed exact agreement with R for usable sample sizes, contingency-table cell counts, estimability classification, and low-expected-count classification, together with essentially machine-precision p-value agreement and complete agreement in significance decisions. These results support the validity of Sigma’s implementation of 2×2 Pearson χ² testing with Yates’ correction, two-sided Fisher’s exact test, and the reported binary effect measures for routine analytical use.

Unpaired (Independent Samples) t-Test

Validation objective

The Sigma unpaired t-test module was validated against R with respect to the statistical procedures implemented in the tool: the Welch t-test for unequal variances and the Student t-test with pooled variance under the equal-variance assumption. In addition to p-values, the validation covered group-specific descriptive statistics, mean differences, confidence intervals, and the handling of invalid cases. Sigma reports the Welch two-sided p-value as the primary inferential quantity and additionally computes the pooled-variance t-test, confidence intervals, and standardized effect sizes.

Reference methods

Validation was performed against standard R implementations for independent-samples t-tests. Welch’s test was compared to the unequal-variance formulation in R, and the pooled-variance test was compared to the equal-variance formulation. Group means, standard deviations, sample sizes, mean differences, confidence intervals, and p-values were compared directly between Sigma and R.

Validation dataset

The validation dataset comprised 900 paired analysis cases. Of these, 896 cases were valid in both Sigma and R, while 4 cases were invalid because fewer than two usable observations remained in at least one group after filtering. Sigma classified these invalid cases consistently with the reference results.

Validation metrics

Agreement between Sigma and R was assessed at the following levels:

agreement in the number of usable observations per analysis,
agreement in group-specific descriptive statistics,
agreement in Welch test statistics, degrees of freedom, confidence intervals, and p-values,
agreement in the pooled-variance t-test results, and
agreement in the final significance decision at α = 0.05.

Results of the validation

Agreement between Sigma and R was exact for case validity and usable sample sizes. Among the 896 valid analyses, the number of usable observations matched in all cases. The remaining 4 analyses were consistently classified as invalid because at least one group contained fewer than two usable observations after exclusion of missing or non-numeric values.

Group-specific descriptive statistics were numerically identical up to floating-point precision. The maximum absolute mean difference was approximately 9.6 × 10⁻¹⁵, and the maximum absolute standard-deviation difference was approximately 4.8 × 10⁻¹⁴. These differences are negligible and reflect only standard floating-point behavior.

Welch t-test validation

Welch’s test showed excellent agreement between Sigma and R across all valid cases. The two-sided Welch p-value differed only minimally, with a maximum absolute difference of approximately 7.4 × 10⁻¹². After alignment to the same subtraction direction, the maximum absolute difference was approximately 2.6 × 10⁻¹³ for the test statistic and 7.6 × 10⁻¹³ for a confidence-interval limit.

The sign convention for the mean difference was consistent with the Sigma implementation, which defines the effect as Treatment − Control. When results were transformed to the same direction of comparison, Sigma and R agreed throughout.

Pooled-variance t-test validation

The pooled-variance t-test also showed agreement with the corresponding R implementation. The pooled mean difference, standard error, confidence interval, and p-value matched within negligible numerical tolerance. The maximum absolute pooled-test p-value difference was approximately 7.4 × 10⁻¹¹; the maximum aligned test-statistic and confidence-limit differences were approximately 2.6 × 10⁻¹³ and 6.1 × 10⁻¹⁴, respectively. No systematic deviations were observed.

Distribution, effect-size, and parser stress tests

An additional grid of 99 combinations of t statistics and degrees of freedom, including extreme tails and small degrees of freedom, was compared with R. The maximum absolute difference was approximately 2.2 × 10⁻¹² for two-sided p-values and 1.1 × 10⁻¹² for 95% critical values. Cohen’s d uses the pooled sample standard deviation, and the exact gamma-function correction for Hedges’ g was verified over the same degrees-of-freedom grid; the maximum absolute correction-factor difference was approximately 7.3 × 10⁻¹².

Strict numeric parsing was tested in 11/11 targeted cases covering decimal points, decimal commas, valid grouping separators, malformed partial numbers, non-finite tokens, and quoted CSV fields. Malformed values are treated as non-numeric rather than being partially converted.

Decision agreement

At the conventional significance level of α = 0.05, Sigma and R yielded identical inferential decisions for all valid analyses. There were 0 decision mismatches for the Welch and pooled-variance two-sided tests, including the additional distribution grid.

Interpretation

The validation demonstrates that Sigma reproduces the same independent-samples t-test results as R for both unequal-variance and equal-variance formulations. The exact small-sample correction used for Hedges’ g, strict input parsing, and explicit handling of non-estimable standard errors were also verified. Remaining numerical differences were negligible and are fully compatible with standard floating-point precision and numerical evaluation of the t distribution.

Conclusion

The Sigma unpaired t-test module showed exact agreement with R for valid versus invalid case classification and usable sample sizes, together with near-identical descriptive statistics, mean differences, confidence intervals, and p-values. These results support the validity of Sigma’s implementation of the Welch t-test and the pooled-variance independent-samples t-test for routine analytical use.

Paired t-Test

Validation objective

The Sigma paired t-test module was validated against R with respect to the statistical procedure implemented in the tool: the classical two-sided paired-samples t-test based on within-subject differences. The validation covered the number of complete pairs, descriptive statistics at Time 1 and Time 2, mean differences, test statistics, degrees of freedom, p-values, and confidence intervals. The post-correction validation harness calls the same shared statistical core used by the production page; p-values and confidence limits are therefore validated as direct production-core outputs rather than reconstructed from exported t-statistics. Sigma defines the paired difference as Time 2 − Time 1 and computes the paired t-statistic, confidence interval, and standardized effect sizes on this basis.

Reference methods

Validation was performed against the paired-samples formulation of R’s stats::t.test with paired = TRUE, a two-sided alternative, mu = 0, and a 95% confidence level. Agreement was assessed for the complete-case paired sample used in each analysis and for the corresponding inferential results based on the vector of within-subject differences.

Validation dataset

The validation dataset comprised 900 paired analysis cases. All 900 cases could be matched between Sigma and R. The number of usable complete pairs matched exactly in all 900 cases, and all 900 reference cases were estimable in both implementations.

Validation metrics

Agreement between Sigma and R was assessed at the following levels:

agreement in the number of complete paired observations,
agreement in descriptive statistics for Time 1, Time 2, and the paired differences,
agreement in the paired t-statistic and degrees of freedom,
agreement in p-values and 95% confidence intervals, and
agreement in the final significance decision at α = 0.05.

Results of the validation

Agreement between Sigma and R was exact for the complete paired sample. The number of usable complete pairs matched in 900/900 cases. Descriptive statistics also showed excellent agreement. The maximum absolute differences were approximately 4.89 × 10⁻¹⁵ for each timepoint mean, 8.22 × 10⁻¹⁵ for the mean paired difference, and 3.73 × 10⁻¹⁴ for the standard deviation of the differences.

The paired t-statistic matched R up to machine precision after alignment of the sign convention. The maximum absolute difference in the aligned t-statistic was approximately 3.27 × 10⁻¹³. The corresponding p-values were numerically equivalent, with a maximum absolute difference of approximately 2.80 × 10⁻¹².

Confidence interval validation

The 95% confidence interval for the mean paired difference also showed excellent agreement. After alignment to the same direction of subtraction, the maximum absolute difference in the lower and upper confidence limits was approximately 4.62 × 10⁻¹⁴. These differences are numerically negligible and are fully compatible with ordinary floating-point precision.

Targeted numerical and input-edge tests

The 900-seed campaign was supplemented with deterministic tests of behaviors that are not guaranteed to occur in the generated reference data. These tests cover strict locale-aware numeric parsing, quoted CSV fields, escaped quotes, trailing empty fields, pairwise exclusion of invalid values, subtraction overflow, fewer than two complete pairs, zero and numerically almost-zero variance of differences, and the explicit Time 2 − Time 1 direction.

An extreme-tail R reference with t = 10.878964153196 and df = 99 produced a non-zero two-sided p-value of approximately 1.33613 × 10⁻¹⁸ in the corrected production core. This verifies that the direct incomplete-beta tail calculation avoids the former loss of very small p-values through subtraction from a CDF rounded to one. Cohen’s d_z and the exact gamma-function correction for Hedges’ g_z were checked against deterministic analytic reference values. At df = 1, where the exact correction is undefined, Sigma reports the t-test but leaves g_z blank and emits a warning.

Direction of the paired difference

Interpretation of the paired t-statistic depends on the direction used to define the within-subject difference. Sigma defines the paired difference as Time 2 − Time 1. Accordingly, the sign of the mean difference, t-statistic, and confidence interval is tied to this direction. When Sigma and R results are expressed using the same subtraction order, the paired t-test results agree throughout.

Decision agreement

At the conventional significance level of α = 0.05, Sigma and R yielded identical inferential decisions in 900/900 analyses; the number of decision discrepancies was 0.

Interpretation

The validation demonstrates that Sigma reproduces the same paired-samples t-test results as R for complete paired observations. Remaining differences are negligible at the reported precision and arise from floating-point evaluation of descriptive statistics and Student-t functions. The edge-case suite additionally verifies explicit rejection or reporting of non-estimable inputs instead of silently presenting invalid inference.

Conclusion

The Sigma paired t-test module showed exact agreement with R for the number of complete pairs and near-identical agreement for paired means, mean differences, t-statistics, p-values, and confidence intervals, with no significance-decision discrepancies across 900 reference cases. Together with the targeted numerical and input-edge tests, these results support the validity of Sigma’s implementation of the classical paired-samples t-test, subject to the stated assumptions and complete-case limitations.

One-Way ANOVA — Welch, Classical, Tukey, and Games–Howell

Validation objective

The production calculation core of the One-Way ANOVA module was validated against an independently executed R workflow. The comparison covered data inclusion, group descriptives, classical and Welch omnibus tests, Brown–Forsythe variance testing, eta squared, omega squared, Tukey/Tukey–Kramer comparisons, and Games–Howell comparisons. All comparisons used unrounded full-precision values rather than the values formatted for the user interface.

Reference environment and procedures

The final reference run used R 4.5.2. Classical one-way ANOVA and its sums-of-squares table were obtained with stats::aov; Welch ANOVA used stats::oneway.test(..., var.equal = FALSE). Brown–Forsythe testing applied aov to absolute deviations from the group medians. Tukey estimates, adjusted p-values, and simultaneous confidence intervals used stats::TukeyHSD. Games–Howell results were calculated independently in R from group means, sample variances, Welch–Satterthwaite pairwise degrees of freedom, and stats::ptukey/stats::qtukey.

A common constant was subtracted before the inferential R calculations. Translation leaves every ANOVA, contrast, test statistic, and confidence interval mathematically unchanged and prevents loss of precision in the dedicated 10^9 + x offset stress scenario. Reported means were translated back to their original scale.

Deterministic validation design

Master seed 20260809 generated 90 independent CSV datasets. Sample sizes were n = 10, 30, 60, 120, 300, 800, 1500, 3000, 5000, with 10 replicates at every size. Dataset-specific seeds and configuration metadata are retained in the index. The design produced 770 complete analysis configurations.

Nine scenario types were represented: two-group null data, a two-group mean shift, a three-group equal-variance shift, an unbalanced four-group heterogeneous-variance design, skewed outcomes, rounded ties, missing groups and outcomes, a large numerical offset of (10^9), and five-group heavy-tailed data with an injected outlier. The suite therefore crossed two to five groups with null and alternative effects, balanced and unbalanced sizes, variance equality and inequality, non-normal shapes, missingness, ties, outliers, and numerical stress.

At n = 10, configurations used two or three groups so that every group contained at least three observations and the R studentized-range reference retained pairwise degrees of freedom of at least two. Four- and five-group configurations were included from n = 30 onward. This restriction concerns R-comparable inferential reference values; separate deterministic unit tests cover controlled insufficient- sample and zero-variance states.

Compared outputs

R and Sigma emitted a common long-format result schema. Rows were paired by dataset, analysis configuration, result section, group, pairwise comparison, and statistic. The comparison included:

file rows, usable observations, analyzed groups, exclusions, and result-status labels,
group n, mean, variance, SD, SE, t-based mean interval, quartiles, median, and IQR,
classical sums of squares, degrees of freedom, mean squares, F, p, eta squared, and omega squared,
Welch weighted mean, F, numerator and denominator degrees of freedom, and p-value,
Brown–Forsythe F, degrees of freedom, and p-value,
Tukey contrast, SE, q, degrees of freedom, adjusted p-value, critical value, and simultaneous interval, and
Games–Howell contrast, SE, q, pairwise degrees of freedom, adjusted p-value, critical value, and simultaneous interval.

Numeric acceptance criteria

Structural presence and text/status fields had to match exactly. Statistics not involving the studentized-range approximation used abs(Sigma − R) ≤ 5e−7 + 5e−9 × abs(R). Separate, still sub-reporting-scale criteria were retained for values in which R and Sigma intentionally use independent studentized-range algorithms:

Tukey adjusted p-values: 5e−7 + 1e−8 × abs(R);
Tukey critical values and simultaneous interval limits: 2e−6 + 1e−8 × abs(R);
Games–Howell adjusted p-values: 1e−5 + 1e−8 × abs(R);
Games–Howell critical values: 1.2e−3 + 1e−8 × abs(R); and
Games–Howell simultaneous interval limits: 2.5e−3 + 1e−8 × abs(R).

These separate limits account for the independently implemented studentized-range algorithms. R documents 16-point Legendre integration for ptukey and secant inversion for qtukey, whose quantiles are stated to be accurate to the fourth decimal place; Sigma uses 64-point quadrature and bracketed inversion. Every raw value and difference remains available in the comparison file. In addition to numeric tolerances, the validation required exact agreement of all p-value decisions at α = 0.05 and of the sign of every simultaneous confidence-interval boundary.

Validation results

R and Sigma each produced 104,770 result rows. All 104,770/104,770 keyed comparisons passed, with 0 structural, status, text, or numeric failures. Agreement by section was:

inclusion and preprocessing: 3,850/3,850,
group descriptives: 27,830/27,830,
classical ANOVA and effect sizes: 11,550/11,550,
Welch ANOVA: 6,160/6,160,
Brown–Forsythe: 3,850/3,850,
Tukey/Tukey–Kramer: 24,920/24,920, and
Games–Howell: 26,610/26,610.

All 8,770/8,770 comparisons of p-value decisions at α = 0.05 matched. The signs of all 17,980/17,980 pairwise confidence-interval boundaries also matched, so no comparison differed in whether its interval excluded zero.

Magnitude and source of observed differences

Omnibus p-values agreed particularly closely. The maximum absolute p-value differences were approximately 1.15 × 10⁻¹² for classical ANOVA, 1.30 × 10⁻¹² for Welch ANOVA, and 9.94 × 10⁻¹² for Brown–Forsythe. The maximum Tukey adjusted-p difference was approximately 4.30 × 10⁻⁸; for Games–Howell it was approximately 4.59 × 10⁻⁶.

The largest absolute difference anywhere in the suite was 0.00222164 at a Games–Howell confidence-interval boundary. It arose in an intentionally difficult four-group heterogeneous case with pairwise df = 2.003613. R returned a critical value of approximately 9.77916297, whereas Sigma's direct numerical integration returned approximately 9.77821353. For this pair the mean difference, SE, q statistic, and degrees of freedom agreed to floating-point precision, and the adjusted p-value differed by only approximately 1.83 × 10⁻⁹. The larger CI-scale difference is therefore attributable to the independently implemented low-df quantile calculations, not to a different Games–Howell formula or data cohort.

Absolute mean and quartile differences up to approximately 5.13 × 10⁻⁶ occurred only on the 10^9 outcome scale and correspond to relative differences near machine precision. Numerically centered sums of squares and inferential results remained stable.

Supplementary implementation and edge-case checks

Deterministic JavaScript tests separately verify analytical reference examples, the exact two-group studentized-range/t identity, locale-aware row parsing, large-offset stability, missing-value accounting, singleton groups, constant outcomes, zero group variance, zero pooled within-group variance, non-estimable post-hoc states, and invalid distribution arguments. The browser workflow was exercised with both a complete reference CSV and a file containing missing, non-numeric, and insufficient-group observations; the displayed estimates and controlled warnings matched the calculation core.

Reproducibility

The validation bundle is stored under validation tests/anova/full_validation. It contains the deterministic R seed generator, all CSV seeds, configuration index, independent R reference implementation, Sigma production-core runner, row-level comparator, complete result files, environment record, and text/JSON summaries. Running npm run validate:anova-full regenerates the seeds and both result sets before comparing them. A repeated generation followed by SHA-256 comparison reproduced all 90 seed files and the configuration index byte for byte.

Scope and interpretation

This internally conducted study establishes structural and numerical agreement with the stated R procedures across the tested independent one-way designs. It does not establish that ANOVA is appropriate for a specific scientific dataset, that observations are independent, that a selected outcome has a meaningful mean, or that multiplicity across several outcomes has been addressed. It is not an external software audit or regulatory qualification, and it does not validate methods outside the module, such as repeated-measures, factorial, multilevel, robust-rank, permutation, or covariate-adjusted analyses.

Conclusion

Within the documented scope, Sigma reproduced R's data cohorts, descriptive summaries, classical ANOVA, Welch ANOVA, Brown–Forsythe testing, classical effect sizes, Tukey/Tukey–Kramer inference, and Games–Howell inference across sample sizes from 10 to 5000. All 104,770 numeric/structural comparisons and every inferential decision passed the recorded criteria. The production One-Way ANOVA core is therefore validated for the configurations, assumptions, and limitations described in this report.

Univariable Logistic Regression

Validation objective

Sigma v1.3 was validated against R for the complete univariable production path: pairwise complete-case selection, event coding, automatic and manual predictor typing, treatment and ordinal coding, maximum- likelihood fitting, estimability classification, coefficient inference, and global term inference. Metric, binary categorical, multilevel categorical, and ordinal predictors were included.

Reference methods

R 4.5.2 independently parsed the data, constructed each configured design matrix, and fitted stats::glm.fit(..., family=binomial(link="logit")) with an IRLS tolerance of 10⁻¹² and a 100-iteration limit. The covariance reference was independently recomputed as \((X^\mathsf{T}WX)^{-1}\) at the final R MLE, matching Sigma's documented final-information calculation. This avoids the slight lag that can occur when using the QR matrix cached from R's final working IRLS fit.

Validation corpus

The campaign contained 1,670 configurations over sample sizes from 5 to 5,000. It retained all 900 historical numeric cases and added 760 generated mixed-type configurations plus 10 deterministic edge cases. The generated cases used four sample sizes (30, 100, 500, and 2,000) with ten repetitions per stochastic scenario. In total, the corpus represented 1,341,256 source rows before configuration-specific complete-case selection.

Scenarios covered null, moderate, and strong effects; normal, uniform, skewed, and binary numeric inputs; two-, three-, and five-level factors; balanced, imbalanced, and sparse levels; linear and deliberately nonlinear ordinal truths; custom ordinal order; reference changes; 0/1/2 interpretation overrides; European decimal/thousands separators; missing tokens; malformed numeric cells; constant predictors; absent outcome classes; and complete/quasi-complete separation.

Validation metrics

Agreement between Sigma and R was assessed at the following levels:

row counts, complete-case inclusion, events, non-events, event rate, and exclusion reasons,
automatic type/reason, manual override, level order, reference, and encoded column count,
estimable versus non-estimable status,
coefficients, standard errors, Wald z-statistics, p-values, odds ratios, and 95% Wald intervals,
global term degrees of freedom, Wald statistics, and p-values, and
reference-change and ordinal/metric coding invariants.

Overall results

R and Sigma each produced exactly 68,206 registered result rows. All 68,206/68,206 structural and numerical comparisons passed. The inclusion section passed 18,370/18,370 rows, design/coding 21,710/21,710, fit status 1,670/1,670, coefficient results 19,776/19,776, and global term results 6,680/6,680.

Estimability and inferential decisions

Both implementations classified exactly 1,355 configurations as estimable and 315 as not estimable. Thus every status decision agreed, including absent outcome classes, constant predictors, zero-column categorical terms, and complete or quasi-complete separation.

All 3,088/3,088 p-value decisions at \(\alpha=0.05\) matched. All 3,466/3,466 confidence-limit decisions relative to the null odds ratio of 1 also matched.

Numerical agreement

Across estimable models, the maximum absolute coefficient difference was 2.3054 × 10⁻¹⁰; standard error, 2.2178 × 10⁻⁹; Wald z, 5.9185 × 10⁻¹¹; coefficient/one-degree-of-freedom p-value, 1.0888 × 10⁻¹¹; global Wald statistic, 2.1251 × 10⁻⁹; and global p-value, 1.0888 × 10⁻¹¹.

Exponentiation can turn minute log-scale differences into large absolute differences. The largest absolute upper confidence-limit difference was 1.0252 × 10³² in an astronomically large interval; its scaled relative difference was only 4.5775 × 10⁻⁹. This was also the maximum scaled relative difference anywhere in the campaign.

Results by predictor interpretation

Every requested-type stratum passed: automatic interpretation 12,080/12,080 rows, metric 40,922/40,922, categorical 9,247/9,247, and ordinal 5,957/5,957. Every validation suite also passed: historical numeric 33,300/33,300, generated mixed type 34,512/34,512, and deterministic edge cases 394/394.

The design comparison verified the exact auto-detection reason, effective type, ordinal-candidate flag, deterministic levels, complete-case levels, reference, ordinal order, numeric ratio, and model-column count for every configuration—not only the final regression numbers.

Configurable missing-value validation

A separate independent R campaign targeted the shared missing-value layer used by both regression tools. Seven deterministic configurations covered standard missing tokens, English and German spreadsheet errors, user-defined missing codes, explicit inclusion overrides, suspicious hash-prefixed categories, automatic interpretation of 0/1/2, categorical and ordinal coding, locale-aware metric parsing, outcome exclusions, and multivariable listwise deletion. R reconstructed the policy and design directly from raw character cells before fitting stats::glm.fit.

All 717/717 registered structural and numerical comparisons passed. This included row inclusion, reason-specific exclusion counts, detected and effective type, levels, references, ordinal order, encoded model columns, coefficients, standard errors, Wald statistics, p-values, odds ratios, and confidence intervals. The maximum absolute numerical difference was 3.997×10⁻¹².

Coding and reference invariants

A separate production-only suite passed 675/675 invariance assertions. Of these, 235 tested treatment-reference status, global Wald invariance, coefficient sign reversal, pairwise log-odds reconstruction, and reciprocal odds ratios. The remaining 440 verified automatic categorical handling of 0/1/2, manual metric and ordinal overrides, contrast-column count, and equality between ordinal scores 0/1/2 and equivalent metric coding. One separated factor pair had no finite MLE; its matching non-estimable status passed while numerical reparameterization was correctly omitted.

Acceptance criteria

Each comparison row records its tolerance. Structural quantities used 10⁻¹² + 10⁻¹²|R|; coefficients 2×10⁻⁷ + 2×10⁻⁸|R|; standard errors and z statistics 5×10⁻⁶ + 2×10⁻⁷|R|; p-values 5×10⁻⁸ + 2×10⁻⁷|R|; exponentiated estimates 5×10⁻⁶ + 5×10⁻⁷|R|; and global Wald statistics 10⁻⁵ + 5×10⁻⁷|R|. Observed differences remained substantially below these limits.

Reproducibility and limitations

The full audit package is stored under validation tests/unilogit/full_validation. It contains the case generator, complete seed/configuration index, R environment, independent R runner, Sigma production runner, row-level comparison, machine-readable summaries, invariant checks, and a detailed report. Both comparators exit with a non-zero status on failure. The focused missing-value audit is stored under validation tests/logistic_missing_values. The campaign validates the implemented classical binary-logit MLE; it does not validate causal interpretation, dataset-specific model specification, penalized/Firth regression, splines, clustered covariance, repeated measures, or mixed-effects models.

Conclusion

Within its documented scope, Sigma v1.3 reproduced R for metric, binary categorical, multilevel categorical, and ordinal univariable logistic regression. All 68,206 registered comparisons, all estimability classifications, all inferential decisions, all 675 coding/reference invariants, and all 717 focused missing-policy checks passed. The production univariable module is therefore validated at R-equivalent numerical accuracy for the configurations and limitations documented here.

Multivariable Logistic Regression

Validation objective and reference

Sigma v1.3 was subjected to a full production-path validation of multivariable binary logistic regression against R 4.5.2. The independent reference used stats::glm.fit(..., family = binomial(link = "logit")). Validation covered the complete analysis chain: outcome mapping, automatic and manual predictor typing, strict numeric parsing, listwise deletion, metric/ordinal/treatment coding, reference levels, design-matrix columns, constant and aliased columns, estimability, intercept and adjusted coefficients, final Fisher covariance, standard errors, Wald z tests, odds ratios, 95% Wald intervals, per-term joint Wald tests, and the model-wide Wald test. Full-precision values were compared rather than displayed rounded values.

Validation corpus

The reproducible campaign contained 2,613 configured analyses:

900 historical numeric models with three predictors,
900 historical numeric models with five predictors,
800 newly generated mixed-type and reparameterized models, and
13 deterministic edge cases.

Sample sizes ranged from 5 to 5,000. The stochastic designs included null, moderate, strong and mixed adjusted effects; independent, correlated, highly correlated and exactly dependent predictors; rare events; leverage points; missing outcomes and predictors; malformed numeric tokens; European decimal formatting; binary and multilevel factors; three- to five-level ordinal terms; non-linear ordinal truth; alternate treatment references; and predictor scaling by 10⁻⁹ and 10⁹. Edge cases included a single outcome class, no complete rows, all-constant designs, post-listwise constants, exact and near aliases, duplicated factors, saturated models, complete linear-combination separation, and pure factor-level quasi-separation.

Independent reference workflow

R independently reconstructed each design from the raw character cells and the recorded configuration. The analysis sample used listwise complete cases across the outcome and all selected predictors. Metric terms contributed one numeric column, ordinal terms one score column with scores 0, …, L−1, and categorical terms L−1 treatment columns. The reference then applied the documented deterministic constant/rank policy and estimated the retained model with glm.fit. Covariance was recomputed from Fisher information at the final R estimate.

Existence of a finite maximum-likelihood estimate was tested before accepting inference. For signed design rows a_i = (2y_i−1)x_i, a bounded linear program tested whether a non-zero direction satisfies a_i^Tb ≥ 0 for every observation with at least one strict inequality. This covers complete and quasi-separation, including separation produced by a combination of covariates or a pure categorical level. Both R and Sigma therefore classify mathematical non-existence consistently rather than relying on implementation-specific IRLS warnings.

Acceptance criteria

Every long-format result row carried an absolute-plus-relative tolerance. Structural fields used 10⁻¹² + 10⁻¹²|R|; coefficients 3×10⁻⁷ + 3×10⁻⁸|R|; standard errors and z statistics 8×10⁻⁶ + 3×10⁻⁷|R|; p-values 8×10⁻⁸ + 3×10⁻⁷|R|; exponentiated estimates 8×10⁻⁶ + 8×10⁻⁷|R|; and joint Wald statistics 2×10⁻⁵ + 8×10⁻⁷|R|. Missing/non-estimable values and all text classifications had to agree exactly. No comparison was accepted merely because its rounded display matched.

Results

R and Sigma produced 347,905 registered comparison rows each. All 347,905 comparisons passed. Results by validation suite were:

historical three-predictor numeric suite: 98,100 / 98,100,
historical five-predictor numeric suite: 148,500 / 148,500,
new mixed-type suite: 99,840 / 99,840, and
deterministic edge cases: 1,465 / 1,465.

Estimability classifications were identical: 1,966 models had a finite accepted MLE and 647 were non-estimable. Every section passed: inclusion 42,875 / 42,875, term design 116,148 / 116,148, design columns 20,638 / 20,638, model structure 20,904 / 20,904, intercept 10,452 / 10,452, coefficients 87,720 / 87,720, term tests 38,716 / 38,716, and model tests 10,452 / 10,452.

All 18,247 / 18,247 inferential p-value decisions at α = 0.05 and all 14,872 / 14,872 odds-ratio confidence-boundary decisions matched. The maximum scaled relative numerical difference was 5.47×10⁻⁹. Absolute differences can be very large for deliberately extreme but finite exponentiated estimates, so acceptance used the declared combined absolute/relative criterion; non-finite exponentiation was consistently reported as not representable.

Configurable missing-value validation

The shared focused R audit additionally exercised listwise analysis with standard missing tokens, localized spreadsheet errors, user-defined missing codes, explicit inclusion overrides, suspicious hash-prefixed categories, low-cardinality 0/1/2 detection, categorical and ordinal coding, and locale-aware numeric parsing. Across its seven univariable and multivariable configurations, all 717/717 structural and numerical comparisons passed; the maximum absolute numerical difference was 3.997×10⁻¹².

Independent invariant checks

A second audit tested mathematical relationships without using R values as expected answers. All 3,393 / 3,393 invariants passed. These covered invariance of adjusted model and term tests under factor-reference changes; algebra of reparameterized contrasts; equivalence of ordinal scores 0/1/2 and explicit metric coding in the same adjusted model; preservation of all unaffected coefficients; coefficient, standard-error, z, p and joint-Wald behavior under 10⁻⁹/10⁹ scaling; deterministic constant/alias selection; and rejection of separated, saturated, single-class and zero-column models. One separated reference pair had no finite numerical reparameterization to compare; its matching structural status remained part of the passing checks.

Corrections identified by the campaign

The full audit identified and corrected three production-level edge conditions: multivariable complete/quasi-separation is now established explicitly by the linear-program criterion; joint Wald tests are solved through the covariance-derived correlation matrix so that they are invariant to predictor units; and mixed numeric/text level labels now have a transitive deterministic order (numeric labels first, followed by lexical text labels). Targeted regression tests cover each correction.

Reproducibility and limitations

The complete audit package is stored under validation tests/multilogit/full_validation. It contains the seeded corpus generator, case index, independent R reference, exact Sigma production runner, memory-bounded comparator, row-level results, diagnostics, invariant checks, environment record, machine-readable summaries, and a detailed report. The comparators return a non-zero exit status on any failure. The focused missing-value evidence is stored under validation tests/logistic_missing_values.

This campaign validates the implemented classical fixed-effects binary-logit MLE and Wald inference. It does not validate causal interpretation, dataset-specific model selection, interactions or non-linear terms not explicitly created by the user, penalized/Firth or exact logistic regression, clustered/robust covariance, repeated measures, or mixed-effects models.

Conclusion

Within this documented scope, Sigma v1.3 reproduced R for multivariable models containing metric, binary categorical, multilevel categorical and ordinal predictors. All registered structural and numerical comparisons, all estimability decisions, every tested inferential decision, and all independent invariants and focused missing-policy checks passed. The production multivariable logistic-regression path is therefore validated at R-equivalent numerical accuracy for the configurations and limitations stated here.

Propensity Score Matching (PSM)

Validation objective

The production JavaScript core of the Sigma Propensity Score Matching module was compared directly with an independently executed R reference workflow. The comparison covered strict preprocessing, treatment and binary-variable coding, logistic propensity-score estimation, deterministic ATT-oriented greedy nearest-neighbor matching, calipers, exact pair identity, all displayed balance values, and overlap diagnostics. Display rounding was not used; comparisons used the underlying full-precision values.

The validation also targeted the numerical defect corrected in this release: fitted propensity scores and matches must remain invariant when numeric covariates are expressed in very small or very large units.

Reference environment and methods

The final reference run used R 4.5.2 and MatchIt 4.7.2. Logistic models were fit independently with R glm.fit(..., family = binomial()). Matching used MatchIt::matchit(method = "nearest", estimand = "ATT", ratio = 1, replace = FALSE, m.order = "largest") with the R-fitted propensity scores supplied as the distance. Calipers and standardization settings were passed identically. When a valid caliper yielded no pairs, MatchIt's “No units were matched” condition was normalized to the same valid empty matched set reported by Sigma.

Every displayed balance statistic was recomputed independently in R from the matched indices using the same stated definitions. Exact pairs were compared by treated and control identifiers rather than by the presentation order of MatchIt's match matrix.

Deterministic validation design

Master seed 20260808 generated 180 seed datasets. Sample sizes were n = 10, 30, 60, 120, 300, 800, 1500, 3000, 5000, with 10 repetitions at each size in each of two cohort structures: approximately balanced treatment groups and an approximately 5:1 control-to-treated distribution. Dataset-specific seeds are retained in the archive. Good, moderate, and deliberately limited-overlap scenarios were represented.

The generated data included metric covariates, text-coded binary variables (F/M), binary numeric labels other than 0/1 (1/2), multi-level categorical covariates, deterministic missing values, and invalid numeric tokens. Each dataset was evaluated with three model scales:

the original numeric covariates,
all numeric covariates multiplied by 10⁻⁹, and
all numeric covariates multiplied by 10⁹.

This produced 540 logistic model comparisons. Six matching configurations were evaluated per dataset, for 1,080 matching comparisons:

standardized PS calipers 0.05, 0.20, and 0.50 on the base model,
an absolute PS caliper of 0.20 on the base model, and
a standardized PS caliper of 0.20 for each rescaled model.

Validation metrics

The final comparator evaluated 1,816,729 full-precision result rows, including:

included and excluded rows, exclusion reasons, binary mappings, and design-matrix columns,
convergence, estimable rank, rank deficiency, deviance, propensity scores, and linear predictors,
matching status and counts, effective calipers, exact treated–control identities, and pair distances,
every pre- and post-match mean/proportion, ATT-standardized SMD, and descriptive z-difference, and
pre- and post-match overlap ranges, outside-support proportions, and warnings.

Text, mappings, counts, rank, and pair identities required exact equality. Numerical tolerances were 2×10⁻⁸ for propensity scores, pair distances, overlap, and effective calipers; 2×10⁻⁶ for linear predictors; 2×10⁻⁹ relative to magnitude for balance values; and 2×10⁻⁷ for deviance. Iteration counts were retained as algorithmic diagnostics but were not required to be identical. Scale invariance required a maximum propensity-score difference of 5×10⁻¹³ and exact pair identity.

Results

The complete validation passed without an unmatched or missing reference row and without a recorded mismatch. The pass counts were:

540/540 logistic model comparisons,
1,080/1,080 matching comparisons,
1,080/1,080 matching runs with exact treated–control pair identity, and
360/360 scale-invariance comparisons.

These totals include runs in which a strict caliper correctly yielded zero pairs. Such runs were compared as valid empty matched sets, not discarded as failed analyses.

Numerical agreement

Maximum absolute differences were 3.22×10⁻¹⁵ for propensity scores, 3.89×10⁻¹⁵ for pair distances, and 2.94×10⁻¹⁵ for overlap quantities. The maximum propensity-score difference between an original model and either rescaled Sigma model was 1.50×10⁻¹⁵.

The largest absolute balance difference, 1.53×10⁻⁵, occurred for a displayed post-match mean of approximately 6.48×10¹⁰ in a deliberately 10⁹-rescaled dataset; its relative difference was only 2.35×10⁻¹⁶. Across balance values, the largest relative difference was 9.05×10⁻¹³. The apparent absolute maximum is therefore a unit-scale artifact at machine precision, not a scientific discrepancy.

Balanced and 5:1 cohorts

Both the approximately balanced cohorts and the 5:1 control-dominant cohorts were included in every sample-size tier and all three overlap scenarios. Agreement was complete for preprocessing, model output, matched counts, exact pairs, balance, and overlap in both cohort structures. This directly replaces the earlier validation that compared only aggregated max-|SMD| summaries and a smaller number of available reference exports.

Scale invariance

Internal standardization removed the earlier dependence on raw predictor units. For all 360/360 original-versus-rescaled comparisons, the fitted Sigma propensity scores remained within the prespecified 5×10⁻¹³ tolerance and the treated–control pairs were identical. The observed maximum was 1.50×10⁻¹⁵.

Interpretation and scope

The results support numerical equivalence of the implemented complete-case, 1:1 nearest-neighbor PSM workflow to the specified R/MatchIt reference under the tested conditions. They do not establish that a chosen propensity-score model is causally sufficient, that overlap is adequate for a particular research question, or that unmeasured confounding is absent. Those remain study-design judgments. Likewise, outcome inference after matching must account for the paired matched structure and is outside this module.

Conclusion

Across 180 deterministic datasets, 540 model comparisons, 1,080 matching comparisons, and 1,816,729 full-precision result rows, Sigma showed no validation mismatch against R 4.5.2 and MatchIt 4.7.2. Exact pair identity was obtained in every matching run, and predictor rescaling by factors of 10⁻⁹ and 10⁹ did not change the matched design.

These findings support the corrected implementation for scientific use as a transparent design-stage tool for complete-case, ATT-oriented, 1:1 nearest-neighbor propensity-score matching without replacement, provided that model specification, overlap, measured-confounder selection, and the subsequent paired outcome analysis are assessed by the researcher.

Survival Analysis — Kaplan–Meier, Log-Rank, RMST, and Cox Regression

Validation objective

The Survival Analysis module was validated by comparing the production JavaScript calculation core directly with an independently executed R reference workflow. Validation covered preprocessing and exclusions, Kaplan–Meier estimation, reverse Kaplan–Meier median follow-up, fixed-time survival estimates, the omnibus log-rank test, restricted mean survival time (RMST), Cox proportional hazards regression, robust inference, stratified baseline hazards, concordance, and proportional-hazards diagnostics. Display rounding was not used in the comparison; all tests used the underlying full-precision values.

Reference environment and methods

The final reference run used R 4.5.2 with survival 3.8.3. Reference calculations used survfit with complementary log-log confidence limits, reverse-event survfit for median follow-up, survdiff(rho = 0) for the log-rank test, the restricted-mean output from summary.survfit, coxph for Cox models, concordance for Harrell's C, and cox.zph(transform = "km", terms = TRUE, global = TRUE) for term-level and global PH diagnostics. Efron and Breslow tie handling, model-based and sandwich covariance, clustering, and Cox strata were passed to the corresponding R procedures according to each configuration.

Deterministic validation design

Master seed 20260803 generated 90 independent seed datasets. The tested sample sizes were n = 10, 30, 60, 120, 300, 800, 1500, 3000, 5000, with 10 replicates at every size. Dataset-specific seeds are retained in the configuration index. The design produced 1,800 analysis configurations: 360 nonparametric and 1,440 Cox configurations.

The generated data included continuous follow-up, deliberately tied event times, heavy censoring, two- and three-level groups, missing group assignments, missing predictors, metric and categorical predictors, user-selected categorical reference levels, cluster identifiers, Cox strata, and deliberately non-proportional effects. Small datasets used lower-dimensional models where necessary to avoid making singularity inevitable, while the complete advanced-option factorial remained represented at every sample size.

Nonparametric coverage

Four nonparametric configurations were evaluated for each seed. They covered Overall, two-group, and three-group Kaplan–Meier estimates; risk sets, events, censoring, survival, Greenwood standard errors and confidence limits at every observed step; survival at requested time points; median survival and its interval; reverse-KM median follow-up; log-rank chi-square, degrees of freedom, p-value, and group-specific observed and expected events; and RMST/RMTL estimates, standard errors, intervals, differences, p-values, and ratios at a common restriction time.

Cox coverage

The principal Cox factorial crossed Efron versus Breslow ties, model-based versus subject-robust versus cluster-robust variance, and unstratified versus stratified baseline hazards. Additional configurations exercised purely metric models, common complete-case analysis with missing predictors, custom categorical references, and deliberately non-proportional categorical effects. Every Cox configuration included the multivariable model and common-cohort univariable models.

Compared outputs included coefficients, standard errors, z statistics, p-values, hazard ratios and confidence intervals; model-based and robust covariance matrices; null and fitted log partial likelihoods; AIC; likelihood-ratio, Wald, and score tests; overall factor tests; convergence metadata; Harrell's C, its standard error and comparable-pair count; and term-level and global PH tests.

Structural matching and numeric criteria

R and Sigma emitted the same long-format schema. Every row was matched in order by dataset, configuration, result section, model, cohort, term, statistic, and evaluation point. Presence and text values had to match exactly. Closed-form and nonparametric results, covariance matrices, concordance, and PH diagnostics used the strict criterion abs(Sigma − R) ≤ 5e−7 + 5e−7 × abs(R).

Cox coefficient, model, and term-test sections depend on separate iterative solvers and floating-point accumulation order. These sections therefore used abs(Sigma − R) ≤ 1e−5 + 1e−5 × abs(R). This tolerance is applied to full-precision values and is substantially smaller than ordinary reporting precision. The raw R value, Sigma value, absolute difference, scaled relative difference, and pass decision are retained for every comparison; the tolerance does not alter or round either program's result.

Validation results

R and Sigma each produced 3,833,940 result rows. All 3,833,940/3,833,940 keyed comparisons passed, with 0 structural, text, presence, or numeric failures. Agreement by result section was:

analysis metadata: 11,880/11,880,
Kaplan–Meier step rows: 3,457,700/3,457,700,
Kaplan–Meier summary and fixed-time rows: 48,510/48,510,
log-rank and group-detail rows: 3,330/3,330,
RMST estimates and comparisons: 10,800/10,800,
Cox coefficients: 76,720/76,720,
Cox model and term statistics: 154,880/154,880,
model-based and robust covariance rows: 52,600/52,600, and
PH diagnostics: 17,520/17,520.

The maximum absolute numeric difference was approximately 3.6261 × 10⁻⁴. It occurred for a Cox Wald chi-square statistic of approximately 117.3 in a dataset with n = 5000; its scaled relative difference was approximately 3.09 × 10⁻⁶. Across all values, the maximum scaled relative difference was approximately 5.34 × 10⁻⁶. Thus even the largest observed solver-related difference remained below the iterative Cox criterion and far below meaningful displayed precision.

Supplementary implementation checks

The Survival Analysis core additionally passed 22/22 deterministic JavaScript unit tests covering input parsing, distribution functions, near-tied time correction, Kaplan–Meier edge cases, RMST, log-rank calculations, Cox risk sets and ties, strata, robust covariance, PH diagnostics, and controlled error states. The optimized concordance implementation was compared with direct quadratic pair enumeration across 250 deterministic randomized small datasets containing time ties, prediction ties, strata, and clusters. Comparable-pair counts, concordance estimates, and standard errors agreed in every check.

Reproducibility

The complete validation bundle is stored under validation tests/survival_analysis/full_validation. It contains the deterministic seed generator, configuration index, R reference implementation, Sigma production-core runner, streaming row-level comparator, generated results, environment record, and concise summary. Running run_validation.sh rebuilds all seeds and both result sets before performing the comparison. The final one-command reproduction completed successfully with the counts reported above.

Scope and interpretation

This study establishes numerical equivalence to the stated R reference procedures across the tested right-censored, single-event scope. It does not validate methods that the module does not implement, including delayed entry, start-stop records, time-varying covariates, interactions, splines, frailty, recurrent events, interval censoring, or weighted and survey analyses. Simulation-based cross-validation cannot establish that a chosen model is scientifically appropriate for a particular dataset, and this internally conducted validation is not a substitute for an external software audit or regulatory qualification.

Conclusion

Within the documented scope, the validation demonstrates exact structural agreement and high-precision numerical equivalence between Sigma and R for Kaplan–Meier estimation, log-rank testing, RMST, Cox regression, robust and cluster-robust inference, stratified baseline hazards, concordance, and PH diagnostics over sample sizes from 10 to 5000. The production calculation core is therefore validated for the configurations, assumptions, and acceptance criteria described in this report.

Competing Risk Analysis — CIFs, Gray Test, and Fine & Gray Regression

Validation objective

The production calculation core of the Sigma competing-risk module was validated against an independent R reference for all analytical components implemented in the tool: non-parametric cumulative incidence functions (CIFs), their Aalen variances and pointwise confidence intervals, Gray’s two-group test (ρ = 0), and binary Fine & Gray proportional subdistribution-hazards regression. The comparison also covered preprocessing, pooled and group-specific censoring weights, both possible reference groups, model convergence, and controlled non-estimability.

Reference implementation

The reference calculations used R 4.5.2 and cmprsk 2.2-12. CIFs and Gray tests were obtained with cmprsk::cuminc; CIF estimates and variances were evaluated at every observed event time with cmprsk::timepoints. Fine & Gray models were fitted with cmprsk::crr using failcode = 1 or 2, cencode = 0, gtol = 10⁻⁶, at most ten Newton updates, and robust variance estimation. Comparisons used full-precision values rather than rounded display output.

Validation design

The deterministic suite contained 900 datasets: six nominal sample sizes (n = 5, 10, 20, 100, 1000, 5000), 15 scenarios per sample size, and ten independent replicates per scenario. The scenarios covered no group effect, effects on either cause, high and low censoring, rare causes, 80:20 group imbalance, differing censoring patterns between groups, rounded and integer ties, missing time, missing status, a piecewise non-proportional cause-1 effect, and a combined high-censoring / rare-cause / ties stress setting.

For every dataset, Sigma and R evaluated overall and group-specific CIFs for every detected cause at every distinct event time. Gray’s test was requested for causes 1 and 2. Fine & Gray regression was evaluated for both causes, with pooled and group-specific censoring distributions, and with Control and Treatment each used as the reference group. This produced:

2,089,198 paired CIF time points across 4,905 curves,
1,800 paired Gray-result rows, and
7,200 paired Fine & Gray result rows.

Rows with missing time or status were removed under the same rules on both sides. Numeric comparisons were performed only when both implementations classified a result as estimable and converged; all non-numeric status classifications were compared separately and required to match exactly.

Predefined acceptance criteria

Before the final comparison, the following absolute-difference limits were specified:

CIF time grid: 1 × 10⁻¹² and CIF estimate, variance, SE, and confidence limits: 1 × 10⁻¹⁰;
Gray statistic: 1 × 10⁻⁸ and Gray p-value: 1 × 10⁻¹²;
Fine & Gray coefficient, robust SE, z statistic, and p-value: 1 × 10⁻⁸;
reference-group symmetry checks: 1 × 10⁻¹⁰; and
exact agreement for dataset structure, detected causes, and all result-status classifications.

Cumulative incidence functions

All 2,089,198 CIF points were paired on identical file, scope, group, cause, and time keys; there were 0 key or time-grid mismatches. Across the 4,905 curves, the maximum curve-wise absolute differences were 5.55 × 10⁻¹⁶ for the CIF, 1.40 × 10⁻¹⁵ for the Aalen variance, 9.91 × 10⁻¹⁴ for the SE, 2.18 × 10⁻¹³ for the lower confidence limit, and 1.72 × 10⁻¹³ for the upper confidence limit. All results were well inside the predefined CIF tolerance.

Gray’s test

Status classifications matched in all 1,800/1,800 rows: 1,677 paired results were estimable, 44 were classified as not estimable, and 79 causes were not detected. Among the 1,677 estimable pairs, the maximum absolute difference was 3.85 × 10⁻¹¹ for the test statistic and 2.51 × 10⁻¹³ for the p-value; degrees of freedom matched exactly.

Fine & Gray regression

Status classifications matched in all 7,200/7,200 rows. The paired counts were 5,606 converged and estimable models, 1,032 non-estimable models, 246 non-converged models, and 316 rows in which the requested cause was not detected. Observation counts, total target-cause events, and events in both reference and comparison groups matched exactly.

Across the 5,606 estimable pairs, the maximum absolute differences were 1.41 × 10⁻¹³ for the coefficient, 9.19 × 10⁻¹⁵ for the robust SE, 2.64 × 10⁻¹² for the z statistic, and 3.72 × 10⁻¹⁴ for the p-value. The derived SHR, lower confidence limit, and upper confidence limit had maximum absolute differences of 1.13 × 10⁻¹³, 1.04 × 10⁻¹³, and 1.71 × 10⁻¹², respectively.

Reversing the binary reference group produced 2,803 estimable model pairs. The largest absolute coefficient sum was 1.63 × 10⁻¹³, the largest SE difference was 5.70 × 10⁻¹⁵, the largest p-value difference was 8.32 × 10⁻¹³, and the largest deviation of the reciprocal SHR product from one was 1.63 × 10⁻¹³.

Reproducibility

The reproducibility bundle is stored under validation tests/competingrisk/revalidation_v2. It contains the Sigma runner, the independent R runner, the comparison script, per-shard environment records, full row-level outputs, curve-level CIF comparisons, status comparisons, sample-size summaries, and the machine-readable final summary. The seed index in validation tests/competingrisk/Data/cases_index.csv records the scenario, replicate, and deterministic seed for every dataset.

Interpretation and limitations

The suite demonstrates exact structural and status agreement and numerical equivalence to the stated R reference over the validated designs, including very small samples, sparse causes, ties, missing time or status, unequal groups, high censoring, and sample sizes up to 5000. Agreement of a numerical result does not make a model scientifically appropriate: very small event counts can still make large-sample inference unreliable, and the Fine & Gray proportional subdistribution-hazards assumption must be judged for the study at hand. The non-proportional scenario validates numerical agreement under that input pattern; it does not validate proportional-hazards interpretation when the assumption is violated. Likewise, scenarios with differing censoring patterns test computation and do not remove the requirement for independent censoring within the selected censoring model.

This is an internally conducted numerical cross-validation against one specified reference implementation. It is not an external software audit, regulatory qualification, or proof that every possible dataset, operating environment, or scientific use case has been validated.

Conclusion

All predefined acceptance criteria passed. Within the documented scope, Sigma reproduces cmprsk results to high numerical precision for CIF estimation and uncertainty, Gray’s test, and binary Fine & Gray regression with pooled or group-specific censoring and either reference group. The revised production core is therefore validated for the configurations and definitions described in this report.