Statistical Agnostic Regression: A Machine Learning Method to Validate Regression Models

Authors

Affiliations

Juan M. Gorriz

Dept. of Psychiatry, University of Cambridge

DaSCI Institute, University of Granada

ibs.Granada

Javier Ramirez

DaSCI Institute, University of Granada

ibs.Granada

Fermin Segovia

DaSCI Institute, University of Granada

ibs.Granada

Carmen Jimenez-Mesa

DaSCI Institute, University of Granada

ibs.Granada

Francisco J. Martinez-Murcia

DaSCI Institute, University of Granada

ibs.Granada

John Suckling

Dept. of Psychiatry, University of Cambridge

Abstract

Regression analysis is a central topic in statistical modeling. Classical methods for solving linear regression problems, such as Ordinary Least Squares (OLS), often form the foundation for more advanced machine learning (ML) techniques. However, ML models have been successfully applied without a formal definition of statistical significance, often relying on empirical measures or permutation testing on small samples. In this paper, we introduce Statistical Agnostic Regression (SAR) for evaluating the statistical significance of ML-based linear regression models. This is achieved by analyzing concentration inequalities of the actual risk (expected loss) and considering the worst-case scenario. SAR introduces a threshold ensuring evidence of a linear relationship in the population with a high probability under non-parametric assumptions. Simulations demonstrate that SAR emulates the classical multivariate F-test without relying on classical assumptions (like Gaussianity or homoscedasticity) and effectively controls the overinflated false positive rate found in standard ML methods like K-fold cross-validation.

1. THE PROBLEM: IS CROSS-VALIDATION FAILING US?

Ordinary Least Squares (OLS) is the gold standard for linear regression due to its optimal statistical properties. On the other hand, modern Machine Learning (ML) techniques—such as Ridge, Lasso, and Support Vector Regression (SVR)—excel at minimizing expected loss and handling complexity.

However, current AI approaches share a critical flaw: a lack of rigorous statistical significance analysis. Most ML researchers rely on empirical measures like \(K\)-fold cross-validation (CV) or permutations to evaluate their models. As highlighted in recent neuroimaging literature, utilizing CV on limited sample sizes can lead to massive variability and an inflated false positive rate, finding spurious correlations where none exist.

2. THE SOLUTION: STATISTICAL AGNOSTIC REGRESSION (SAR)

To bridge the gap between classical OLS statistics and advanced ML methods, this work introduces Statistical Agnostic Regression (SAR).

SAR provides a formal, non-parametric statistical test for assessing the significance of ML regression models. Rooted in Statistical Learning Theory (SLT) and PAC-Bayesian theory, SAR establishes an upper bound on the actual risk (expected loss) of a linear support vector regressor under the worst-case scenario.

By comparing this bounded actual risk to the expected loss under the null hypothesis \(H_0\) (no linear relationship), SAR calculates a robust threshold. If the corrected risk falls below this threshold, \(H_0\) is rejected, providing mathematical evidence of a true linear relationship with a probability of at least \(1-\eta\).

KEY ADVANTAGES OF SAR

• Agnostic & Non-Parametric: Does not rely on rigid classical assumptions like Gaussianity or homoscedasticity.
• Controls False Positives: Radically outperforms K-fold and Leave-One-Out (LOO) CV in controlling Type I errors in small sample sizes.
• Robust to Heteroscedasticity: Through the Breusch-Pagan test, SAR residuals prove superior in detecting varying variance in data.
• Bridging the Gap: Provides a perfect trade-off between the stability of OLS and the powerful fitting capabilities of ML methods.

3. EXPERIMENTAL VALIDATION & NEUROIMAGING APPLICATIONS

The SAR framework was rigorously tested across three distinct scenarios:

1. Synthetic Gaussian & Non-Gaussian Data: Evaluated against simulated datasets featuring scaling, rotation transforms, and forced heteroscedasticity.
2. Multivariate Cancer Dataset: Real-world socio-economic predictors tested against cancer mortality rates.
3. The ADNI Dataset (Alzheimer’s Disease): Evaluating the correlation between cognitive decline (MMSE scores) and 3D structural MRI brain changes (Grey Matter) across Healthy Controls (NC), Mild Cognitive Impairment (MCI), and Alzheimer’s (AD) patients.

Figure 1: Graphical Abstract: The SAR framework methodology bridging OLS and ML regressions.

3.1. RESULTS: THE ADNI DATASET

When applying standard ML cross-validation to the Healthy Control (NC) group—where no strong correlation between brain structure and cognitive decline should exist—ML models falsely reported significant correlations. This overfitting ruins the validity of the baseline when comparing against MCI and AD groups.

Figure 2: 3D regression hyperplanes using SAR and OLS for MMSE vs PCA predictors.

SAR corrected this anomaly. By applying the SAR test, the false positive rates were suppressed, accurately confirming that no significant relationship existed in the healthy baseline. However, when evaluating the entire dataset (AD, MCI, NC), SAR successfully detected true linearity with a lower sample requirement, penalizing dimensionality appropriately.

4. CONCLUSIONS

The proposed Statistical Agnostic Regression (SAR) provides a much-needed formal foundation for hypothesis testing within Artificial Intelligence.

• Standard ML validation methods (like K-fold CV) tend to overinflate false positives with small samples.
• SAR features effectively detect true linearity and seamlessly integrate into classical statistical frameworks (like the multivariate F-test).
• It acts as a safety net: informing researchers when there is insufficient evidence to establish a linear relationship, preventing the publication of non-replicable scientific results.

This methodology is poised to become a vital tool in medical research, neuroimaging, and any field where data scarcity and model reliability are critical.