Model Selection Using AIC and BIC: Valid Comparisons Without Treating Them as Cutoffs
AIC and BIC are meaningful only relative to candidate models using the same data and comparable likelihoods. An absolute value of 240 or 2,400 does not establish good or poor fit.
AIC = −2 log L + 2k; BIC = −2 log L + k log n. Prefer smaller values within the same candidate set, inspect differences and model weights where useful, and still examine residuals, theory and validation.
AIC and BIC require a valid comparison
- Same data
Ensure compatible samples and variables
- Comparable basis
Check likelihood and estimation
- AIC / BIC
Read differences between models
- Theory
Consider purpose and specification
Conceptual diagram. Do not treat AIC or BIC as an absolute fit cutoff.
Source: Lê Minh · Conceptual illustration, not software output.1. Both reward fit but penalize complexity differently
Both trade off likelihood against parameter count k. A smaller value indicates a relative preference, with no universal passing AIC/BIC threshold (Aho et al., 2014; Burnham & Anderson, 2004).
AIC and BIC can select different models because their asymptotic objectives differ. Do not average them into a single score.
| Criterion | Formula | Tendency |
|---|---|---|
| AIC | −2 log L + 2k | Optimizes approximation/prediction; penalty 2k |
| BIC | −2 log L + k log n | Stronger penalty at larger n; seeks the true model under ideal conditions |
2. Conditions for valid comparison
If observations differ because of missingness or different outcomes, log-likelihoods no longer share a comparison basis. Use the same complete-case set or consistent missing-data treatment.
- Same outcome and observations.
- Likelihoods defined on the same scale.
- Same missing-data treatment.
- Justified candidate models, not models generated through result hunting.
- Consistent parameter counting.
- An estimator supporting the relevant information criterion.
3. Read differences rather than absolute values
Calculate ΔAICᵢ = AICᵢ − AICmin and similarly for BIC. A small Δ indicates continuing relative support; a large difference indicates less support within the candidate set.
Akaike weights can describe relative evidence and support model averaging. Δ conventions remain guidance, not p-value tests.
| Model | k | AIC | ΔAIC | BIC | ΔBIC | Validation |
|---|---|---|---|---|---|---|
| M1 | … | … | … | … | … | RMSE … |
| M2 | … | … | … | … | … | RMSE … |
4. Reporting template
Reporting example in English
Models were estimated on the same N = [ ] using the same likelihood. Model [ ] had AIC = [ ], ΔAIC = [ ], BIC = [ ], ΔBIC = [ ]. [AIC/BIC] favored [model], whereas validation indicated [ ]. The final choice reflected [objective], theory, diagnostics and complexity.
- Compare the same data and outcome.
- Do not interpret absolute values.
- Report k and Δ.
- Do not treat AIC/BIC as p-values.
- Check residuals and validation.
- Disclose the candidate set.
Conclusion
AIC and BIC support relative selection among specified models. They do not replace theory, residual diagnostics or testing with new data.
References
- Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference. Sociological Methods & Research, 33(2), 261–304.https://doi.org/10.1177/0049124104268644
- Aho, K., Derryberry, D., & Peterson, T. (2014). Model selection for ecologists. Ecology, 95(3), 631–636.https://doi.org/10.1890/13-1452.1
Methodological note: For AICc in small samples or highly parameterized models, use the appropriate correction. Software may use different likelihood constants; compare within the same system unless compatibility has been established.