cross-validation 交叉检验（二）

最新推荐文章于 2024-11-19 12:21:10 发布

原创

最新推荐文章于 2024-11-19 12:21:10 发布 · 4.6k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#validation #training #intervals #network #conflict #methods

交叉验证在模型选择中可能遇到问题，尤其是当数据发生微小变化时可能导致所选模型的大幅变动。研究表明，对于线性回归，10折和5折交叉验证优于留一法。此外，留一法在小样本中可能过度拟合，而适当大小的v折交叉验证可以改善这一情况。SBC在样本量趋于无穷大时提供一致的子集选择，而AIC则不然。因此，对于小样本，留一法可能过度拟合，而v折交叉验证应表现更好。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Leave-one-out cross-validation can also run into trouble with various model-selection methods. Again, one problem is lack of continuity--a small change in the data can cause a large change in the model selected (Breiman,1996). For choosing subsets of inputs in linear regression, Breiman and Spector (1992) found 10-fold and 5-fold cross-validation to work better than leave-one-out. Kohavi (1995) also obtained good results for 10-fold cross-validation with empirical decision trees (C4.5). Values of k as small as 5 or even 2 may work even better if you analyze several different random k-way splits of the data to reduce the variability of the cross-validation estimate.

Leave-one-out cross-validation also has more subtle deficiencies for model selection. Shao (1995) showed that in linear models, leave-one-out cross-validation is asymptotically equivalent to AIC (and Mallows' C_p), but leave-v-out cross-validation is asymptotically equivalent to Schwarz's Bayesian criterion (called SBC or BIC) when v =n[1-1/(log(n)-1)], where n is the number of training cases. SBC provides consistent subset-selection, while AIC does not. That is, SBC will
choose the "best" subset with probability approaching one as the size of the training set goes to infinity. AIC has an asymptotic probability of one of choosing a "good" subset, but less than one of choosing the "best" subset
(Stone, 1979). Many simulation studies have also found that AIC overfits badly in small samples, and that SBC works well (e.g., Hurvich and Tsai, 1989; Shao and Tu, 1995). Hence, these results suggest that leave-one-out cross-validation should overfit in small samples, but leave-v-out cross-validation with appropriate v should do better. However, when true models have an infinite number of parameters, SBC is not efficient, and other criteria that are asymptotically efficient but not consistent for model selection may produce better generalization (Hurvich and Tsai, 1989).

Shao (1993) obtained the surprising result that for selecting subsets of inputs in a linear regression, the probability of selecting the "best" does not converge to 1 (as the sample size n goes to infinity) for leave-v-out cross-validation unless the proportion v/n approaches 1. At first glance, Shao's result seems inconsistent