重采样方法 (Resampling Methods) (CV, Bootstrap)

Introduction

Resampling methods involve repeatedly drawing samples from a training set and refitting a model of interest on each sample in order to obtain additional information about the fitted model. (e.g. cross-validation, bootstrap)

• Estimates of test-set prediction error (CV)
• S.E. and bias of estimated parameters (Bootstrap)
• C.I. of target parameter (Bootstrap)

Cross-Validation

The training error rate often is quite different from the test error rate, and in particular the former can dramatically underestimate the latter.

• Model Complexity Low: High bias, Low variance

• Model Complexity High: Low bias, High variance

Prediction Error Estimates

• Large test set
• Mathematical adjustment
  • $C_p=\frac{1}{n}(SS{E}_d+2d{\hat{\sigma }}^2)$
  • $AIC=\frac{1}{n{\hat{\sigma }}^2}(SS{E}_d+2d{\hat{\sigma }}^2)$
  • $BIC=\frac{1}{n{\hat{\sigma }}^2}(SS{E}_d+log(n)d{\hat{\sigma }}^2)$
• CV: Consider a class of methods that estimate the test error rate by holding out a subset of the training observations from the fitting process, and then applying the statistical learning method to those held out observations.

The Validation Set Approach

A random splitting into two halves: left part is training set, right part is validation set.

Drawbacks

• The validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set.
• Only a subset of the observations are used to fit the model.
• Validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set.

Leave-One-Out Cross-Validation

LOOCV involves splitting the set of observations into two parts. However, instead of creating two subsets of comparable size, a single observation $(x_1,y_1)$ is used for the validation set, and the remaining observations $(x_2,y_2),...,(x_n,y_n)$ make up the training set.

In Linear Regression

$$C{V}_{(n)}=\frac{1}{n}\sum _{i=1}^n(\frac{y_i-\widehat{y_i}}{1-h_i}{)}^2$$

• $C{V}_n$ bacomes a weighted MSE

Drawbacks

• Estimates from each fold are highly correlated and hence their average can have high variance.

K-fold Cross-Validation

This approach involves randomly dividing the set of observations into k groups, or folds, of approximately equal size. The first fold is treated as a validation set, and the method is fit on the remaining k − 1 folds. This procedure is repeated k times; each time, a different group of observations is treated as a validation set. This process results in k estimates of the test error. The k-fold CV estimate is computed by averaging these values. If k=n, then it is LOOCV.
$$C{V}_{(k)}=\frac{1}{k}$$