【算法检验】StepM和MCS

继上节SPA继续介绍StepM和MCS检验

【算法检验】SPA_Checkmate9949的博客-优快云博客一、概念The multiple comparison procedures all allow for examining aspects of superior predictive ability.用于检验更优的预测能力There are three available:SPA - The test of Superior Predictive Ability, also known as the Reality Check (and accessible as RealityChechttps://blog.youkuaiyun.com/Checkmate9949/article/details/120370004

一、StepM检验

1、概念

SM类似于SPA，都具有同样的零假设。主要不同：SM可以识别比基准模型好的一系列模型，而不是只是验证某个模型更好。

Stepwise Multiple Testing is similar to the SPA and has the same null. The primary difference is that it identifies the set of models which are better than the benchmark, rather than just asking the basic question if any model is better.

2、例子

from arch.bootstrap import StepM

stepm = StepM(bm_losses, model_losses)
stepm.compute()
#识别出优于基准模型的模型
print("Model indices:")
print([model.split(".")[1] for model in stepm.superior_models])



#求各模型的平均Loss
test2 = model_losses.mean(0)
better_models = pd.concat([model_losses.mean(0), model_losses.mean(0)],axis=1)
better_models.columns = ["Same or worse", "Better"]
#得出better模型的index
better = better_models.index.isin(stepm.superior_models)
#得出worse模型的index
worse = np.logical_not(better)
#写为正确的值
better_models.loc[better, "Same or worse"] = np.nan
better_models.loc[worse, "Better"] = np.nan
#画图：绿色为更优的模型
fig = better_models.plot(style=["o", "s"], rot=270)

 二、MCS检验

1、概念

The Model Confidence Set

MCS将一系列Loss作为输入，以发现类似的算法集。其主要输出为P值，其模型的P值高于Size。较小的P值代表模型能够被其他更好的模型拒绝。

The model confidence set takes a set of losses as its input and finds the set which are not statistically different from each other while controlling the familywise error rate. The primary output is a set of p-values, where models with a pvalue above the size are in the MCS. Small p-values indicate that the model is easily rejected from the set that includes the best.

2、例子

from arch.bootstrap import MCS

# 取出25个算法，500/20=25
losses = model_losses.iloc[:, ::20]
mcs = MCS(losses, size=0.10)
mcs.compute()
# 显示各模型的MSC P值
print("MCS P-values")
print(mcs.pvalues)
# Included: P>0.1
print("Included")
included = mcs.included
print([model.split(".")[1] for model in included])
# Excluded: P<=0.1
print("Excluded")
excluded = mcs.excluded
print([model.split(".")[1] for model in excluded])

Included为更优秀的算法集，P越大，算法越好。