机器学习基石笔记（七）：VC维度

Lecture 7: The VC Dimension

Definition of VC Dimension

$m_{\mathcal{H}}(N)$ breaks at k(good $H$) and N large enough(good $D$) => $E_{\mathrm{o}\mathrm{u}\mathrm{t}}\approx E_{\mathrm{i}\mathrm{n}}$

$A$ picks a $g$ with small $E_{in}$ (good $A$) => $E_{\mathrm{i}\mathrm{n}}\approx 0$

最后学到了东西，到达了要求 => $E_{\mathrm{o}\mathrm{u}\mathrm{t}}\approx 0$

VC Dimension

VC Dimension is the maximum of non-break point.
   largest N for which $m_H(N)$ = $2^N$
   $d_{vc}$ = ‘minimum k’ - 1
   the most inputs $H$ that can shatter
VC Dimension反映了假设集在数据集上的性质：数据集的最大分类情况。
$d_{vc}$和minimum break point都是"一线曙光"的分界点。
$ifN\ge 2,d_{\mathrm{v}\mathrm{c}}\ge 2,m_{\mathcal{H}}(N)\le N^{d_{\mathrm{v}\mathrm{c}}}$

和之前一样(minimum break point finite)，我们希望$d_{vc}$ finite.

Fun Time

If there is a set of $N$ inputs that cannot be shattered by $H$. Based only on this information, what can we conclude about $d_{VC}(H)$?
1 $d_{\mathrm{v}\mathrm{c}}(\mathcal{H})>N$
2 $d_{\mathrm{v}\mathrm{c}}(\mathcal{H})=N$
3 $d_{\mathrm{v}\mathrm{c}}(\mathcal{H})<N$
4 no conclusion can be made  $✓$


Explanation
这道题目当初也是百思不得其解，直到我又学习了一遍。
VC Dimension：the most inputs $H$ that can shatter
有N个数据no shatter不能代表其他的N个数据no shatter。
举例说明：
1维感知器 break point为3， 2维感知器 break point为4。
在2维空间中，连成一条直线的三个点（2维退化到1维），分类数只有6($<2^3$)种 。而2维感知器的break point却为4，这就是因为不在同一条直线上的三个点分类数可以为8。所以导致2维感知器的最小break point增大了。
这里面也蕴含着提升维度，增大了自由度。?

VC Dimension of Perceptrons

1D perceptron (pos/neg rays)：$d_{\mathrm{v}\mathrm{c}}$ = 2
2D perceptrons: $d_{\mathrm{v}\mathrm{c}}$ = 3
d-D perceptrons：$d_{\mathrm{v}\mathrm{c}}=d+1$
证明分两步:
  • $d_{\mathrm{v}\mathrm{c}}\ge d+1$：存在(d+1)个点 shatter.
  • $d_{\mathrm{v}\mathrm{c}}\le d+1$：任何(d+2)个点 no shatter.

Extra Fun Time

What statement below shows that $d_{\mathrm{v}\mathrm{c}}\ge d+1$
1 There are some d + 1 inputs we can shatter.  $✓$
2 We can shatter any set of d + 1 inputs.
3 There are some d + 2 inputs we cannot shatter.
4 We cannot shatter any set of d + 2 inputs.


Explanation
存在(d+1)个点 shatter => $d_{\mathrm{v}\mathrm{c}}\ge d+1$

Special X（d+1个点） 对于任意的y，都有对应的hypothesis(w) .

Extra Fun Time

What statement below shows that $d_{\mathrm{v}\mathrm{c}}\le d+1$
1 There are some d + 1 inputs we can shatter.
2 We can shatter any set of d + 1 inputs.
3 There are some d + 2 inputs we cannot shatter.
4 We cannot shatter any set of d + 2 inputs.  $✓$


Explanation
任何(d+2)个点 no shatter. => $d_{\mathrm{v}\mathrm{c}}\le d+1$

$x_{d+2}$与$x_1,x_2,...,x_{d+1}$线性相关，假设这任意d+2个点可以shatter，那么$a_1,a_2,...,a_{d+1},a_{d+2}$系数可以任意取，当$a_i=sign(w^T{x}_i)$时，$w^T{x}_d>0$，所以d+2个点no shatter.

Fun Time

Based on the proof above, what is d VC of 1126-D perceptrons?
1 1024
2 1126
3 1127  $✓$
4 6211


Explanation
d-D perceptrons：$d_{\mathrm{v}\mathrm{c}}=d+1$

Physical Intuition of VC Dimension

$d_{\mathrm{v}\mathrm{c}}(H)$: powerfulness of $H$
$d_{\mathrm{v}\mathrm{c}}$ ≈ #free parameters
$d_{\mathrm{v}\mathrm{c}}$代表了自由度(与自由变量的个数相关)，我在学统计学时接触过自由度这个概念，若$\mu$（均值）确定，数据大小为N的自由度就为N-1，因为$\mu$确定，知道N-1个点，就能求出第N个点。
$d_{VC}(H)$就代表了$H$关于数据的自由度。

M and $d_{VC}$

so choose the right $hypothesisset$($M$ or $d_{\mathrm{v}\mathrm{c}}$).

Fun Time

Origin-crossing Hyperplanes are essentially perceptrons with $w_0$ fixed at 0. Make a guess about the$d_{\mathrm{v}\mathrm{c}}$ of origin-crossing hyperplanes in $R_d$.
1 1
2 d  $✓$
3 d+1
4 $\infty$


Explanation
d-D perceptrons：$d_{\mathrm{v}\mathrm{c}}=d+1$
而穿过原点，加了一个限制条件，使自由度降1，故变成了d.

Interpreting VC Dimension

Penalty for Model Complexity

$$\begin{gathered} {\mathbb{P}}_{\mathcal{D}}\left[\underset{\text{ BAD }}{\underbrace{\left[E_{\text{ in }}(g)-E_{\text{ out }}(g)|>ϵ\right]}}\right]\le \underset{\delta }{\underbrace{4(2N{)}^{d_{vc}}\exp \left(-\frac{1}{8}{ϵ}^{2}N\right)}} \\ probability\ge 1-\delta ,\mathrm{GOOD}:|E_{in}(g)-E_{out}(g)|\le ϵ \\ \delta =4(2N{)}^{d_{\mathrm{v}\mathrm{c}}}\exp \left(-\frac{1}{8}{ϵ}^{2}N\right) \\ ϵ=\sqrt{\frac{8}{N}\ln \left(\frac{4(2N{)}^{d_{\mathrm{v}\mathrm{c}}}}{\delta }\right)} \\ {E}_{in}(g)-\sqrt{\frac{8}{N}\ln \left(\frac{4(2N{)}^{d_{\mathrm{v}\mathrm{c}}}}{\delta }\right)}\le E_{out}(g)\le E_{in}(g)+\sqrt{\frac{8}{N}\ln \left(\frac{4(2N{)}^{d_{\mathrm{v}\mathrm{c}}}}{\delta }\right)} \\ \underset{\Omega (N,\mathcal{H},\delta )}{\underbrace{\sqrt{\frac{8}{N}\ln \left(\frac{4(2N{)}^{d_{\mathrm{v}\mathrm{c}}}}{\delta }\right)}}}\text{ penalty for model complexity} \end{gathered}$$

Sample Complexity

$$\begin{gathered} {\mathbb{P}}_{\mathcal{D}}\left[\underset{\text{ BAD }}{\underbrace{|E_{\text{ in }}(g)-E_{\text{ out }}(g)|>ϵ}}\right]\le \underset{\delta }{\underbrace{4(2N{)}^{d_0}\exp \left(-\frac{1}{8}{ϵ}^{2}N\right)}} \\ \text{given}ϵ=0.1,\delta =0.1,d_{\mathrm{v}\mathrm{c}}=3 \\ \begin{array}{cc}N& \text{ bound }\\ 100& 2.82\times 10^7\\ 1,000& 9.17\times 10^9\\ 10,000& 1.19\times 10^8\\ 100,000& 1.65\times 10^{-38}\\ 29,300& 9.99\times 10^{-2}\end{array} \\ \text{sample complexity:}N\approx 10,000d_{\mathrm{v}\mathrm{c}}\text{in theory} \end{gathered}$$

import math


# 计算VC Bound
def count_vc_bound(N, epsilon=0.1, delta=0.1, d_vc=3):
    vc_bound = 4*(2*N)**d_vc*math.exp(-0.125*epsilon**2*N)
    return vc_bound


if __name__ == '__main__':
    print('%e' % count_vc_bound(100))
    print('%e' % count_vc_bound(1000))
    print('%e' % count_vc_bound(10000))
    print('%e' % count_vc_bound(100000))
    print('%e' % count_vc_bound(29300))
    print(count_vc_bound(0))
    print(count_vc_bound(1))
    delta = 0.1
    n = 1
    while count_vc_bound(n) > delta:
        n = n + 1
    print(n)
    print('%e' % count_vc_bound(29299))
    print('%e' % count_vc_bound(29300))
    print('%e' % count_vc_bound(29301))

Looseness of VC Bound

$$N\approx 10,000d_{\mathrm{v}\mathrm{c}}\text{in theory},N\approx 10d_{\mathrm{v}\mathrm{c}}\text{in practice}$$

我们能基于VC Bound原理提升整个机器学习流程。

Fun Time

Consider the VC Bound below. How can we decrease the probability of getting BAD data?
1 decrease model complexity $d_{\mathrm{v}\mathrm{c}}$
2 increase data size $N$ a lot
3 increase generalization error tolerance $ϵ$
4 all of the above  $✓$


Explanation
降低模型复杂度：减少假设集选择
增大数据量：使训练数据更接近真实数据分布
增大容忍程度：扩大好数据范围

Summary

本篇讲义主要讲了模型复杂度，数据数量级以及VC Bound的宽松程度。

在有限的$d_{\mathrm{v}\mathrm{c}}$，足够大的$N$，足够小的$E_{in}$，我们真正能学到模型。

讲义总结

Definition of VC Dimension
   maximum non-break point

VC Dimension of Perceptrons
   $d_{VC}(H)=d+1$

Physical Intuition of VC Dimension
   $d_{VC}$ ≈ #free parameters

Interpreting VC Dimension
  loosely: model complexity & sample complexity

参考文献

《Machine Learning Foundations》(机器学习基石)—— Hsuan-Tien Lin (林轩田)