机器学习基石 - Theory of Generalization

机器学习基石上 (Machine Learning Foundations)—Mathematical Foundations
Hsuan-Tien Lin, 林轩田，副教授 (Associate Professor)，资讯工程学系 (Computer Science and Information Engineering)

Theory of Generalization

Restriction of Break Points

• growth function $m_{\mathcal H}(N)$: max number of dichotomies
• 漏出一线曙光的点 break point
• break point $k$ restricts maximum possible $m_{\mathcal H}(N)$ a lot for $N>k$

Bounding Function: Basic Cases

• $B(N,k)$: maximum possible $m_{\mathcal H}(N)$ when break point = k
  
• 表格
  

Bounding Function: Inductive Cases

• $B(4,3)$ 的估计
  
  • 
  • 
• Putting It All Together
  
• $B(N,k)\leq \sum_{i=0}^{k-1}\left(\begin{array}{c} N \\ i \end{array}\right)$
  
  • 数学归纳法
  • $C_{N-1}^{\ i}+C_{N-1}^{\ i+1}=C_{N}^{\ i+1}$
  • actually $\leq$ can be $=$
  • $B(N,k)\geq 2B(N-1,k-1)+(B(N-1,k)-B(N-1,k-1))$
• can bound $m_{\mathcal H}(N)$ by only one break point

A Pictorial Proof

• Step 1: Replace $E_{out}$ by $E_{in}^{\ '}$
  
• Step 2: Decompose $\mathcal H$ by Kind
  
• Step 3: Use Hoeffding without Replacement
  
• Vapnik-Chervonenkis (VC) bound