16、WBIC: Theory, Calculation, and Application in Machine Learning

WBIC: Theory, Calculation, and Application in Machine Learning

1. Introduction to WBIC

The Watanabe Bayesian Information Criterion (WBIC) is an important concept in machine learning. When the inverse temperature is $\beta > 0$, WBIC is defined as:
[
WBIC_n := E_{\beta}\left[-\sum_{i = 1}^{n}\log p(x_i|\theta)\right]
]

For the generalized free energy $F_n(\beta) = -\log\int_{\Theta}\prod_{i = 1}^{n}p(x_i|\theta)^{\beta}\phi(\theta)d\theta$ ($\beta > 0$), we have:
- $F’ n(\beta) = E {\beta}\left[-\sum_{i = 1}^{n}\log p(x_i|\theta)\right] = WBIC_n$
- $F’‘ n(\beta) = -E {\beta}\left[\left(\sum_{i = 1}^{n}\log p(x_i|\theta)\right)^2\right] + \left(E_{\beta}\left[\sum_{i = 1}^{n}\log p(x_i|\theta)\right]\r