机器学习（五）——广义线性模型

5.广义线性模型

• 指数族

$$p(y;\eta )=b(y)e^{({\eta }^{T}T(y)-a(\eta ))}$$

​ $\eta$ 叫做此分布的自然参数，一般$T(y)=y$

• 伯努利分布属于指数族
  p(y;ϕ)=ϕy(1−ϕ)1−y=e[ylogϕ+(1−y)log(1−ϕ)]=e{[log(ϕ1−ϕ)]y+log(1−ϕ)} \large p(y;ϕ)=ϕ^y(1−ϕ)^{1−y}=e^{[ylogϕ+(1−y)log(1−ϕ)]}= e^{\{[log(\frac{ϕ}{1−ϕ})]y+log(1−ϕ)\}} p(y;ϕ)=ϕy(1−ϕ)1−y=e[ylogϕ+(1−y)log(1−ϕ)]=e{[log(1−ϕϕ​)]y+log(1−ϕ)}

  $$\begin{array}{rl}T(y)& =y\\ a(\eta )& =-log(1-\varphi )\\ & =log(1+e\eta )\\ b(y)& =1\end{array}$$

• 正态分布属于指数族

• 多项式分布属于指数族

• 伽马和指数分布属于指数族

• 贝塔和狄利克雷分布属于指数族

构建模型

• 前提：

  1. $y|x;\theta \sim ExponentialFamily(\eta )$，即给定 $x$ 和 $\theta ,y$ 的分布属于指数分布族，是一个参数为 $\eta$ 的指数分布。
  2. $h$ 输出的预测值 $h(x)$ 要满足 $h(x)=E[y|x]$
  3. 自然参数 $\eta$ 和输入值 $x$ 是线性相关的，$\eta ={\theta }^{T}x$，或者如果 $\eta$ 是有值的向量，则有${\eta }_i={\theta }_i^{T}x$。

• SoftMax回归

  假设有K个分类，则分类写作向量形式 $T(y)\in R^{k-1}$:

  $$T(1)=\left[\begin{array}{c}1\\ 0\\ 0\\ ⋮\\ 0\end{array}\right],T(2)=\left[\begin{array}{c}0\\ 1\\ 0\\ ⋮\\ 0\end{array}\right],T(3)=\left[\begin{array}{c}0\\ 0\\ 1\\ ⋮\\ 0\end{array}\right],T(k-1)=\left[\begin{array}{c}0\\ 0\\ 0\\ ⋮\\ 1\end{array}\right],T(k)=\left[\begin{array}{c}0\\ 0\\ 0\\ ⋮\\ 0\end{array}\right]$$

  向量 $T(y)$ 中的第 $i$ 个元素写成$(T(y){)}_i$ 。

  概率写作:$ p (y = i; \phi)=\phi_i ，p (y = k; \phi) = 1 −\sum ^{k−1}_{i=1}\phi_i$:
  p(y;ϕ)=ϕ11{y=1}ϕ21{y=2}…ϕk1{y=k}=ϕ11{y=1}ϕ21{y=2}…ϕk1−∑i=1k−11{y=i}=ϕ1(T(y))1ϕ2(T(y))2…ϕk1−∑i=1k−1(T(y))i=exp((T(y))1log(ϕ1)+(T(y))2log(ϕ2)+⋯+(1−∑i=1k−1(T(y))i)log(ϕk))=exp((T(y))1log(ϕ1ϕk)+(T(y))2log(ϕ2ϕk)+⋯+(T(y))k−1log(ϕk−1ϕk)+log(ϕk))=b(y)exp(ηTT(y)−a(η)) \begin{aligned} p(y;\phi) &=\phi_1^{1\{y=1\}}\phi_2^{1\{y=2\}}\dots \phi_k^{1\{y=k\}} \\ &=\phi_1^{1\{y=1\}}\phi_2^{1\{y=2\}}\dots \phi_k^{1-\sum_{i=1}^{k-1}1\{y=i\}} \\ &=\phi_1^{(T(y))_1}\phi_2^{(T(y))_2}\dots \phi_k^{1-\sum_{i=1}^{k-1}(T(y))_i } \\ &=exp((T(y))_1 log(\phi_1)+(T(y))_2 log(\phi_2)+\dots+(1-\sum_{i=1}^{k-1}(T(y))_i)log(\phi_k)) \\ &= exp((T(y))_1 log(\frac{\phi_1}{\phi_k})+(T(y))_2 log(\frac{\phi_2}{\phi_k})+\dots+(T(y))_{k-1}log(\frac{\phi_{k-1}}{\phi_k})+log(\phi_k)) \\ &=b(y)exp(\eta^T T(y)-a(\eta)) \end{aligned} p(y;ϕ)​=ϕ11{y=1}​ϕ21{y=2}​…ϕk1{y=k}​=ϕ11{y=1}​ϕ21{y=2}​…ϕk1−∑i=1k−1​1{y=i}​=ϕ1(T(y))1​​ϕ2(T(y))2​​…ϕk1−∑i=1k−1​(T(y))i​​=exp((T(y))1​log(ϕ1​)+(T(y))2​log(ϕ2​)+⋯+(1−i=1∑k−1​(T(y))i​)log(ϕk​))=exp((T(y))1​log(ϕk​ϕ1​​)+(T(y))2​log(ϕk​ϕ2​​)+⋯+(T(y))k−1​log(ϕk​ϕk−1​​)+log(ϕk​))=b(y)exp(ηTT(y)−a(η))​
  其中:
  $$\begin{array}{rl}\eta & =\left[\begin{array}{c}\log ({\varphi }_1/{\varphi }_k)\\ \log ({\varphi }_2/{\varphi }_k)\\ ⋮\\ \log ({\varphi }_{k-1}/{\varphi }_k)\end{array}\right],\\ a(\eta )& =-\log ({\varphi }_k)\\ b(y)& =1\end{array}$$
  所以：
  $${\eta }_i=\log \frac{{\varphi }_i}{{\varphi }_k}$$
  于是我们可以以此推出：
  $$\begin{array}{rl}{e}^{{\eta }_i}& =\frac{{\varphi }_i}{{\varphi }_k}\\ {\varphi }_k{e}^{{\eta }_i}& ={\varphi }_i\\ {\varphi }_k\sum _{i=1}^k{e}^{{\eta }_i}& =\sum _{i=1}^k{\varphi }_i=1\\ {\varphi }_k& =\frac{1}{{\sum }_{i=1}^k{e}^{{\eta }_i}}\\ {\varphi }_i& =\frac{e^{{\eta }_i}}{{\sum }_{j=1}^k{e}^{{\eta }_j}}\end{array}$$
  下面这个函数从$\eta$ 映射到了$\varphi$，称为 Softmax 函数：
  $${\varphi }_i=\frac{e^{{\eta }_i}}{{\sum }_{j=1}^k{e}^{{\eta }_j}}$$
  由于前提3，也就是 ${\eta }_i$ 是一个 $x$ 的线性函数。所以就有了 ${\eta }_i={\theta }_i^{T}x(fori=1,...,k-1)$，其中的 ${\theta }_1,...,{\theta }_{k-1}\in R^{n+1}$ 就是我们建模的参数。因此，我们的模型假设了给定 $x$ 的 $y$ 的条件分布为：
  $$\begin{array}{rl}p(y=i|x;\theta )& ={\varphi }_i\\ & =\frac{e^{{\eta }_i}}{{\sum }_{j=1}^k{e}^{{\eta }_j}}\\ & =\frac{e^{{\theta }_i^{T}x}}{{\sum }_{j=1}^k{e}^{{\theta }_j^{T}x}}\end{array}$$
  这个适用于解决 $y\in 1,...,k$ 的分类问题的模型，就叫做 Softmax 回归。 这种回归是对逻辑回归的一种扩展泛化。

  由于前提2，可以发现：
  $$\begin{array}{rl}{h}_{\theta }(x)& =E[T(y)|x;\theta ]\\ & =E\left[\begin{array}{c}{\varphi }_1\\ {\varphi }_2\\ ⋮\\ {\varphi }_{k-1}\end{array}\right]\\ & =E\left[\begin{array}{c}\frac{exp({\theta }_1^{T}x)}{{\sum }_{j=1}^{k}exp({\theta }_j^{T}x)}\\ \frac{exp({\theta }_2^{T}x)}{{\sum }_{j=1}^{k}exp({\theta }_j^{T}x)}\\ ⋮\\ \frac{exp({\theta }_{k-1}^{T}x)}{{\sum }_{j=1}^{k}exp({\theta }_j^{T}x)}\end{array}\right]\end{array}$$
  最后我们通过极大似然拟合参数 ${\theta }_i$ ：
  $$\begin{array}{rl}l(\theta )& =\sum _{i=1}^m\log p(y^{(i)}|x^{(i)};\theta )\\ & =\sum _{i=1}^{m}log\prod _{l=1}^k{\left(\frac{e^{{\theta }_l^T{x}^{(i)}}}{{\sum }_{j=1}^k{e}^{{\theta }_j^T{x}^{(i)}}}\right)}^{1(y^{(i)}=l)}\end{array}$$
  其中$m$是样本数，$k$是分类数，对于每一个种类的概率${\varphi }_i$有一个相对应的参数向量${\theta }_i$(注意：由于第$k$个分类的概率，可以由$1-{\sum }_{i=1}^{k-1}{\varphi }_i$得来，所以我们定义${\theta }_k=0,{\varphi }_k=0$。所以对于逻辑回归(二分类)，参数向量$\theta$只有一个)

  • 对单个样本求导

    继续推导$l(\theta )$函数:
    l(θ)=∑i=1mlog∏l=1k(eθlTx(i))1(y(i)=l)∑j=1keθjTx(i)=∑i=1m{∑l=1k[(1{y(i)=l})(θlTx(i))]−log∑j=1keθjTx(i)} \begin{aligned} l(\theta)&=\sum^m_{i=1}log\frac {\prod ^k_{l=1}\left(e^{\theta_l^Tx^{(i)}}\right)^{1(y^{(i)}=l)}}{\sum^k_{j=1} e^{\theta_j^T x^{(i)}}} \\ &=\sum^m_{i=1}\left\{\sum^k_{l=1}\left[({1\{y^{(i)}=l\}})\left({\theta_l^Tx^{(i)}}\right)\right]-log{\sum^k_{j=1} e^{\theta_j^T x^{(i)}}}\right\} \end{aligned} l(θ)​=i=1∑m​log∑j=1k​eθjT​x(i)∏l=1k​(eθlT​x(i))1(y(i)=l)​=i=1∑m​{l=1∑k​[(1{y(i)=l})(θlT​x(i))]−logj=1∑k​eθjT​x(i)}​
    所以$l(\theta )$对第$d$种分类对应的参数${\theta }_n$求导：
    ∂l∂θd=∑i=1m{[(1{y(i)=d})(x(i))]−eθdTx(i)∑j=1keθjTx(i)x(i)}=∑i=1m{[(1{y(i)=d})−eθdTx(i)∑j=1keθjTx(i)](x(i))}=∑i=1m{[(1{y(i)=d})−SoftMax(eθdTx(i))](x(i))} \begin{aligned} \frac{\partial l}{\partial θ_d}&=\sum^m_{i=1}\left\{\left[({1\{y^{(i)}=d\}})\left({x^{(i)}}\right)\right]-\frac{e^{\theta_d^T x^{(i)}}}{\sum^k_{j=1} e^{\theta_j^T x^{(i)}}}x^{(i)}\right\}\\ &=\sum^m_{i=1}\left\{\left[({1\{y^{(i)}=d\}})-\frac{e^{\theta_d^T x^{(i)}}}{\sum^k_{j=1} e^{\theta_j^T x^{(i)}}}\right]\left({x^{(i)}}\right)\right\}\\ &=\sum^m_{i=1}\left\{\left[({1\{y^{(i)}=d\}})-SoftMax(e^{\theta_d^T x^{(i)}})\right]\left({x^{(i)}}\right)\right\}\\ \end{aligned} ∂θd​∂l​​=i=1∑m​{[(1{y(i)=d})(x(i))]−∑j=1k​eθjT​x(i)eθdT​x(i)​x(i)}=i=1∑m​{[(1{y(i)=d})−∑j=1k​eθjT​x(i)eθdT​x(i)​](x(i))}=i=1∑m​{[(1{y(i)=d})−SoftMax(eθdT​x(i))](x(i))}​

  • 对参数矩阵$\theta$求导

    设：
    $$Y=\left[\begin{array}{cccc}{y}^{(1)}& y^{(2)}& \cdots & y^{(m)}\end{array}\right]$$
    $Y$是$k\times m$的结果矩阵（m为样本数，k为分类总数）,其中$y^{(i)}$是每个样本的分类$T(y)$.
    $$X=\left[\begin{array}{c}—(x^{(1)}{)}^T—\\ —(x^{(2)}{)}^T—\\ ⋮\\ —(x^{(m)}{)}^T—\end{array}\right]$$
    $X$是$m\times n$的特征矩阵(m为样本数，n为特征数)
    $$\theta =\left[\begin{array}{cccc}{\theta }_1& {\theta }_2& \cdots & {\theta }_k\end{array}\right]$$
    $\theta$是$m\times k$的参数矩阵，对于每种不同的分类有一个参数向量${\theta }_i$

    则($\mathbf{E}$为元素全为1的$k\times k$矩阵)：
    $$l(\theta )=\sum _{i=1}^{m}log\left[\frac{e^{(x^{(i)}{)}^T\theta }}{e^{(x^{(i)}{)}^T\theta }\mathbf{E}}\right](y^{(i)})$$
    将$l(\theta )$修改写为矩阵形式
    $$\begin{array}{rl}l(\theta )& =tr(log[Softmax(X\theta )]Y)\\ & =tr(Ylog[Softmax(X\theta )])\\ & =tr(YX\theta -Ylog(e^{X\theta }\mathbf{E}))\end{array}$$
    求导：
    $$\begin{array}{rl}d(l)& =tr(YXd(\theta ))-tr\left(Y\left(\frac{1}{e^{X\theta }\mathbf{E}}\odot d(e^{X\theta }\mathbf{E})\right)\right)\\ & =tr(YXd(\theta ))-tr\left({\left(Y^T\odot \frac{1}{e^{X\theta }\mathbf{E}}\right)}^{T}d(e^{X\theta }\mathbf{E})\right)\\ & =tr(YXd(\theta ))-tr\left(\mathbf{E}{\left(Y^T\odot \frac{1}{e^{X\theta }\mathbf{E}}\right)}^{T}d(e^{X\theta })\right)\\ & =tr(YXd(\theta ))-tr\left(\mathbf{E}Y\left(\frac{1}{e^{X\theta }\mathbf{E}}\odot d(e^{X\theta })\right)\right)\\ & =tr(YXd(\theta ))-tr\left(\mathbf{E}\left(\frac{1}{e^{X\theta }\mathbf{E}}\odot d(e^{X\theta })\right)\right)\\ & =tr(YXd(\theta ))-tr\left({\left(\mathbf{E}\odot \frac{1}{e^{X\theta }\mathbf{E}}\right)}^{T}d(e^{X\theta })\right)\\ & =tr(YXd(\theta ))-tr\left({\left(\frac{1}{e^{X\theta }\mathbf{E}}\right)}^T\left[(e^{X\theta })\odot d(X\theta )\right]\right)\\ & =tr(YXd(\theta ))-tr\left({\left(\frac{1}{e^{X\theta }\mathbf{E}}\odot (e^{X\theta })\right)}^{T}d(X\theta )\right)\\ & =tr(YXd(\theta ))-tr\left({\left(\frac{e^{X\theta }}{e^{X\theta }\mathbf{E}}\right)}^{T}d(X\theta )\right)\\ & =tr(YXd(\theta ))-tr\left({\left(\frac{e^{X\theta }}{e^{X\theta }\mathbf{E}}\right)}^{T}Xd(\theta )\right)\\ & =tr\left({\left(Y^T-\left(\frac{e^{X\theta }}{e^{X\theta }\mathbf{E}}\right)\right)}^{T}Xd(\theta )\right)=tr\left({\left(\frac{\partial l}{\partial \theta }\right)}^{T}d(\theta )\right)\end{array}$$
    所以:
    $$\frac{\partial l}{\partial \theta }=X^T\left[Y^T-\left(\frac{e^{X\theta }}{e^{X\theta }\mathbf{E}}\right)\right]=X^T[Y^T-Softmax(X\theta )]$$