softmax regression 推导

最新推荐文章于 2020-12-24 13:47:07 发布

原创最新推荐文章于 2020-12-24 13:47:07 发布 · 972 阅读

2 ·

CC 4.0 BY-SA版权

machine learning 专栏收录该内容

21 篇文章

订阅专栏

本文详细解析了多分类问题中对数损失函数的定义及其梯度的推导过程，通过数学公式展示了如何从概率分布出发，计算每个样本对于参数θ的贡献，并进一步推导出了损失函数关于参数θ的偏导数。

P (y (i) = k | x (i); θ) = exp ( θ ( k ) ⊤ x ( i ) ) \sum K j = 1 exp ( θ ( j ) ⊤ x ( i ) )

$P(y^{(i)} = k | x^{(i)} ; \theta) = \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)}) }$

似然函数

L = \prod i = 1 M \prod k = 1 K P (y (i) = k | x (i); θ) 1 {y (i) = k}

$L=\prod_{i=1}^{M}\prod_{k=1}^{K} P(^{y^{(i)} } =k|x^{(i)};\theta )^{1\{y^{ (i)}{=k} \} }$

对数损失函数为:

J (θ) = - ⎡ ⎣ \sum i = 1 m \sum k = 1 K 1 {y (i) = k} log exp ( θ ( k ) ⊤ x ( i ) ) \sum K j = 1 exp ( θ ( j ) ⊤ x ( i ) ) ⎤ ⎦

$\begin{align} J(\theta) = - \left[ \sum_{i=1}^{m} \sum_{k=1}^{K} 1\left\{y^{(i)} = k\right\} \log \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}\right] \end{align}$

1{⋅} is the ”‘indicator function,”’ so that 1{a true statement}=1, and 1{a false statement}=0.

现在对对数损失函数求偏导

\nabla θ (n) J (θ) = \sum i = 1 m y (i) \partial P ( y ( i ) = n | x i ; θ ) \partial θ ( n ) + \sum k = 1, k \neq n K y (i) \partial P ( y ( i ) = k | x i ; θ ) \partial θ ( n )

$\nabla_{\theta^{(n)}}J(\theta)= \sum_{i=1}^{m} y^{(i)} \frac{\partial P(y^{(i)}=n|x^{i};\theta)}{\partial \theta^{(n)}} + \sum_{k=1,k\ne n}^{K} y^{(i)} \frac{\partial P(y^{(i)}=k|x^{i};\theta)}{\partial \theta^{(n)} }$
其中,

P (y (i) = n | x i; θ) = log exp ( θ ( n ) ⊤ x ( i ) ) \sum K j = 1 exp ( θ ( j ) ⊤ x ( i ) )

$P(y^{(i)}=n|x^{i};\theta)=\log \frac{\exp(\theta^{(n)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}$

P (y (i) = k | x i; θ) = log exp ( θ ( k ) ⊤ x ( i ) ) \sum K j = 1 exp ( θ ( j ) ⊤ x ( i ) )

$P(y^{(i)}=k|x^{i};\theta) = \log \frac{\exp(\theta^{(k)\top} x^{(i)})}{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})} \\$

\partial P ( y ( i ) = n | x i ; θ ) \partial θ ( n ) = \sum K j = 1 exp ( θ ( j ) ⊤ x ( i ) ) exp ( θ ( n ) ⊤ x ( i ) ) * ⎛ ⎝ ⎜ ⎜ exp ( θ ( n ) ⊤ x ( i ) ) * x ( i ) \sum K j = 1 exp ( θ ( j ) ⊤ x ( i ) ) - exp ( θ ( n ) ⊤ x ( i ) ) * exp ( θ ( n ) ⊤ x ( i ) ) x ( i ) [ \sum K j = 1 exp ( θ ( j ) ⊤ x ( i ) ) ] 2 ⎞ ⎠ ⎟ ⎟ = x (i) - exp ( θ ( n ) ⊤ x ( i ) ) x ( i ) \sum K j = 1 exp ( θ ( j ) ⊤ x ( i ) ) = x (i) (1 - P (y (i) = n | x i; θ))

$\frac { \partial P(y^{(i)}=n|x^{i};\theta) }{\partial \theta^{(n)} }\\ = \frac{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}{\exp(\theta^{(n)\top} x^{(i)})} *\left( \frac{ \exp(\theta^{(n)\top} x^{(i)})* x^{(i)}}{ \sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)}) } -\frac{ \exp(\theta^{(n)\top} x^{(i)}) *\exp(\theta^{(n)\top} x^{(i)}) x^{(i)} }{ \left[ \sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)}) \right]^2 }\right)\\ =x^{(i)}-\frac{ \exp(\theta^{(n)\top} x^{(i)}) x^{(i)} }{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}\\ =x^{(i)}\left(1-P(y^{(i)}=n|x^{i};\theta)\right)$

另外一个，

\partial P ( y ( i ) = k | x i ; θ ) \partial θ ( n ) = \sum K j = 1 exp ( θ ( j ) ⊤ x ( i ) ) exp ( θ ( k ) ⊤ x ( i ) ) ⎛ ⎝ ⎜ ⎜ - exp ( θ ( k ) ⊤ x ( i ) ) * exp ( θ ( n ) ⊤ x ( i ) ) x ( i ) [ \sum K j = 1 exp ( θ ( j ) ⊤ x ( i ) ) ] 2 ⎞ ⎠ ⎟ ⎟ = - exp ( θ ( n ) ⊤ x ( i ) ) x ( i ) \sum K j = 1 exp ( θ ( j ) ⊤ x ( i ) ) = - P (y (i) = n | x i; θ) x (i)

$\frac{\partial P(y^{(i)}=k|x^{i};\theta) } {\partial \theta^{(n)} }\\ = \frac{\sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)})}{\exp(\theta^{(k)\top} x^{(i)})} \left( -\frac{ \exp(\theta^{(k)\top} x^{(i)}) *\exp(\theta^{(n)\top} x^{(i)}) x^{(i)} }{ \left[ \sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)}) \right]^2 } \right)\\ =-\frac{ \exp(\theta^{(n)\top} x^{(i)}) x^{(i)} }{ \sum_{j=1}^K \exp(\theta^{(j)\top} x^{(i)}) }\\ =-P(y^{(i)}=n|x^{i};\theta)x^{(i)}$

\nabla θ (k) J (θ) = - \sum i = 1 m [x (i) (1 {y (i) = k} - P (y (i) = k | x (i); θ))]

$\begin{align} \nabla_{\theta^{(k)}} J(\theta) = - \sum_{i=1}^{m}{ \left[ x^{(i)} \left( 1\{ y^{(i)} = k\} - P(y^{(i)} = k | x^{(i)}; \theta) \right) \right] } \end{align}$