Logistic Regression cost function and Maximum likehood estimate_logistic regression maximum likehood-优快云博客

本文链接：https://blog.youkuaiyun.com/Solo95/article/details/84316322

本文深入探讨了逻辑回归的成本函数，详细解释了如何通过最大似然估计原理来优化参数，以实现模型对训练数据集标签的准确预测。文章涵盖了从单例到多例的损失函数推导过程，以及如何通过最小化损失函数来达到最大化概率的目的。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Logistic Regression cost function

The original form is y^, here we simplified by using y’ because of the latex grammar.

$I f y = 1 : p (y ∣ x) = y^{'}$
$I f y = 0 : p (y ∣ x) = 1 - y^{'}$

$y'^y (1-y)^{1-y}$

This one equation can express that:
$I f y = 1 : p (y ∣ x) = y^{'}$
$I f y = 0 : p (y ∣ x) = 1 - y^{'}$

The log function is a strictly monotonically increasing.
Maximizing $l o g (p (y ∣ x))$ give you a similar result that is optimizing p(y|x) and if you compute log of p(y|x)
->
$p(y∣x)=log(y′y(1−y)1−y)=ylogy′+(1−y)log(1−y′)log\ p(y|x)=log (y'^y (1-y)^{1-y}) =ylogy' +(1-y)log(1-y')$
$= - l (y^{'}, y)$ note: l represents loss function here.
Minimizing the loss function corresponds to maximum the log of the probability.
This is what the loss funcion on a single example looks like.

Cost on m examples

$set)=log∏i=1mp(y′i,y′)log\ p(labels\ in \ the training \ set) = log \prod_{i=1}^mp(y'^i,y')$
$p(yi∣xi)=−∑i=1ml(y′i,yi)log\ p(...) = \sum_{i=1}^mlog\ p(y^i|x^i)=-\sum_{i=1}^ml(y'^i,y^i)$

Maximum likelihood estimation

And so in statistics, there’s a principle called the principal of maximum likelihood estimation,which just means choose the parameters that maximizes this thing(refer to above).

Cost function:
Because we want to minimize the cost, instead of maximizing likelihood we’ve got rid of negative. And then finally for convenience, we make sure that our quantities are better scale, we just add a 1 over m extra scaling factor there.
$=\frac{1}{m}\sum_{i=1}^ml(y'^i,y^i)$

But to summarize, by minimizing this cost function J(w,b), we’re really carrying out maximum likelihood estimation Under the assumption that our training examples were IID or identically independently distributed.

Reference

https://mooc.study.163.com/learn/2001281002?tid=2001392029#/learn/content?type=detail&id=2001702014
Maximum likelihood Estimate
https://blog.youkuaiyun.com/zengxiantao1994/article/details/72787849