逻辑回归(Logistic Regression)
判别模型:我们只需要学习P(y|x)。
让步比(odds ratio):
假设一个特征有0.9的概率属于类别1,P(y=1)=0.9。那让步比为:P(y=1)/P(y=0) = 0.9/0.1 = 9。让步比范围0到正无穷。取对数后将所有0到1之间的概率映射到负无穷到正无穷,更高的概率对应于更高的让步比对数。
线性等式:
等式替换(用p替换y):
求解pi:
Binary case:
sigmoid函数:
损失函数:
优化算法
梯度下降算法:
特征:
对于逻辑回归,我们可以通过已经掌握的系数(clf.coef_
)直接就能获知这个特征的影响力。一个特征的系数越高,就在模型预测过程中的作用越大;负值系数告诉我们对负类的贡献度。
多项逻辑斯蒂回归
将LR泛化到多分类。
Multinomial case:
损失函数:
应用
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
clf.fit(X, y)
重要参数:
- C:Inverse of regularization strength; must be a positive float. Like in support vector machines, smaller values specify stronger regularization.
- penalty : str, ‘l1’ or ‘l2’, default: ‘l2’
小结
Given this model formulation, we want to learn parametes {c_i} that maximise the conditional likehood of the data according to the model.
Due to the softmax function we only construct a classifier, but learn probablity distributions over classifications.
These are many ways to chose weights {c_i}:
- Percptron: Find misclassified examples and move weights in the direction of their correct class
- Margin-Based: Methods such as Support Vector Machines can be used for learning weights
- Logistic Regression: Directly maximise the conditional log-likelihood via gradient descent.
《Building Machine Learning Systems with Python》
scikit-learn.org/stable/modules/linear_model.html#logistic-regression
http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
《统计学习方法》