机器学习 2Logisti回归与Softmax回归

最新推荐文章于 2022-10-16 16:16:15 发布

原创最新推荐文章于 2022-10-16 16:16:15 发布 · 284 阅读

0 ·

CC 4.0 BY-SA版权

机器学习专栏收录该内容

12 篇文章

订阅专栏

本文深入探讨了Logistic回归及其在二分类问题中的应用，介绍了sigmoid激活函数、损失函数、梯度下降法，并提供了Python实现。接着，文章转向Softmax回归，解释了其在多分类问题中的作用，同样涵盖损失函数和更新规则。通过实例展示了Logistic和Softmax回归的训练过程。

2 Logistc回归

2.1 Logistc回归
2.2 Softmax回归

2.1 Logistc回归

$h_{w,b}=\sigma (g(x)) = \frac{1}{1+exp^{-(w^Tx+b)}}$

$w = (b,w1,w2,...,w_n)^T$

$x = (1,x1,x2,...,x_n)^T$

$z = g(x)=w^Tx$

$P(y=1|x;\theta)=h_\theta(x)$

$P(y=0|x;\theta)=1-h_\theta(x)$

似然函数

$L(w)=P(y|x:w)=\prod_{i=1}^nP(y^i|x^i;w)=\prod_{i=1}^n(h_w(x^i))^{y^i}(1-h_w(x^i))^{(1-y^i)}$

损失函数
$l(w)=-\frac{1}{m}ln(L(w))=\sum_i^m (y_iln(h_w(x_i))+ (1-y_i)ln(1-h_w(x_i)))$

w的导数
$\frac{\alpha J(w)}{w_j}=-\frac{1}{m}\frac{\alpha \sum_i^m (y_iln(h_w(x_i))+ (1-y_i)ln(1-h_w(x_i)))}{w_j}$

$-\frac{1}{m}(\sum_i^m(y_i \frac{\alpha lnh_w(x_i)}{\alpha w_j} +(1-y_i) \frac{\alpha ln(1-h_w(x_i))}{\alpha w_j} ))$

$-\frac{1}{m}(\sum_i^m(y_i \frac{1}{h_w(x_i)}\frac{\alpha h_w(x_i)}{z_i}\frac{\alpha z_i}{w_j} +(1-y_i)\frac{1}{1-h_w(x_i)}\frac{-\alpha h_w(x_i)}{z_i}\frac{\alpha z_i}{w_j})$

$-\frac{1}{m}(\sum_i^m(y_i \frac{h_w(x_i)(1-h_w(x_i))}{h_w(x_i)} +(1-y_i) \frac{h_w(x_i)(1-h_w(x_i))}{1-h_w(x_i)} )\frac{\alpha z_i}{w_j})$

$-\frac{1}{m}\sum_i^m(y_i-h_w(x_i))\frac{\alpha z_i}{w_j})$

$\frac{1}{m}\sum_i^m(h_w(x_i)-y_i)x_{ij}$

$w-\eta \frac{\alpha J(w)}{w_j}$

'''
Logistic Regression
(y_pre -y)x
'''
import numpy as np

class LogisticRegression:
    def __init__(self,n_iter=500, eta=1e-3, tol=None):
        self.n_iter = n_iter
        self.eta = eta
        self.tol = tol
        self.w = None

    def _process_data(self,X):
        m,n = X.shape
        X_ = np.ones([m,n+1])
        X_[:,1:]=X
        return X_

    def _sigmoid(self,z):
        return 1.0/(1.0+np.exp(-z))

    def _predict_prob(self,X,w):  # 1 处理数据；2 线性+激活
        z = np.matmul(X,w)  # [m,n+1] [n+1,1]
        return self._sigmoid(z)

    def predict(self, X):
        X= self._process_data(X)
        o = self._predict_prob(X,self.w)
        print(o.shape)
        return np.where(o > 0.5, 1, 0)

    def _loss(self,y,y_pre):
        return -np.sum(y*np.log(y_pre)+(1-y)*np.log(1-y_pre))/y.size

    def gradient(self,X,y,w):    # w -= eta *  (X.T *(y_pre-y)) [m,1]  [m,n+1]
        if self.tol:
            loss_old = np.inf

        loss_list = []
        for _ in range(self.n_iter):
            y_pre = self._predict_prob(X,w)
            loss = self._loss(y,y_pre)
            loss_list.append(loss)
            if _%100 == 0:
                print(loss)

            if self.tol:
                if loss_old - loss <self.tol:
                    break
                loss_old = loss

            w -= self.eta * np.matmul(X.T,(y_pre-y))

    def train(self,X_train,y_train):
        X_train = self._process_data(X_train)
        m,n = X_train.shape         # [20,3]
        self.w = np.random.random(n).reshape([-1,1]) # [3,1]
        self.gradient(X_train,y_train,self.w)


if __name__ == '__main__':
    x = np.random.random(10)
    x1 = np.array([x*1+2,x*1+3])
    x2 = np.array([x*2+5,x*2+7])
    X_train = np.concatenate([x1,x2],axis=0).reshape([-1,2])
    y_train = np.zeros(20).reshape([-1,1])
    y_train[10:] = 1
    log = LogisticRegression(n_iter=3000, eta=1e-3, tol=1e-5)
    log.train(X_train, y_train)
    w = log.w
    y_pre = log.predict(X_train)