[Machine Learning]--Logistic Regression

Classification with Logistic Regression and the Sigmoid Function: a Tractable Step Function

We don’t use Heaviside step function.Just as the following equation:

$$H(x) = \begin{cases} 0, & \text {$x$ < 0}\\ \frac{1}{2}, &\text{$x=0$}\\ 1, & \text {$x$ > 0} \end{cases}$$
$$\sigma(z) = \frac{1}{1+e^{-z}}$$

Logistic regression gradient ascent optimization function

Fist, i will show the code

def gradAscent(dataMatIn, classLabels):
    dataMatrix = mat(dataMatIn)
    labelMat = mat(classLabels).transpose()
    m, n = shape(dataMatrix)
    alpha = 0.001
    maxCycles = 500
    weights = ones((n, 1))
    for k in range(maxCycles):
        h = sigmoid(dataMatrix*weights)
        error = (labelMat - h)
        weights = weights + alpha * dataMatrix.transpose()*error
    return weights

However, it can be quite time-consuming. Quite expensive in terms of computing resource!
dataMatrix [m*n]
weights: [n*1]
h: [m*1]
error: [m*1]
so, dataMatrix*weights means m*n*n times calculation!
h
So, we can use the Stochastic Gradient Ascent to do the optimization.
Let’s see the code!

def stocGradAscent0(dataMatrix, classLabels):
    m, n = shape(dataMatrix)
    alpha = 0.01
    weights = ones(n)                                # weights  [1, n]
    for i in range(m):
        h = sigmoid(sum(dataMatrix[i]*weights))      # not matrix calculation, but element * element,
                                                     # dataMatrix[i]: [1, n]    weights:[1, n]
        error = classLabels[i] - h
        weights = weights + alpha * error * dataMatrix[i]
    return weights

Very similar to gradient ascent except that the variable h and the error ate now single values rather than vectors!

However the Convergence Rate is not very good, so modify this optimization algorithm.

def stocGradAscent1(dataMatrix, classLabels, numIter=150):
    m,n = shape(dataMatrix)
    weights = ones(n)
    for j in range(numIter):
        dataIndex = range(m)
        for i in range(m):
        # alpha changes on each iteration, which will improve the oscillations that occur.
            alpha = 4/(1.0 + i + j) + 0.1
            # randomly select some instance to use in updating the weights. Be sure to delete what you have used.
            randIndex = int(random.uniform(0, len(dataIndex)))
            h = sigmoid(sum(dataMatrix[randIndex]*weights))
            error = classLabels[randIndex] - h
            weights = weights + alpha * error * dataMatrix[randIndex]
    return weights

Classifying with logistic regression

The weights we get is used to classify.
See the classify function!

def classifyVector(inX, weights):
    prob = sigmoid(sum(inX*weights))
    if prob > 0.5: return 1.0
    else: return 0.0