classification problems——logistic regression

classification problems
——where the variable y that you want to predict is valued.
negative class-positive class
“-“-“+”
y(i)->the label for the training example

way1:
linear regression
threshold the classifier outputs at 0.5
if hypothesis >= 0.5 ->y = 1
else ->y = 0
it makes sense
however,if there is a training example way out there on the right,it doesn’t actually change anything,but this magenta line out here to this blue line over here, and caused it to give us a worse hypothesis
way2:
logistic regression

$$h_\theta(x)=g(\theta^Tx)$$
sigmoid function/logistic function
$$g(z)=\frac{1}{1+e^{-z}}$$
it looks like

eventally
$h_\theta(x)=$ estimated probability y=1 on input x

namely, $h_\theta(x)=P(y=1|x;\theta)$

at the same time,

$$P(y=0|x;\theta)=1-P(y=1|x;\theta)$$
decision boundary
obviously,
$\theta^Tx\ge0\Rightarrow y=1$
$\theta^Tx\lt0\Rightarrow y=0$
then,with Linear programming we can have two divided areas

The decision boundary is the line that separates the area where y = 0 and where y = 1. It is created by our hypothesis function (not training examples).
Again, the input to the sigmoid function g(z) doesn’t need to be linear, and could be a function that describes a circle or any shape to fit our data.

cost function
cost function in linear regression:

$J(\theta)=\frac{1}{m}\sum_{i=1}^m Cost(h_\theta(x^{(i)}),y^{(i)})$
$Cost(h_\theta(x),y)=\frac{1}{2}(h_\theta(x)-y)^2$

the cost function for linear regression cannot be used, because it will be a non-convex function.
Instead,our cost function for logistic regression looks like
(writing the cost function in this way guarantees that J(θ) is convex for logistic regression):

$J(\theta)=\frac{1}{m}\sum_{i=1}^m Cost(h_\theta(x^{(i)}),y^{(i)})$
$Cost(h_\theta(x),y)= \begin{cases} -\log^{h_\theta(x)} & \text{if y=1}\\ -\log^{(1-h_\theta(x))} & \text{if y=0} \end{cases}$

if we compress them,then

$$Cost(h_\theta(x),y)=-y\log^{h_\theta(x)}-(1-y)\log^{(1-h_\theta(x))}$$
A vectorized implementation is:
$$h=g(X\theta) \\ J(\theta)=\frac{1}{m}(-y^T\log^{(h)}-(1-y)^T\log^{(1-h)})$$
When y = 1, the following plot for J(θ) vs hθ(x):

Similarly, when y = 0, the following plot for J(θ) vs hθ(x):

Gradient descent
want min J(\theta):
repeat{
$\theta_j:=\theta-\frac{\alpha}{m}\sum_{i=1}^{m} (h_\theta(x^{(i)})-y^{(i)})x_j^{(i)}$
(simultaneously update all $\theta_j$)
}
Notice that this algorithm is identical to the one we used in linear regression.
A vectorized implementation is:

$$\theta:=\theta-\frac{\alpha}{m}X^T(g(X \theta)-y)$$

multiclass classfication
we have y = {0,1…n}.
in this way, divide our problem into n binary classification problems

$$h_\theta^{(1)}(x)=P(y=1|x;\theta) \\ h_\theta^{(2)}(x)=P(y=2|x;\theta) \\ ... \\ h_\theta^{(n)}(x)=P(y=n|x;\theta) \\ prediction=max(h_\theta^{(i)}(x))$$

---

Advanced Optimization
“Conjugate gradient”, “BFGS”, and “L-BFGS” are more sophisticated, faster ways to optimize θ that can be used instead of gradient descent.Use the libraries.