From perceptrons to machine learning (一）

1. Start from perceptrons
   1.1 Developed in the 1950s and 1960s by the scientist Frank Rosenblatt.
   A perceptron takes several binary inputs, x1, x2, …, and produces a single binary output with variant weights.
   
   By varying the weights and the threshold, we can get different models of decision-making.
   
   简单理解： 我要决定明天是否去听一个音乐会，会影响这个决定的因素如下：
   a. 我女朋友去不去
   b. 会不会下雨
   c. 这个音乐会是不是很吸引我
   每个因素会有个权重，也就是weight， 假设每个因素输入都是0和1， 那么乘以权重，跟自己设定的threshold对比就可以决定是否去参加！权重决定了每个因素的重要性。

1.2 Modify the way we describe perceptrons as below, introduce bias(b) instead of threshold.

it’s interesting that perceptrons can be used to compute the elementary logic functions(与或门一类的，参考数字电路）
1 * (-2) + 1 * (-2) + 3 = -1

Sample of NAND gate.

2. From perceptrons to machine learning.
   2.1 What is machine learning
   Now the problem is that we don’t know the weights and bias in above perceptrons, but we have a lot of inputs-outputs which we can refer to. For example :
   person a : 我女朋友去，下雨，我喜欢这个音乐会 ->我去
   person b : 我女朋友去，不下雨，我喜欢这个音乐会 ->我去。。。
   person x : …
   We need to know the weights and bias that we can setup the perceptron for future decision making.

We can setup some initial value(randomly) of weights and bias, then use the sample inputs to do calculation. Compare with the actual results, to know the differences. Then we can adjust the weights and bias according to the results, then run the perceptons again. --this is machine learning.

2.2 Introduce Sigmoid neurons
In above machine learning procedures, we need to adjust the weights and bias according to result differences. Then we can modify the weights and bias a little, and we want to see just small changes in the output. But go back to the above perceptron, we can see that the output’s chart as below, the output can only be 0 or 1. We cannot have small change of output with small change of weights and bias.

We need to make the chart smooth, and differentiable, so we introduced sigmoid function.
σ(z) = 1/(1+exp(-z)). As below chart, the smoothness of the sigmoid function means small changes of W and B will produce a small change of output.