how to train network(李宏毅课程)

Model Bias

1. add the features
2. make your model complex, improve the level of the net

Optimization Issue

1. If deeper networks do not obtain smaller loss on training data
2. problems: gradient is close to zero(critical point)

local minima: mostly the question you face is not local minima for the dimensions you choose
saddle point

how to distinguish the two points

Overfitting

2. solution

1 more training data
2 data augmentation

3 constrained model

less parameters
sharing parameters: CNN
early stopping
less features
regularization
dropout

Mismatch

your training and testing data have different distributions

Batch

1. shuffle: every epoch, the batch is different
2. small batch

more noisy and better performance on training data and testing data
more time consume

Momentum

Gradient Descent + Momentum

Error Surface

norm of gradient is not low, but loss can not reduce

but if we reduce the learning rate, the speed of converge is too slow

Solution:
for different parameters

for the same parameter in different time

Learning rate scheduling

1. Learning rate decay: as the training goes, we are closer to the destination, so we reduce the learning rate
2. warm up: increase and then decrease

summary