Lr_policy， 持续更新.....

reference:

https://stackoverflow.com/questions/30033096/what-is-lr-policy-in-caffe/30045244

I just try to find out how I can use Caffe. To do so, I just took a look at the different .prototxt files in the examples folder. There is one option I don’t understand:

The learning rate policy

lr_policy: "inv"
Possible values seem to be:

"fixed"
"inv"
"step"
"multistep"
"stepearly"
Could somebody please explain those options?

If you look inside the /caffe-master/src/caffe/proto/caffe.proto file (you can find it online here) you will see the following descriptions:

// The learning rate decay policy. The currently implemented learning rate
// policies are as follows:
//    - fixed: always return base_lr.
//    - step: return base_lr * gamma ^ (floor(iter / step))
//    - exp: return base_lr * gamma ^ iter
//    - inv: return base_lr * (1 + gamma * iter) ^ (- power)
//    - multistep: similar to step but it allows non uniform steps defined by
//      stepvalue
//    - poly: the effective learning rate follows a polynomial decay, to be
//      zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)
//    - sigmoid: the effective learning rate follows a sigmod decay
//      return base_lr ( 1/(1 + exp(-gamma * (iter - stepsize))))
//
// where base_lr, max_iter, gamma, step, stepvalue and power are defined
// in the solver parameter protocol buffer, and iter is the current iteration.
iter=1:50000;
max_iter=50000;
base_lr=0.01;
gamma=0.0001;
power=0.75;
step_size=5000;

画图比较各lr_policy:

iter=1:50000;
max_iter=50000;
base_lr=0.01;
gamma=0.0001;
power=0.75;
step_size=5000;
min_lr=0.0005;

% - fixed: always return base_lr.
lr=base_lr*ones(1,50000);
subplot(2,4,1)
plot(lr)
title('fixed')
% - step: return base_lr * gamma ^ (floor(iter / step))
lr=base_lr .* gamma.^(floor(iter./10000));
subplot(2,4,2)
plot(lr)
title('step')
% - exp: return base_lr * gamma ^ iter
lr=base_lr * gamma .^ iter;
subplot(2,4,3)
plot(lr)
title('exp')
% - inv: return base_lr * (1 + gamma * iter) ^ (- power)
lr=base_lr.*(1./(1+gamma.*iter).^power);
subplot(2,4,4)
plot(lr)
title('inv')
% - multistep: similar to step but it allows non uniform steps defined by
% stepvalue
% - poly: the effective learning rate follows a polynomial decay, to be
% zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)
lr=base_lr *(1 - iter./max_iter) .^ (power);
subplot(2,4,5)
plot(lr)
title('poly')
% - sigmoid: the effective learning rate follows a sigmod decay
% return base_lr ( 1/(1 + exp(-gamma * (iter - stepsize))))
lr=base_lr *( 1./(1 + exp(-gamma * (iter - step_size))));
subplot(2,4,6)
plot(lr)
title('sigmoid')

%-stepdecr: return base_lr * (1 - gamma * (floor(iter / step))
%-min_lr: The minimum learning rate.
step_size=floor(max_iter/27); %27 is the number of descending. We can change it 
lr=max(min_lr,base_lr * (1 - gamma * (floor(iter / step_size))));
subplot(2,4,7)
plot(lr)
title('stepderc')

效果如下：