之前用caffe时,权值是用的xavierFiller填充的,配合以比较小的学习速率(SGD),发现效果出奇好,随便配置下网络,就能干到94%+的准确率,不论训练集还是测试集。
不知道原因找的准不准,贴出配置,供查阅。
下面是solver的配置:
# The train/test net protocol buffer definition
net: "./car_detector_train_test.prototxt"
# test_iter specifies how many forward passes the test should carry out.
# In the case of PLAN3, we have test batch size 64 and 2 test iterations,
# covering the full 106 testing images.
test_iter: 2
# Carry out testing every 100 training iterations.
test_interval: 100
# The base learning rate, momentum and the weight decay of the network.
base_lr: 0.01
momentum: 0.9
weight_decay: 0.05 #0.0005
# The learning rate policy
lr_policy: "inv"
gamma: 0.0001
power: 0.75
# Display every 100 iterations
display: 100
# The maximum number of iterations
max_iter: 10000
# snapshot intermediate results
snapshot: 200
snapshot_prefix: "./solve/car_detector"
# solver mode: CPU or GPU
solver_mode: CPU
下面是model的配置:
name: "car_detector_train_test"
layer {
name: "car_detector"
type: "Data"
top: "data"
top: "label"
include {
phase: TRAIN
}
transform_param {
mean_file: "../train_set/car_detector_mean.binaryproto"
scale: 0.00390625
}
data_param {
source: "../train_set"
batch_size: 64
backend: LMDB
mirror: true
}
}
layer {
name: "car_detector"
type: "Data"
top: "data"
top: "label"
include {
phase: TEST
}
transform_param {
mean_file: "../train_set/car_detector_mean.binaryproto"
scale: 0.00390625
}
data_param {
source: "../test_set"
batch_size: 64
backend: LMDB
}
}
layer {
name: "conv1"
type: "Convolution"
bottom: "data"
top: "conv1"
param {
name: "conv1_w"
lr_mult: 1
}
param {
name: "conv1_b"
lr_mult: 2
}
convolution_param {
num_output: 20
kernel_size: 5
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "pool1"
type: "Pooling"
bottom: "conv1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "conv2"
type: "Convolution"
bottom: "pool1"
top: "conv2"
param {
name: "conv2_w"
lr_mult: 1
}
param {
name: "conv2_b"
lr_mult: 2
}
convolution_param {
num_output: 40
kernel_size: 5
stride: 1
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "pool2"
type: "Pooling"
bottom: "conv2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: "ip1"
type: "InnerProduct"
bottom: "pool2"
top: "ip1"
param {
name: "ip1_w"
lr_mult: 1
}
param {
name: "ip1_b"
lr_mult: 2
}
inner_product_param {
num_output: 10
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "relu1"
type: "ReLU"
bottom: "ip1"
top: "ip1"
}
layer {
name: "ip2"
type: "InnerProduct"
bottom: "ip1"
top: "ip2"
param {
name: "ip2_w"
lr_mult: 1
}
param {
name: "ip2_b"
lr_mult: 2
}
inner_product_param {
num_output: 2
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
layer {
name: "accuracy"
type: "Accuracy"
bottom: "ip2"
bottom: "label"
top: "accuracy"
#include {
# phase: TEST
#}
}
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "ip2"
bottom: "label"
top: "loss"
}
#layer {
# name: "prob"
# type: "Softmax"
# bottom: "ip2"
# top: "prob"
#}
解释下solver的配置:https://github.com/BVLC/caffe/blob/master/src/caffe/solver.cpp
SGD优化器(默认的),
lr_policy inv: base_lr * (1 + gamma * iter) ^ (- power) ,这里的iter我认为是计算过的batch的数量,比如batch=64的话,每计算过64个样本算一个迭代,iter++。
// policies are as follows:
// - fixed: always return base_lr.
// - step: return base_lr * gamma ^ (floor(iter / step))
// - exp: return base_lr * gamma ^ iter
// - inv: return base_lr * (1 + gamma * iter) ^ (- power)
// - multistep: similar to step but it allows non uniform steps defined by
// stepvalue
// - poly: the effective learning rate follows a polynomial decay, to be
// zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)
// - sigmoid: the effective learning rate follows a sigmod decay
// return base_lr ( 1/(1 + exp(-gamma * (iter - stepsize))))
那么在torch中时怎么一回事呢?如果使用的是optim包中的sgd,查看代码可见:https://github.com/torch/optim/blob/master/sgd.lua
local clr = lr / (1 + nevals*lrd)
lr是基础学习速率,nevals是调用optimMethod(。。。)的次数,也就是一次batch一次。lrd是学习速率缩小因子。
这样一来,通过降低lrd,如调整为0.0004,这个值在我的实验中大约为完整的过一次样本库,降低量刚好为0.1。
lr=0.01的情况下,重现了比较好的结果。
所以,学习率是要缩小的,但是不能太快。至于基础学习速率,0.01~0.001差不多,根据caffe网站上的建议:http://caffe.berkeleyvision.org/tutorial/solver.html