利用caffe的python接口进行训练时遇到的问题

最新推荐文章于 2022-06-13 22:07:16 发布

原创最新推荐文章于 2022-06-13 22:07:16 发布 · 1.7k 阅读

0 ·

CC 4.0 BY-SA版权

Caffe学习同时被 2 个专栏收录

44 篇文章

订阅专栏

Deep Learning

39 篇文章

订阅专栏

本文介绍如何在Caffe的训练过程中输出各层的梯度信息，通过修改solver.cpp并重新编译，最终实现梯度输出。文章指出，若未从系统路径中移除已安装的_caffe.so，即使修改了源码也无法在训练过程中看到梯度输出。

利用caffe的python接口来训练：python ./train.py, 想输出梯度，便在solver.cpp里做了修改，添加代码：

int max_len = 0;
      for (int l = 0; l < net_->layers().size(); ++l) {
        Layer<Dtype>& layer = *net_->layers()[l].get();
        if (layer.layer_param().name().length()
            > max_len)
          max_len = layer.layer_param().name().length();
      }

      // bottom blobs and their gradient
      const vector<vector<Blob<Dtype>*> >& bottom_vecs = net_->bottom_vecs();
      for (int l = 0; l < net_->layers().size(); ++l) {
        Layer<Dtype>& layer = *net_->layers()[l].get();
        for (int b=0; b<bottom_vecs[l].size(); ++b) {
          const Blob<Dtype>* blob = bottom_vecs[l][b];
          Dtype data_sum = 0;
          Dtype diff_sum = 0;
          data_sum = blob->asum_data();
          diff_sum = blob->asum_diff();
          data_sum /= blob->count();
          diff_sum /= blob->count();
          LOG_IF(INFO, Caffe::root_solver()) << std::left << std::setw(max_len + 1) << std::setfill(' ')
                    << layer.layer_param().name()
                    << " bottom_blob " << b << ": " << std::scientific
                    << data_sum << " [" << diff_sum << "]";
        }
      }

      LOG(INFO) << std::endl;

      // output learnable weights and their gradients
      for (int l = 0; l < net_->layers().size(); ++l) {
        Layer<Dtype>& layer = *net_->layers()[l].get();
        for (int bp = 0; bp < layer.blobs().size(); ++bp) {
          Blob<Dtype>& para_blob = *layer.blobs()[bp].get();
          const Dtype* para_blob_cpu_data = para_blob.cpu_data();
          const Dtype* para_blob_cpu_diff = para_blob.cpu_diff();
          Dtype para_data_sum = 0;
          Dtype para_diff_sum = 0;
          for (int i = 0; i < para_blob.count(); ++i) {
            para_data_sum += (para_blob_cpu_data[i] > Dtype(0.)) ? para_blob_cpu_data[i]
                : - para_blob_cpu_data[i];
            para_diff_sum += (para_blob_cpu_diff[i] > Dtype(0.)) ? para_blob_cpu_diff[i]
                : - para_blob_cpu_diff[i];
          }
          para_data_sum /= para_blob.count();
          para_diff_sum /= para_blob.count();
          LOG_IF(INFO, Caffe::root_solver()) << std::left << std::setw(max_len + 1) << std::setfill(' ')
                    << layer.layer_param().name()
                    << " param_blob " << bp << ": " << std::scientific
                   << para_data_sum << " [" << para_diff_sum << "]";
        }
      } // end add by Zzp

然而，继续运行train.py，并没有输出梯度，找了很久的原因才发现，原来_caffe.so已经被安装到系统路径中，即‘/usr/lib/python2.7/dist-packages/caffe’中，所以在运行train.py的时候无论你对caffe的源码做了什么样的修改，都会默认调用系统路径里的库。所以，需要将/usr/lib/python2.7/dist-packages 中的caffe库全部删掉，然后用PYTHONPATH指定路径。

1 进入/usr/lib/python2.7/dist-packages ， ls

mini@mini:/usr/lib/python2.7/dist-packages$ ls
adium_theme_ubuntu-0.3.4.egg-info  PAM.x86_64-linux-gnu.so
ANSI.py                            pandas
ANSI.pyc                           pandas-0.13.1.egg-info
antlr.py                           patsy
antlr.pyc                          patsy-0.2.1.egg-info
apt                                pcardext.la
aptdaemon                          pcardext.so
apt_inst.so                        pep8-1.4.6.egg-info
apt_pkg.so                         pep8.py
aptsources                         pep8.pyc
apt_xapian_index-0.45.egg-info     pexpect
astroid                            pexpect-3.1.egg-info
astroid-1.0.1.egg-info             PIL
axi                                PILcompat
beautifulsoup4-4.2.1.egg-info      PILcompat.pth
blinker                            Pillow-2.3.0.egg-info
blinker-1.3.egg-info               piston_mini_client
boost                              piston_mini_client-0.7.5.egg-info
bs4                                pkg_resources.py
caffe                              pkg_resources.pyc
cairo                              protobuf-2.5.0.egg-info
chardet                            psutil
chardet-2.0.1.egg-info             psutil-1.2.1.egg-info
CommandNotFound                    _psutil_linux.x86_64-linux-gnu.so
command_not_found-0.3.egg-info     _psutil_posix.x86_64-linux-gnu.so
Crypto                             pxssh.py
cupsext.la                         pxssh.pyc
cupsext.so                         pycrypto-2.6.1.egg-info
cupshelpers                        pycups-1.9.66.egg-info
cups.so                            pydot-1.0.28.egg-info
cvxopt                             pydot.py
cvxopt-1.1.4.egg-info              pydot.pyc
Cython                             pyflakes
Cython-0.20.1post0.egg-info        pyflakes-0.8.1.egg-info
cython.py                          pygments
cython.pyc                         Pygments-1.6.egg-info
dateutil                           pygobject-3.12.0.egg-info
dbus                               pygtkcompat
_dbus_bindings.so                  pygtk.pth
_dbus_glib_bindings.so             pygtk.py
deb822.py                          pygtk.pyc
deb822.pyc                         pyinotify-0.9.4.egg-info
debconf.py                         pyinotify.py
debconf.pyc                        pyinotify.pyc
debian                             pylint
debian_bundle                      pylint-1.1.0.egg-info
debtagshw                          pyOpenSSL-0.13.egg-info
debtagshw-0.1.egg-info             pyparsing-2.0.1.egg-info
decorator-3.4.0.egg-info           pyparsing.py
decorator.py                       pyparsing.pyc
decorator.pyc                      PyQt4
defer                              pyserial-2.6.egg-info
defer-1.0.6.egg-info               pysmbc-1.0.14.1.egg-info
dirspec                            pysvn
dirspec-13.10.egg-info             python_apt-0.9.3.5ubuntu2.egg-info
docutils                           python_dateutil-1.5.egg-info
docutils-0.11.egg-info             python_debian-0.1.21_nmu2ubuntu2.egg-info
dot_parser.py                      python_gflags-1.5.1.egg-info
dot_parser.pyc                     python-support.pth
drv_libxml2.py                     pytz
drv_libxml2.pyc                    pytz-2012c.egg-info
duplicity                          pyxdg-0.25.egg-info
duplicity-0.6.23.egg-info          pyximport
easy_install.py                    PyYAML-3.10.egg-info
easy_install.pyc                   pyzmq-14.0.1.egg-info
fdpexpect.py                       README
fdpexpect.pyc                      reportlab
flask                              reportlab-3.0.egg-info
Flask-0.10.1.egg-info              requests
flaskext                           requests-2.2.1.egg-info
flask_socketio                     roman-2.0.0.egg-info
Flask_SocketIO-0.6.0.egg-info      roman.py
flask_wtf                          roman.pyc
Flask_WTF-0.11.egg-info            rope
FSM.py                             rope-0.9.2.egg-info
FSM.pyc                            samba
gevent                             scanext.la
gevent-1.0.egg-info                scanext.so
gevent_socketio-0.3.6.egg-info     scikit_image-0.9.3.egg-info
geventwebsocket                    scikit_learn-0.14.1.egg-info
gevent_websocket-0.9.3.egg-info    scipy
gflags.py                          scipy-0.13.3.egg-info
gflags.pyc                         screen.py
gflags_validators.py               screen.pyc
gflags_validators.pyc              serial
gi                                 sessioninstaller
glib                               sessioninstaller-0.0.0.egg-info
gobject                            setuptools
google                             setuptools-3.3.egg-info
greenlet-0.4.2.egg-info            setuptools.pth
greenlet.so                        simplegeneric-0.8.1.egg-info
gtk-2.0                            simplegeneric.py
gtk-2.0-pysupport-compat.pth       simplegeneric.pyc
gunicorn                           simplejson
gunicorn-17.5.egg-info             simplejson-3.3.1.egg-info
gyp                                sipconfig_nd.py
gyp-0.1.egg-info                   sipconfig_nd.pyc
h5py                               sipconfig.py
h5py-2.2.1.egg-info                sipconfig.pyc
hgext                              sip.so
hpmudext.la                        six-1.5.2.egg-info
hpmudext.so                        six.py
html5lib                           six.pyc
html5lib-0.999.egg-info            skimage
httplib2                           sklearn
httplib2-0.8.egg-info              smbc
ibus                               _smbc.so
indicator_keyboard                 socketio
IPython                            softwarecenter_aptd_plugins
ipython-1.2.1.egg-info             software_center_aptd_plugins-0.0.0.egg-info
itsdangerous-0.22.egg-info         sphinx
itsdangerous.py                    Sphinx-1.2.2.egg-info
itsdangerous.pyc                   spyder-2.2.5.egg-info
jinja2                             spyderlib
Jinja2-2.7.2.egg-info              spyderplugins
ldb.so                             ssh_import_id-3.21.egg-info
_ldb_text.py                       statsmodels
_ldb_text.pyc                      statsmodels-0.5.0.egg-info
leveldb-0.1.egg-info               system_service-0.1.6.egg-info
leveldb.so                         tables
libxml2mod.so                      tables-3.1.1.egg-info
libxml2.py                         talloc.so
libxml2.pyc                        tdb.so
lmdb                               _tdb_text.py
lmdb-0.87.egg-info                 _tdb_text.pyc
lockfile-0.8.egg-info              twisted
lockfile.py                        Twisted_Core-13.2.0.egg-info
lockfile.pyc                       Twisted_Web-13.2.0.egg-info
logilab                            ubuntu-sso-client
logilab_common-0.61.0.egg-info     ubuntu-sso-client.pth
lsb_release.py                     UbuntuSystemService
lsb_release.pyc                    unity_lens_photos-1.0.egg-info
lxml                               urllib3
lxml-3.3.3.egg-info                urllib3-1.7.1.egg-info
_markerlib                         werkzeug
markupsafe                         Werkzeug-0.9.4.egg-info
MarkupSafe-0.18.egg-info           wtforms
mercurial                          WTForms-2.0.1.egg-info
mercurial-2.8.2.egg-info           xapian
mx                                 xdg
networkx                           xdiagnose
networkx-1.8.1.egg-info            xdiagnose-3.6.3build2.egg-info
nose                               xlrd
nose-1.3.1.egg-info                xlrd-0.9.2.egg-info
numexpr                            xlwt
numexpr-2.2.2.egg-info             xlwt-0.7.5.egg-info
numpy                              yaml
numpy-1.8.2.egg-info               _yaml.so
oauthlib                           zeitgeist
oauthlib-0.6.1.egg-info            zmq
oneconf                            zope
oneconf-0.3.7.14.04.1.egg-info     zope.interface-4.0.5.egg-info
OpenSSL                            zope.interface-4.0.5-nspkg.pth

包含caffe。

2 进入/usr/lib/python2.7/dist-packages/caffe， lld _caffe.so

mini@min:/usr/lib/python2.7/dist-packages/caffe$ ldd _caffe.so  | grep caffe
    libcaffe-nv.so.0.14 => /usr/lib/x86_64-linux-gnu/libcaffe-nv.so.0.14 (0x00007f655b84b000)
#libcaffe被安装到了系统路径，所以在运行train.py的时候无论你对caffe的源码做了什么样的修改，都会默认调用系统路径里的库

3 删除

mini@mini:/usr/lib/python2.7/dist-packages$ sudo rm -rf caffe

4 编辑 .bashrc，添加PYTHONPATH环境变量export PYTHONPATH='/home/mm/SoftWare/caffe/python'，
然后..bashrc,生效
然后popd