scribe 研究

最新推荐文章于 2025-08-12 18:06:10 发布

wgr0218

最新推荐文章于 2025-08-12 18:06:10 发布

阅读量998

点赞数

CC 4.0 BY-SA版权

文章标签： dependencies behavior library python facebook ruby

本文链接：https://blog.youkuaiyun.com/wgr0218/article/details/5314276

本文详细介绍如何在CentOS5环境下安装Facebook发布的分布式日志收集系统Scribe。从安装基础依赖到配置环境变量，再到最终安装Scribe，每一步都有详细说明。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

scribe 研究 - 一种分布式日志系统（ facebook ）

一张 scribe 结构图

先转一下，留作后用。

Installing Scribe For Log Collection

Scribe is a newly released log collection tool that dumps log files from various nodes in a cluster to Scribe servers, where the logs are stored for further use. Facebook describes their usage of Scribe by saying, “[Scribe] runs on thousands of machines and reliably delivers tens of billions of messages a day.” It turns out that Scribe is rather difficult to install, so the hope of this post is to help those of you attempting to install Scribe. The first step is to get dependencies installed.

Dependencies
Scribe has many dependencies that must be installed in order for Scribe to be built properly. They are listed here:

ruby (ruby and ruby-dev)
python (python and python-dev)
libevent (libevent and libevent-dev)
boost v1.36
Thrift
fb303 (included in Thrift in contrib/fb303)

The order in which these are installed is important. First, you must install libevent, then libevent-dev, then boost, then Thrift, and finally fb303. I installed libevent and libevent-dev from RPMs, whereas boost, Thrift, and fb303 were installed from source. I was unable to get Scribe, Thrift, and fb303 to locate the boost libraries and includes correctly with the default boost install directories, so I installed boost in /usr/local/boost, /usr/local/boost/bin, /usr/local/boost/lib, and /usr/local/boost/include. Run ‘./configure –help’ when configuring boost to see how to specify these options.

When configuring Thrift and fb303, you must specify your location of boost with the “–with-boost=/path/to/boost/root” option and also set your BOOST_ROOT environment variable.

Finally, you must make sure your LD_LIBRARY_PATH environment variable contains the lib folders that house the Thrift, fb303, boost, and libevent C++ libraries. LD_LIBRARY_PATH follows the same pattern as the PATH variable. That is, directories that contain libraries are separated by colons. If you forget to set your LD_LIBRARY_PATH variable, then you’ll get the following error when running scribed:

scribed: error while loading shared libraries: libboost_system-gcc41-mt-1_36.so.1.36.0: cannot open shared object file: No such file or directory

Install Scribe

Once you’ve successfully installed all dependencies, installing Scribe is easy. Scribe ships with a fairly comprehensive README file, but the instructions involving boost’s configuration are slightly incorrect. I needed to pass the “–with-boost=/usr/local/boost” options while configuring. However, the README file says to use “–with-boost /usr/local/boost”. Here is my full configure statement:

./configure –with-boost=/usr/local/boost –with-boost-system=boost_system-gcc41-mt-1_36 –with-boost-filesystem=boost_filesystem-gcc41-mt-1_36

We installed Scribe from trunk instead of releases/scribe-2.0.

Configure Scribe

Scribe ships with a few good examples in $SCRIBE_ROOT/examples; just take a look at the README in the examples directory, and you should be ready to rock. However, the README doesn’t document that the scribe_ctrl and scribe_cat programs are in the examples directory.

I hope this tutorial was helpful! Send us an email if you have any issues. We’ll make a follow-up post later talking about more in-depth Scribe configurations and benchmarking.

Tuesday, October 28th, 2008 at 9:13 pm by Alex Loddengaard, filed under data collection .

转自：

http://www.cloudera.com/blog/2008/10/28/installing-scribe-for-log-collection/

centos 5 安装

在开源 Thrift ， Hive 和 Cassandra 之后，Facebook 于2008 年10 月29 日开源了其分布式日志收集框架Scribe ，而且一上来就是stable 的2.0 版本。Scribe 是一个用于分布式收集日志的Thrift 服务，Facebook 描述它” 运行于数以千计的节点之上，每天可靠的发送数以百亿的消息“ 。

Scribe 本身的代码量不大，但安装相当烦琐，本文的安装环境是CentOS 5 ，其他distribution 如Ubuntu 的安装类似。Scribe 安装的依赖项包括：

* ruby (ruby and ruby-dev)
* python (python and python-dev)
* libevent (libevent and libevent-dev)
* boost 1.36
* thrift
* fb303

首先是安装ruby, python 和libevent ，这些比较容易，
yum install ruby ruby-devel
yum install python python-devel
yum install libevent libevent-devel

然后是boost ，Scribe 要求1.36 版本，先去boost 官方(http://www.boost.org/users/download/) 上下一个安装包，如果没装g++ ，

yum install gcc-c++

假设boost 安装的目录为/usr/local/boost

tar xzvf boost_1_36_0.tar.gz
cd boost_1_36_0
./configure --prefix=/usr/local/boost
make install

boost 编译需要的时间比较长，安装完boost 就可以装Thrift 了，注意要先thrift 依赖automake,libtool 等

yum install automake byacc libtool flex bison
wget -O thrift.tgz "http://gitweb.thrift-rpc.org/?p=thrift.git;a=snapshot;h=HEAD;sf=tgz"
tar -xzf thrift.tgz
cd thrift
./configure --with-boost=/usr/local/boost
make
make install

然后安装fb303,facebook 的支持库, 源文件在thrift 目录下的contrib/fb303
./configure --with-boost=/usr/local/boost
make
make install

终于可以装Scribe 了
设置环境变量BOOST_ROOT 为boost 安装路径
export BOOST_ROOT=/usr/local/boost

设置环境变量LD_LIBRARY_PATH ，指定包含Thrift, fb303, boost, libevent 的路径
export LD_LIBRARY_PATH=/usr/local/boost/lib::/usr/lib:/usr/local/lib

安装
./configure --with-boost=/usr/local/boost
make
make install

安装后可以试一下scribe 自带的例子，在目录examples 下。