baiyangfu的专栏

IMHO, Flume has lots of built-in data sources, decorators (batching,archiving) and sinks (HDFS etc). So its easy to adopt on existingsystem and easy to extend, especially for logs files.Kafka co

通过这篇文章的介绍：           http://tutorials.github.io/pages/retrieving-storm-data-from-nimbus.html#.UdPrdJyZvFA写了一个简单的脚本，监控nimbus：     cluster_info = nimbus.getClusterInfo()        topology_id = Non

storm 测试集群上一个supervisor 挂掉，查看日志发现：java.io.FileNotFoundException: File '/home/q/storm/storm-data/supervisor/stormdist/Track_NotFind-66-1372059953/stormconf.ser' does not exist        at org.apache.

昨天夜里，storm的一个程序出问题了，诚惶诚恐啊。报kafka.common.OffsetOutOfRangeException 异常：分析一下原因：是由于 kafka-spout 获取kafka的数据的时候获取不到，kafka 的 offset 超出现存的范围。深入分析：        由于storm程序启动的时候，读取kafka数据的方式是从头读取（-2），当ka

要了解trident的用法和原理，需要先了解trident的state;trident state     Trident provides a fully fledged batch processing API to process those small batches. The API is very similar to what you see in high level

今天终于将 flume + kafka + storm + mysql 这条数据流走通了，虽然只是一个简单的测试例子，但是依据这条数据流可以做的事情很多。先简单看一下这几个工具的架构吧，架构图会更好说明：flume的架构图：kafka的架构图：storm的架构图：我们使用的  flume + kafka + storm +mysq

We are looking for a tool that can replace our traditional log processing model of saving activity streams into log files.  Seems like the new & popular way is to use Publish/Subscribe model.  Could S

首先声明，有些部分参考了下面文章：http://blog.sina.com.cn/s/blog_8ae7b3fe010124mr.htmlhttp://xumingming.sinaapp.com/179/twitter-storm-搭建storm集群/ https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-clus

public static void initConnectDB(){ primaryKey = "id"; rdbmsUrl = "jdbc:mysql://hadoop/fuqingwuDB" ; rdbmsUserName = "fuqingwu"; rdbmsPassword = "fuqingwu"; connector = new RDBMSConnecto