flume整合kafka之kafka接收flume数据

1.创建配置文件 基本配置如下：


#source
a1.sources = sysSrc
a1.sources.sysSrc.type=avro
a1.sources.sysSrc.bind=0.0.0.0
a1.sources.sysSrc.port=33333
a1.sources.sysSrc.channels=fileChannel


#channel
a1.channels = fileChannel
a1.channels.fileChannel.type =memory
a1.channels.fileChannel.capacity=10000


#sink
a1.sinks = kafkaSink
a1.sinks.kafkaSink.channel=fileChannel
a1.sinks.kafkaSink.type=org.apache.flume.sink.kafka.KafkaSink
a1.sinks.kafkaSink.brokerList=h4:9092
a1.sinks.kafkaSink.custom.partition.key=kafkaPartition
a1.sinks.kafkaSink.topic=T20161031
a1.sinks.kafkaSink.serializer.class=kafka.serializer.StringEncoder




2. 保证zookeeper正常启动同时kafka也正常启动的情况下启动flume
bin/flume-ng agent --conf conf --conf-file conf/sink2Kafka.conf --name a1 -Dflume.root.logger=INFO,console


3. 启动kafka消费者现场 
bin/kafka-console-consumer.sh --zookeeper h2:2181,h3:2181,h4:2181,h8:2181,h9:2181,h10:2181 --topic T20161031 --from-beginning


4. 利用flume带的avro客户端向flume发送消息
bin/flume-ng avro-client --conf conf --host 0.0.0.0 --port 33333 --filename ../mydata/log2.txt


结果：可以看到kafka主题T20161031正确收到来自flume的avro客户端发送的消息


补充：其他flume的source如spooldir  netcat等按照固定规则测试即可