《Kafka 4》--Flume与Kafka的集成

本文链接：https://blog.youkuaiyun.com/yk_3215123/article/details/99699301

本文介绍了如何将Flume配置为监控指定目录下的access.log文件，并将数据实时发送到Kafka。首先，启动Zookeeper和Kafka Broker，然后创建主题，接着配置并启动Flume Agent。确保在启动Flume之前运行Kafka消费者，最后通过发送消息到Kafka并从消费者端读取验证集成效果。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Flume与Kafka集成

1.需求：监控{/home/hyxy/tmp/flume/logs}目录下的access.log文件，将此文件的实时数据sink到Kafka；
通过Kafka的消费者消费来自Flume采集的数据；
2.编写Flume Agent
exec-kafka.conf

    a1.sources=s1
    a1.channels=c1
    a1.sinks=k1

    #定义agent的source属性
    a1.sources.s1.type=exec
    a1.sources.s1.command=tail -F /home/hyxy/Desktop/access.20120104.log

    #配置agent的sink属性 type类型要写kafa全称
    a1.sinks.k1.type=org.apache.flume.sink.kafka.KafkaSink
    a1.sinks.k1.topic=mytopic-from-flume
    a1.sinks.k1.brokerList=master:9092

    #配置agent的channel的属性
    a1.channels.c1.type = memory

    a1.sources.s1.channels = c1
    a1.sinks.k1.channel = c1

3.开启zookeeper
$>zookeeper-server-start.sh /home/hyxy/soft/kafka/config/zookeeper.properties
4.开启Kafka的Broker
$>kafka-server-start.sh /home/hyxy/soft/kafka/config/server.properties
5.创建Topic主题
$>kafka-topics.sh --zookeeper localhost:2181 --create --topic mytopic-from-flume --partitions 1 --replication-factor 1


   6.开启Flume采集系统的Agent
$>flume-ng agent -n a1 -c /home/hyxy/soft/flume/conf/ -f /home/hyxy/soft/flume/conf/exec-kafka.conf - Dflume.root.logger=INFO,console


   发送消息
   echo "helloword" /home/hyxy/Desktop/access.20120104.log ---> kafka消费者端接收消息

   7.开启消费端：
$>kafka-console-consumer.sh --zookeeper localhost:2181 --topic mytopic-from-flume --from-beginning

API测试(直接第6步，开启flume会报错，需要先开启kfafa)

1)先运行Consumer API进行消费 2) 再开启Flume采集系统的Agent

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;

import java.util.Arrays;
import java.util.Properties;

public class KafkaConsumerDemo {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "master:9092");
        props.put("group.id", "test");
        props.put("enable.auto.commit", "true");
        props.put("auto.commit.interval.ms", "1000");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
        consumer.subscribe(Arrays.asList("mytopic-from-flume"));
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(100);
            for (ConsumerRecord<String, String> record : records)
            System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
        }
    }
}