Storm-Kafka 一分钟例子

最新推荐文章于 2023-12-15 15:17:24 发布

原创最新推荐文章于 2023-12-15 15:17:24 发布 · 1.1k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#storm #kafka

kafka 同时被 2 个专栏收录

2 篇文章

订阅专栏

storm

1 篇文章

订阅专栏

本文介绍如何在Docker环境下配置Kafka作为Apache Storm的数据源，并通过示例代码展示如何实现这一过程。具体包括配置Docker环境、Gradle依赖、Spout配置及消息处理等关键步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

以下文章使用了kafka作为storm的sport数据源，依赖于docker-compose环境,如果自己本机已经有了zookeeper与kafka环境则可以使用自己的环境。

使用教程

docker-compose.yml

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.1.186
      KAFKA_ZOOKEEPER_CONNECT: 192.168.1.186:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

build.gradle

dependencies {
    compile group: 'org.apache.storm', name: 'storm-core', version: '1.2.2'
    compile group: 'org.apache.storm', name: 'storm-kafka-client', version: '1.2.2'
}

Application.java

public class Application {
    public static void main(String[] args) throws Exception {
        KafkaSpoutConfig.Builder<String,String> kafkaBuild = KafkaSpoutConfig.builder("192.168.1.186:32770","test","test1","test2");
        kafkaBuild.setFirstPollOffsetStrategy(KafkaSpoutConfig.FirstPollOffsetStrategy.UNCOMMITTED_LATEST);
        kafkaBuild.setOffsetCommitPeriodMs(100);//设置多长时间向kafka提交一次offset
        kafkaBuild.setProp(ConsumerConfig.GROUP_ID_CONFIG,"testGroup");
        kafkaBuild.setProp(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG,1);
        kafkaBuild.setProp(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG,0);
        KafkaSpoutConfig<String,String> build = kafkaBuild.build();
        KafkaSpout<String,String> kafkaSpout = new KafkaSpout<>(build);
        TopologyBuilder builder = new TopologyBuilder();
        builder.setSpout("kafkaSport",kafkaSpout,1);
        builder.setBolt("print-bolt",new PrintBolt(),1)
        .shuffleGrouping("kafkaSport");
        Config config = new Config();
        config.setNumWorkers(1);
        if(args.length==0){
            config.setDebug(true);
            LocalCluster localCluster = new LocalCluster();
            localCluster.submitTopology("test",config,builder.createTopology());
        }else{
            config.setDebug(false);
            StormSubmitter.submitTopology("test",config,builder.createTopology());
        }
    }
}

PrintBolt.java

public class PrintBolt extends BaseRichBolt {
    private OutputCollector collector;

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {}

    @Override
    public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
        this.collector = collector;
    }

    @Override
    public void execute(Tuple input) {
        String topic = input.getString(0);
        long messageOffet = input.getLong(2);
        String content = input.getString(4);
        System.out.println(content);
        collector.ack(input);
    }
}

往kafka发送一个消息

docker exec -ti 4f27fbb6655c bash /opt/kafka/bin/kafka-console-producer.sh --broker-list 192.168.1.186:32770 --topic test

输入消息回车即可看到程序输出

root@localhost /h/lake# docker exec -ti 4f27fbb6655c bash /opt/kafka/bin/kafka-console-producer.sh --broker-list 192.168.1.186:32770 --topic test
>hello world.
>

输出

...
312034 [Thread-20-print-bolt-executor[3 3]] INFO  o.a.s.d.executor - Processing received message FOR 3 TUPLE: source: kafkaSport:2, stream: default, id: {-3468855975737070311=-8649143164499739979}, [test, 0, 16, null, hello world.]
312034 [Thread-22-__acker-executor[1 1]] INFO  o.a.s.d.executor - Processing received message FOR 1 TUPLE: source: kafkaSport:2, stream: __ack_init, id: {}, [-3468855975737070311 -8649143164499739979 2]
hello world.
312034 [Thread-20-print-bolt-executor[3 3]] INFO  o.a.s.d.task - Emitting: print-bolt __ack_ack [-3468855975737070311 -8649143164499739979]
312034 [Thread-22-__acker-executor[1 1]] INFO  o.a.s.d.executor - BOLT ack TASK: 1 TIME: -1 TUPLE: source: kafkaSport:2, stream: __ack_init, id: {}, [-3468855975737070311 -8649143164499739979 2]
...