storm 读取kafka数据单机测试

最新推荐文章于 2021-12-26 09:48:20 发布

weixin_33964094

最新推荐文章于 2021-12-26 09:48:20 发布

阅读量362

点赞数

CC 4.0 BY-SA版权

文章标签：大数据 java python

原文链接：https://my.oschina.net/renzhimin/blog/1855392

本文介绍如何使用Apache Storm从Kafka中读取数据并进行本地测试。通过搭建单机版Kafka并创建topic，配置Maven依赖，编写Topology和Bolt，实现数据的读取与打印。

2019独角兽企业重金招聘Python工程师标准>>>

本地测试storm读取kafka数据然后打印。

搭建单机版kafka 创建topic ：test

1：maven依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.hbhr</groupId>
    <artifactId>storm</artifactId>
    <version>1.0-SNAPSHOT</version>
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.10</artifactId>
            <version>0.8.2.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-kafka</artifactId>
            <version>1.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.storm</groupId>
            <artifactId>storm-core</artifactId>
            <version>1.1.0</version>
            <!--<scope>provided</scope>-->
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>log4j-over-slf4j</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-api</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>5.1.2</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>transport</artifactId>
            <version>5.1.2</version>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
        </dependency>

    </dependencies>
    <build>
        <plugins>
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <mainClass>cn.ljh.storm.helloworld.ExclamationTopology</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

2：编写topology ,简单编写

public class myTopology {

    public static void main(String[] args) {
        //本地单机版zookeeper
        ZkHosts zkHosts = new ZkHosts("192.168.3.141:2181");
        //test:kafka的topic,/aa 应该是在zookeeper的目录 ,uuid唯一标识
        SpoutConfig spoutConfig = new SpoutConfig(zkHosts, "test",                                     "/aa",UUID.randomUUID().toString());
        TopologyBuilder topologyBuilder = new TopologyBuilder();
        //之前工程启动报错，work-die 日志查看为类型转换错误
        //spoutConfig.scheme =new SchemeAsMultiScheme(new StringScheme());添加这段代码应该可             //以是bolt在转换kafka数据的时候直接tuple.getString(0)获取不会出现work-die错误
        spoutConfig.scheme =new SchemeAsMultiScheme(new StringScheme());
        topologyBuilder.setSpout("kafkaSpout",new KafkaSpout(spoutConfig),1);
        topologyBuilder.setBolt("kafkaBolt",new myBolt(),1).shuffleGrouping("kafkaSpout");
        Config config = new Config();
        config.setDebug(true);
        LocalCluster localCluster = new LocalCluster();
        localCluster.submitTopology("myStorm",config,topologyBuilder.createTopology());
    }
}

3:创建bolt ：接收数据打印出来。

public class myBolt implements IRichBolt {
    OutputCollector collector;
    public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
        this.collector = outputCollector;
    }

    public void execute(Tuple tuple) {
        System.err.println(tuple.getValue(0));
    }

    public void cleanup() {

    }

    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {

    }

    public Map<String, Object> getComponentConfiguration() {
        return null;
    }
}

转载于:https://my.oschina.net/renzhimin/blog/1855392