java实现kafka+storm

最新推荐文章于 2022-07-18 15:51:00 发布

Lord_zang

最新推荐文章于 2022-07-18 15:51:00 发布

阅读量236

点赞数

本文链接：https://blog.youkuaiyun.com/Lord_zang/article/details/103292969

版权

本文介绍了一种使用Kafka作为消息中间件，结合Storm进行实时数据处理的方案。通过配置KafkaSpout从主题读取数据，利用WordBolt进行单词切分并附加时间戳，最后由WriteBolt将处理后的数据写入本地文件。该方案展示了如何在Storm中设置Spout和Bolt，以及如何在Bolt中执行数据处理和输出。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

上一篇kafka+storm+flume会说的明白，这是只是简单的实现

主方法

public class KafkaTopo {
    public static void main(String[] args) {
        String topic="dsj";
        String zkRoot="/ks";
        String spountId="kagkaSpount";
        BrokerHosts zkHosts = new ZkHosts("zjgm01:2181,zjgm02:2181,zjgm03:2181");
        TopologyBuilder builder= new TopologyBuilder();
        SpoutConfig conf=new SpoutConfig(zkHosts,topic,zkRoot,spountId);
        conf.forceFromStart=true;
        conf.scheme=new SchemeAsMultiScheme(new MessageScheme());
        builder.setSpout(spountId,new KafkaSpout(conf));
        builder.setBolt("wordBolt",new WordBolt()).shuffleGrouping(spountId);
        builder.setBolt("writeBolt",new WriteBolt()).shuffleGrouping("wordBolt");
        LocalCluster cluster=new LocalCluster();
        Config conf1=new Config();
        conf1.setNumWorkers(4);
        cluster.submitTopology("word",conf1,builder.createTopology());
    }
}

read ，这里加上了时间，其实不好但只是测试所以就先这么写了，应该在写里面写修改的东西。

public class WordBolt extends BaseBasicBolt {

    @Override
    public void execute(Tuple tuple, BasicOutputCollector basicOutputCollector) {
        String line = tuple.getString(0);
        String[] words = line.split(" ");
        for (String w:words){
            //isNotEmpty将空格也作为参数，isNotBlank则排除空格参数。
            if (StringUtils.isNotBlank(w)){
                String s=w.trim();
                SimpleDateFormat sim=new SimpleDateFormat(" yyyy-MM-dd HH:mm:ss");
                String time=sim.format(new Date().getTime());
                s+=time;
//                s=s.toLowerCase(); 小写
                basicOutputCollector.emit(new Values(s));
            }
        }
    }
    @Override
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
        outputFieldsDeclarer.declare(new Fields("s"));
    }
}

写

public class WriteBolt extends BaseBasicBolt {
    FileWriter fileWriter=null;
    //第一次会执行
    @Override
    public void prepare(Map stormConf, TopologyContext context) {
        try {
            fileWriter=new FileWriter("d:\\storm\\"+ UUID.randomUUID().toString());//存放位置
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
    @Override
    public void execute(Tuple tuple, BasicOutputCollector basicOutputCollector) {
        String s = tuple.getString(0);
        try {
            fileWriter.write(s);
            fileWriter.write("\n");
           fileWriter.flush();
        } catch (IOException e) {
            e.printStackTrace();
        }

    }
    @Override
    public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer) {
    }
}

设置kafka格式

public class MessageScheme implements Scheme {
    @Override
    public List<Object> deserialize(byte[] bytes) {
        String msg=new String(bytes);
        return new Values(msg);
    }

    @Override
    public Fields getOutputFields() {
        return new Fields("msg");
    }
}