drl_blogs的博客

import org.apache.kafka.common.serialization.StringDeserializer import org.apache.log4j.{Level, Logger} import org.apache.spark.SparkConf import org.apache.spark.streaming.kafka010.ConsumerStrategies....

1.上传kafka_2.11-0.11.0.0.tgz 2.解压 tar -zxvf kafka_2.11-0.11.0.0.tgz -C /usr/local/ 3.修改server.properties vim server.properties 修改内容如 #broker的全局唯一编号，不能重复 broker.id=0 #删除topic功能使能 delete.topic.enable=t...

生成模拟数据 编写 generate_log.py #coding=UTF-8 import random import time url_paths=[ "class/112.html", "class/128.html", "class/145.html", "class/130.html", "class/146.html", "cla...

flume_kafka.conf agent1.sources = avro-source agent1.channels = logger-channel agent1.sinks = kafka-sink # define source agent1.sources.avro-source.type = avro agent1.sources.avro-source.bind = 0....

文章目录ReceiverDirect Receiver import org.apache.log4j.{Level, Logger} import org.apache.spark.SparkConf import org.apache.spark.streaming.kafka.KafkaUtils import org.apache.spark.streaming.{Seconds, Str...

接受数据后的处理逻辑 import org.apache.kafka.streams.processor.Processor; import org.apache.kafka.streams.processor.ProcessorContext; public class LogProcessor implements Processor<byte[], byte[]> { p...

拦截内容在内容前加时间戳 import java.util.Map; import org.apache.kafka.clients.producer.ProducerInterceptor; import org.apache.kafka.clients.producer.ProducerRecord; import org.apache.kafka.clients.producer.Re...