daimin1的博客

以下一切数据都依据本人虚拟机的真实路径数据日志文件的路径/root/data/flume/prolog.log创建kafka主题kafka-topics.sh --create --topic prolog_02 --partitions 1 --replication-factor 1 --bootstrap-server singlebrown:9092创建flume配置文件vim /root/flume_job/logconf/flume02_kafka.confa1.source

producer类（java）package cn.kgc.stock;import org.apache.kafka.clients.producer.KafkaProducer;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.clients.producer.RecordMetadata;import java.util.Properties;import java.util.

创建工程scala maven添加依赖//版本号 <scala.version>2.12.10</scala.version> <spark.version>3.0.1</spark.version> <spark.scala.version>2.12</spark.scala.version><!-- scala--> <dependency> <gr

依赖就不贴了，嘻嘻object SparkHive extends App { val spark: SparkSession = SparkSession.builder() .master("local[*]") .appName("spark_hive") .enableHiveSupport() .getOrCreate() //读取hive中数据表 /*spark.sql("select * from kb12.testsqoop")

/**spark 根据本地文件来处理数据样例类+RDD创建DataFrame*/具体代码如下：object SparkFile { case class Record(shopId:String,date:String,volume:String) def main(args: Array[String]): Unit = { val spark: SparkSession = SparkSession.builder() .master("local[*]").a

创建工程选择spark的maven工程依赖（参考）先修改scala的版本为2.12.10<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifactId> <version>3.1.2</version> </dependency> <d