Wei_HHH的博客

Spark读写MySQL1.配置信息2.需求示例3.完整代码实现4.输出结果5.总结1.配置信息下面展示一些 内联代码片。#mysql数据库配置mysql.driver=com.mysql.jdbc.Drivermysql.url=jdbc:mysql://localhost:3306/test?serverTimezone=UTCmysql.user=******mysql.password=******2.需求示例1.mysql中score表示例数据mysql> select

一、将DStream输出到文件中Spark Streaming提供了多个上层接口，用于将DStream书出到外部文件，包括saveAsObjectFiles、saveAsTextFiles、saveAsHadoopFiles,可以分别将DStram输出到序列化文件，文本文件及Hadoop文件中。下面简单的词频统计将DStream输出到文本文件中关键步骤1、构建一个流式上线文，配置我们Spa...

Spark Streaming分析Kakfa数据环境spark-2.2.0kafka-2.11-2.30zookeeper-3.5.5kafka Producer开发package docimport java.util.Propertiesimport org.apache.kafka.clients.producer.{KafkaProducer, ProducerReco...

Spark2.2文档（一）RDDpackage doc.rddimport org.apache.spark.{SparkConf, SparkContext}/** * @Program: doc.rdd * @Author: huangwei * @Date: 2019/9/5 18:03 * @description: Spark最重要的一个抽象概念就是弹性分布式数...

Spark2.2文档（二）RDD算子package doc.rddimport org.apache.spark.util.DoubleAccumulatorimport org.apache.spark.{SparkConf, SparkContext}/** * @Program: doc.rdd * @Author: huangwei * @Date: 2019/9/...

SparkSql数据源package doc.dfimport java.util.Propertiesimport org.apache.hadoop.fs.{FileSystem, Path}import org.apache.hadoop.mapred.{FileOutputFormat, JobConf}import org.apache.hadoop.mapred.lib....

用户自定义聚合函数package doc.dfimport org.apache.spark.sql.{Row, SparkSession}import org.apache.spark.sql.expressions.{MutableAggregationBuffer, UserDefinedAggregateFunction}import org.apache.spark.sql.t...

IDEA中使用SparkSQL读写Hive添加依赖libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "2.2.0", "org.apache.spark" %% "spark-sql" % "2.2.0", ...