weixin_46408961的博客

​ 目的：模拟数据不断写入，同时不断处理01.启动相关的服务​ Hdfs文件系统​ Zookeeper服务​ Kafka(主题，生产，消费)​ Hive数仓(元数据库mysql)​ Spark能够直接sql操作Hive，不然使用jdbc方式(base) [root@192 ~]# jps3200 SecondaryNameNode2835 NameNode5011 ConsoleProducer85045 Jps2998 DataNode4390 QuorumPeerMain45

I.Kafka的生产消费01.启动Zookeeper(base) [root@192 ~]# cd $ZOOKEEPER_HOME(base) [root@192 zookeeper-3.4.14]# cd bin/(base) [root@192 bin]# lsREADME.txt zkCleanup.sh zkCli.cmd zkCli.sh zkEnv.cmd zkEnv.sh zkServer.cmd zkServer.sh zkTxnLogToolkit.cmd zk

01.创建对象,设定日志级别from pyspark.sql import SparkSessionspark = SparkSession.builder.config("spark.driver.host","192.168.1.10")\ .config("spark.ui.showConsoleProgress","false")\ .appName("hdfs_hive").master("local[*]").enableHiveSupport().getOrCreate()

只做记录不展示结果(部分结果放在了代码的注释中)：package spark_rddimport org.apache.spark.sql.SparkSessionobject rdd_five extends App{ override def main(args: Array[String]): Unit = { /** * key-value RDD * pair RDD * 2021-10-31

只做记录，部分结果在代码注释部分：aggregate()：是一个聚合函数，接受多个输入，并按照一定的规则运算以后输出一个结果值getNumPartitions：获取RDD的分区数foreachPartition()：遍历分区中的元素,参数为函数，如下参数为foreach函数，foreach函数中再使用匿名函数 : x=>print(x.toString()+",")此处foreach中匿名函数相当于python中的如下用法：package spark_rddim.

只做记录，不展示结果：package spark_dataframeimport org.apache.spark.sql.SparkSessionimport scala.collection.mutable.ArrayBufferimport org.apache.spark.sql.Rowimport org.apache.spark.sql.types.StructTypeimport org.apache.spark.sql.types.StructFieldimport o

#%%from pyspark import SparkConf,SparkContext, rddfrom pyspark.sql import SparkSessionspark = SparkSession.builder.appName("app")\ .getOrCreate()sc = spark.sparkContext#%%import pyspark.sql.functions as Ffrom pyspark.sql.types import *#%%list.

1.代码如下：package spark_dataframeimport org.apache.spark.sql.SparkSessionimport org.apache.spark.sql.types._import org.apache.spark.sql.functionsimport org.apache.spark.sqlobject functionsdf extends App{ override def main(args: Array[String]): U

只记录，不展示结果package spark_dataframeimport org.apache.spark.sql.SparkSessionimport org.apache.spark.sql.types._import scala.collection.mutable.ArrayBufferimport org.apache.spark.sql.Row/** * @yanyyi * 2021-10-29 */object dataframe_one extends

只做记录，不展示运行结果：package spark_rddimport org.apache.spark.SparkConfimport org.apache.spark.SparkContextimport org.apache.spark.sql.SparkSessionobject rdd_three extends App{ override def main(args: Array[String]): Unit = { val spark = SparkS

引入相关的库，生成一个SparkContext对象from pyspark.sql import SparkSessionfrom pyspark import SparkConf,SparkContext, confspark = SparkSession.builder.config(conf=SparkConf()).getOrCreate()from pyspark.sql import Rowfrom pyspark.sql.types import *方式1：利用反射机制推断R

准备工作：将mysql的jar包放在$SPARK_HOME/jars目录下1.生成一个SparkSession()对象，并导入相关的库和接口from pyspark.sql import SparkSessionfrom pyspark import SparkConf,SparkContext, SparkConfspark = SparkSession.builder.config(conf=SparkConf()).getOrCreate()from pyspark.sql impor

1.parallelize()创建RDD：words = sc.parallelize([("hadoop",1),("is",1),("good",1),\ ("spark",1),("is",1),("fast",1),("spark",1),("is",1),\ ("better",1)])wordsres1 = words.groupByKey()wordsres1.collect()2.groupByKey()结果：[('hadoop', <pys

RDD的构建方式一：textFile() 其中的word.txt文件为：Hadoop is goodSpark is goodSpark is betterfrom pyspark.context import SparkContextfrom pyspark.sql.session import SparkSessionsc = SparkContext("local")s...