lv_yishi的博客

通过一些简单的案例，可以知道一些大致的用法1.对每一个批次的数据进行操作：import org.apache.spark.streaming.dstream.{DStream, ReceiverInputDStream}import org.apache.spark.streaming.{Seconds, StreamingContext}import org.apache.spar...

UDF:计算单词的长度import org.apache.spark.sql.types.{StringType, StructField, StructType}import org.apache.spark.sql.{Row, SparkSession}object udfDemo { def main(args: Array[String]): Unit = { va...

import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}object StructuredStreamingDemo { def main(args: Array[String]): Unit = { //获取程序入口sparkSession val sparksession = SparkSession...

直连方式：import kafka.common.TopicAndPartitionimport kafka.message.MessageAndMetadataimport kafka.serializer.StringDecoderimport kafka.utils.{ZKGroupTopicDirs, ZkUtils}import org.I0Itec.zkclient.Zk...

import org.apache.spark.SparkConfimport org.apache.spark.streaming.dstream.ReceiverInputDStreamimport org.apache.spark.streaming.kafka.KafkaUtilsimport org.apache.spark.streaming.{Seconds, Streami...

zookeeper的启动脚本：#!/bin/shecho "start zookeeper server..."hosts="hadoop0300 hadoop0301 hadoop0302"for host in $hostsdo ssh $host "source /etc/profile; /root/app/zookeeper-3.4.7/bin/zkServer....

import org.apache.spark.SparkConfimport org.apache.spark.streaming.{Seconds, StreamingContext}object windowOpObj { def main(args: Array[String]): Unit = { val conf = new SparkConf().setAppN...

import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}object sparkSqlWC { def main(args: Array[String]): Unit = { //定义一个sparkSession,getOrCreate相当于单例，如果有SparkSession对象直接使用，没有就直接创...

（1）RDD之间的joinimport org.apache.spark.sql.SparkSessionobject joinDemo { //BroadcastHashJoin def main(args: Array[String]): Unit = { val sparkSession = SparkSession.builder().appName("join...

 （1）DataFrame使用import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession}object sparkSqlWC { def main(args: Array[String]): Unit = { //定义一个sparkSession,getOrCreate相当于单例，如果有SparkS...

（1）SparkWordCountimport org.apache.spark.{SparkConf, SparkContext}import org.apache.spark.rdd.RDDobject SparkWordCount { def main(args: Array[String]): Unit = { //创建配置信息 val sparkConf...

（1）使用反射的方式import org.apache.spark.sql.{DataFrame, SQLContext}import org.apache.spark.{SparkConf, SparkContext}object sprkSql { def main(args: Array[String]): Unit = { val conf = new SparkC...

排序的方式可以分为6中：（1）使用一个自定义一个普通的类继承Ordered[User] with Serializableimport org.apache.spark.rdd.RDDimport org.apache.spark.{SparkConf, SparkContext}object CustomSort1 { //排序规则：首先按照颜值的降序，如果颜值相等，再按照年...

 /*transform操作，应用在DStream上时，可以用于执行任意的RDD到RDD的转换操作。它可以用于实现，DStream API中所没有提供的操作。比如说，DStream API中，并没有提供将一个DStream中的每个batch，与一个特定的RDD进行join的操作。但是我们自己就可以使用transform操作来实现该功能。DStream.join()，只能join其他D...

窗口的一些简单操作import org.apache.spark.SparkConfimport org.apache.spark.streaming.{Seconds, StreamingContext}object windowOpObj { def main(args: Array[String]): Unit = { val conf = new SparkCon...

sparkStreaming-获取kafka数据并按批次累加import org.apache.spark.{HashPartitioner, SparkConf}import org.apache.spark.streaming.dstream.ReceiverInputDStreamimport org.apache.spark.streaming.kafka.KafkaUtils...