Spark Streaming介绍及基础操作

Spark Streaming介绍：
    基于Spark之上的流处理（rdd）
    流：source ==> compute ==> store
    离线是特殊的流
 
letting you write streaming jobs,the same way you write batch jobs 
 
out of the box  开箱即用 OOTB（内置的）

DStream整合RDD就需要使用transform算子

Steanming、Core 、SQL比较：
Steanming : DStream <=   represents a continuous stream of data
Core：RDD
SQL:  DF/DS

Streaming入口：StreamingContext
Core：SparkContext
SQL： 
    SparkSession
    SQLContext/HiveContext
    
编程模型一 socketTextStream：
import org.apache.spark._
import org.apache.spark.streaming._    
val ssc = new StreamingContext(sc, Seconds(5))
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" "))
val pairs = words.map(word => (word, 1))
val wordCounts = pairs.reduceByKey(_ + _)
wordCounts.print()
ssc.start()             
ssc.awaitTermination()   

该连接模式有receiver，如下图所示：

由以下提示可知，socket: receiver占用一个core 
18/09/07 22:42:41 WARN StreamingContext: 
spark.master should be set as local[n], n > 1 in local mode if you have receivers to get data, otherwise Spark jobs will not get resources to process the received data.

对DStream做一个操作，其实就是对这个DStream底层的所有RDD都做相同的操作

Direct模式：
编程模型二 textFileStream（无Receiver）：
import org.apache.spark._
import org.apache.spark.streaming._    
val ssc = new StreamingContext(sc, Seconds(10))
val lines = ssc.textFileStream("/streaming/input/")
val words = lines.flatMap(_.split("\t"))
val pairs = words.map(word => (word, 1))
val wordCounts = pairs.reduceByKey(_ + _)
wordCounts.print()
ssc.start()
ssc.awaitTermination()

 

编程模型三 CheckPoint&updateStateByKey：
def updateFunction(currentValues: Seq[Int], preValues: Option[Int]): Option[Int] = {
val curr = currentValues.sum
val pre = preValues.getOrElse(0)
Some(curr + pre)
}

import org.apache.spark._
import org.apache.spark.streaming._    
val ssc = new StreamingContext(sc, Seconds(10))
ssc.checkpoint("/streaming/checkpoint/")
val lines = ssc.socketTextStream("hadoop000",9999)
val words = lines.flatMap(_.split(" "))
val pairs = words.map(word => (word, 1))
val result = pairs.updateStateByKey(updateFunction)
result.print()
ssc.start()
ssc.awaitTermination()

打包提交：
./spark-submit \
--master local[2] \
--name StreamingStateApp \
--class com.ruozedata.spark.streaming.day01.StreamingStateApp \
/home/hadoop/lib/g3-spark-1.0.jar

即可累加计算

mapWithState（性能相对较高）：mapWithState实现有状态管理主要是通过两点：a)历史状态需要在内存中维护，updateStateBykey也是一样。b)自定义更新状态的mappingFunction，该函数视业务需求而定。
详细解释及源码分析

WAL（WriteAheadLog）：
WAL框架及实现

数据黑名单：

package com.ruozedata.spark.streaming.day02
import org.apache.spark.{SparkConf, SparkContext}
import scala.collection.mutable.ListBuffer

object LeftJoinApp {

  def main(args: Array[String]) {

    val sparkConf = new SparkConf().setAppName("LeftJoinApp").setMaster("local[2]")
    val sc = new SparkContext(sparkConf)

    // 数据一
    val input1 = new ListBuffer[(String,Long)]
    input1.append(("www.ruozedata.com", 8888))
    input1.append(("www.ruozedata.com", 9999))
    input1.append(("www.baidu.com", 7777))
    val data1 = sc.parallelize(input1)

    // 数据二
    //设置黑名单
    val input2 = new ListBuffer[(String,Boolean)]
    input2.append(("www.baidu.com",true))
    val data2 = sc.parallelize(input2)
    //根据黑名单过滤数据
    data1.leftOuterJoin(data2)
        .filter(x=>{
          x._2._2.getOrElse(false) != true
        })
        .map(x=> (x._1,x._2._1))
      .collect().foreach(println)
    sc.stop()
  }
}

foreachRDD使用的设计模式：官网中讲解了几种常见的错误用法，下述代码为推荐用法
http://spark.apache.org/docs/latest/streaming-programming-guide.html#output-operations-on-dstreams

dstream.foreachRDD { rdd =>
  rdd.foreachPartition { partitionOfRecords =>
    // ConnectionPool is a static, lazily initialized pool of connections
    //通过静态连接池来重用连接对象为最优选择，避免了重复创建对象的开销。
    val connection = ConnectionPool.getConnection()
    partitionOfRecords.foreach(record => connection.send(record))
    ConnectionPool.returnConnection(connection)  // return to the pool for future reuse
  }
}

窗口操作：
参数：窗口长度-窗口的持续时间。  滑动间隔-执行窗口操作的时间间隔。 

示例：

val windowedWordCounts = pairs.reduceByKeyAndWindow((a:Int,b:Int) => (a + b), Seconds(30), Seconds(10))