Spark streaming 初探

关注

原创最新推荐文章于 2026-03-02 01:35:26 发布 1.1k 阅读 AI 写同款· GEO 优化 ›

ROM解包打包实战：从boot.img到system.img的完整流程 Android系统固件由boot、system、vendor等镜像分区组成，第三方ROM制作、系统精简与功能定制都需要对镜像进行解包和重新打包。理解sparse image与raw image的转换、SELinux文件标签和分区大小边界，是避免刷机变砖的关键基础。借助simg2img、mkbootimg、make_ext4fs、signapk等工具链，可实现boot.img内核参数的还原、system.img文件系统的重建及可刷入ZIP的签名。这项技术常用于CM/LineageOS系固件的二次定制、预置应用阅读详情

Spark streaming 初探

一、原理和运行场景

Spark Streaming运行原理图如下：

Spark Streaming把Kafka、HDFS、Socket等系统作为流处理的数据来源，把输入的数据流用时间切片的方式把数据分为一个个小的Batch，然后把这些小的Batch交给Spark引擎处理。

官方wordcount示例代码：

package org.apache.spark.examples.streaming

import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.storage.StorageLevel

/**
 * Counts words in UTF8 encoded, '\n' delimited text received from the network every second.
 *
 * Usage: NetworkWordCount <hostname> <port>
 * <hostname> and <port> describe the TCP server that Spark Streaming would connect to receive data.
 *
 * To run this on your local machine, you need to first run a Netcat server
 *    `$ nc -lk 9999`
 * and then run the example
 *    `$ bin/run-example org.apache.spark.examples.streaming.NetworkWordCount localhost 9999`
 */
object NetworkWordCount {
  def main(args: Array[String]) {
    if (args.length < 2) {
      System.err.println("Usage: NetworkWordCount <hostname> <port>")
      System.exit(1)
    }

    StreamingExamples.setStreamingLogLevels()

    // Create the context with a 1 second batch size
    val sparkConf = new SparkConf().setAppName("NetworkWordCount")
    val ssc = new StreamingContext(sparkConf, Seconds(1))

    // Create a socket stream on target ip:port and count the
    // words in input stream of \n delimited text (eg. generated by 'nc')
    // Note that no duplication in storage level only for running locally.
    // Replication necessary in distributed scenario for fault tolerance.
    val lines = ssc.socketTextStream(args(0), args(1).toInt, StorageLevel.MEMORY_AND_DISK_SER)
    val words = lines.flatMap(_.split(" "))
    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
    wordCounts.print()
    ssc.start()
    ssc.awaitTermination()
  }
}

以上代码首先创建StreamContext对象，然后以1秒钟的时间间隔来进行Streaming流的Batchi划分。与此同时，通过socket的方式监听数据流，然后进行每秒的单词技术操作。下图清晰描述了运行流程：