Spark Broadcast 广播变量

最新推荐文章于 2024-10-21 16:47:51 发布

WYSIWYG-所见即所得

最新推荐文章于 2024-10-21 16:47:51 发布

阅读量850

点赞数

CC 4.0 BY-SA版权

分类专栏： Spark 文章标签： spark broadcast

本文链接：https://blog.youkuaiyun.com/u010809185/article/details/52846068

Spark 专栏收录该内容

3 篇文章

订阅专栏

本文介绍Spark中的广播变量概念及其使用方法。广播变量允许程序将只读变量缓存在每台机器上，而不是为每个任务发送一份副本。这可以高效地让每个节点获得大型输入数据集的副本。文章还提供了一个具体的例子，演示如何配置并使用Spark的广播变量。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

广播变量
A broadcast variable. Broadcast variables allow the programmer to keep a read-only variable * cached on each machine rather than shipping a copy of it with tasks. They can be used, for * example, to give every node a copy of a large input dataset in an efficient manner. Spark also * attempts to distribute broadcast variables using efficient broadcast algorithms to reduce * communication cost.

package org.apache.spark.myExamples

import org.apache.spark.sql.SparkSession

/**
  * Created by wjf on 16-8-30.
  */
object TestBroadCast {
  def main(args: Array[String]): Unit = {
    val blockSize = if(args.length >2) args(2) else "4096"
    val spark =SparkSession.builder().appName("testBroadCast").config("spark.broadcast.blockSize",blockSize).master("local").getOrCreate()

    val sc = spark.sparkContext

    val slice = if(args.length >0) args(0).toInt else 2
    val num = if(args.length >1) args(1).toInt else 10000

    val arr1=(1 to num).toArray

    for(i <- 1 to 3){
      println("Iteration " + i)
      println("--------------------------")
      val start= System.nanoTime()
      val broad = sc.broadcast(arr1)
      val numSize= sc.parallelize(1 to 10,slice).map(_ => broad.value.length)
      numSize.collect().foreach(println)

      println("Iteration %d take %.0f milliseconds".format(i,(System.nanoTime()-start)/1E6) )
    }

  }

}