qq_24325581的博客

一、发生oom的解决方案一、Driver 内存不够1、读取数据太大-、增加Driver内存，–driver-memory2、数据回传-、collect会造成大量数据回传Driver，使用foreach二、Executor 内存不够1、map 类操作产生大量数据，包括 map、flatMap、filter、mapPartitions等-、使用repartition，减少每个 task 计算数据的大小，从而减少每个 task 的输出-、减少中间输出：用 mapPartitions 替代多个 m

Spark版本：2.4.0源代码位置：org/apache/spark/rdd/RDD.scala应用示例：scala> val x=(1 to 10).toListx: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)scala> val df1=x.toDF("df")df1: org.apache.spark.sql.DataFrame = [number: int]scala> df1.rdd.partitions.siz

Spark版本：2.4.0源代码位置：org/apache/spark/util/AccumulatorV2.scala源代码如下：当需要long型累加器时继承如下抽象类/** * An [[AccumulatorV2 accumulator]] for computing sum, count, and average of 64-bit integers. * * @since 2.0.0 */class LongAccumulator extends AccumulatorV2[jl

Spark版本：2.4.0源代码位置：org/apache/spark/SparkContext.scala应用示例：scala> val broadcastVar = sc.broadcast(Array(1, 2, 3))broadcastVar: org.apache.spark.broadcast.Broadcast[Array[Int]] = Broadcast(0)scala> broadcastVar.valueres0: Array[Int] = Array(1,

Spark版本：2.4.0源码位置：org.apache.spark.rdd.OrderedRDDFunctions源代码如下：/** * Sort the RDD by key, so that each partition contains a sorted range of the elements. Calling * `collect` or `save` on the resulting RDD will return or output an ordered list of

Spark版本：2.4.0代码位置：org.apache.spark.rdd.PairRDDFunctionsgroupByKey(): RDD[(K, Iterable[V])]groupByKey(numPartitions: Int): RDD[(K, Iterable[V])]使用示例：val source: RDD[(Int, Int)] = sc.parallelize(Seq((1, 1), (1, 2), (2, 2), (2, 3)))val groupByKeyRDD: RD

Spark版本：2.4.0代码位置：org.apache.spark.rdd.PairRDDFunctionsfoldByKey(zeroValue: V, numPartitions: Int)(func: (V, V) => V): RDD[(K, V)]foldByKey(zeroValue: V)(func: (V, V) => V): RDD[(K, V)]应用示例object FoldByKeyDemo { def main(args: Array[String]):

Spark版本：2.4.0代码位置：org.apache.spark.rdd.PairRDDFunctions相应代码如下：这两个方法会调用方法3进行计算方法1：/** * Aggregate the values of each key, using given combine functions and a neutral "zero value". * This function can return a different result type, U, than the ty

Spark版本：2.4.0代码位置：org.apache.spark.rdd.PairRDDFunctions代码片段如下:/** * Generic function to combine the elements for each key using a custom set of aggregation * functions. This method is here for backward compatibility. It does not provide combiner

代码位置：org.apache.spark.rdd.PairRDDFunctions相关的三个方法片段:######方法1