【spark系列3】spark 3.0.1 AQE(Adaptive Query Exection)分析

Spark调优学习记录(十四)AQE 尽可能更多地申请资源,所以当资源不太多时,适当减小executorAllocationRatio,控制申请maxExecutors。当合并和倾斜同时使用时,会先合并,再调整倾斜。 阅读详情

AQE简介

从spark configuration,到在最早在spark 1.6版本就已经有了AQE;到了spark 2.x版本,intel大数据团队进行了相应的原型开发和实践;到了spark 3.0时代,Databricks和intel一起为社区贡献了新的AQE

spark 3.0.1中的AQE的配置

配置项默认值官方说明分析
spark.sql.adaptive.enabledfalse是否开启自适应查询此处设置为true开启
spark.sql.adaptive.coalescePartitions.enabledtrue是否合并临近的shuffle分区(根据’spark.sql.adaptive.advisoryPartitionSizeInBytes’的阈值来合并)此处默认为true开启,分析见: 分析1
spark.sql.adaptive.coalescePartitions.initialPartitionNum(none)shuffle合并分区之前的初始分区数,默认为spark.sql.shuffle.partitions的值分析见:分析2
spark.sql.adaptive.coalescePartitions.minPartitionNum(none)shuffle 分区合并后的最小分区数,默认为spark集群的默认并行度分析见: 分析3
spark.sql.adaptive.advisoryPartitionSizeInBytes64MB建议的shuffle分区的大小,在合并分区和处理join数据倾斜的时候用到分析见:分析3
spark.sql.adaptive.skewJoin.enabledtrue是否开启join中数据倾斜的自适应处理
spark.sql.adaptive.skewJoin.skewedPartitionFactor5数据倾斜判断因子,必须同时满足skewedPartitionFactor和skewedPartitionThresholdInBytes分析见:分析4
spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes256MB数据倾斜判断阈值,必须同时满足skewedPartitionFactor和skewedPartitionThresholdInBytes分析见:分析4
spark.sql.adaptive.logLeveldebug配置自适应执行的计划改变日志调整为info级别,便于观察自适应计划的改变
spark.sql.adaptive.nonEmptyPartitionRatioForBroadcastJoin0.2转为broadcastJoin的非空分区比例阈值,>=该值,将不会转换为broadcastjoin分析见:分析5

分析1

在OptimizeSkewedJoin.scala中,我们看到ADVISORY_PARTITION_SIZE_IN_BYTES,也就是spark.sql.adaptive.advisoryPartitionSizeInBytes被引用的地方, (OptimizeSkewedJoin是物理计划中的规则)

 /**
   * The goal of skew join optimization is to make the data distribution more even. The target size
   * to split skewed partitions is the average size of non-skewed partition, or the
   * advisory partition size if avg size is smaller than it.
   */
  private def targetSize(sizes: Seq[Long], medianSize: Long): Long = {
    val advisorySize = conf.getConf(SQLConf.ADVISORY_PARTITION_SIZE_IN_BYTES)
    val nonSkewSizes = sizes.filterNot(isSkewed(_, medianSize))
    // It's impossible that all the partitions are skewed, as we use median size to define skew.
    assert(nonSkewSizes.nonEmpty)
    math.max(advisorySize, nonSkewSizes.sum / nonSkewSizes.length)
  }

其中:

  1. nonSkewSizes为task非倾斜的分区
  2. targetSize返回的是max(非倾斜的分区的平均值,advisorySize),其中advisorySize为spark.sql.adaptive.advisoryPartitionSizeInBytes值,所以说
    targetSize不一定是spark.sql.adaptive.advisoryPartitionSizeInBytes值
  3. medianSize值为task的分区大小的中位值

分析2

在SQLConf.scala

def numShufflePartitions: Int = {
    if (adaptiveExecutionEnabled && coalesceShufflePartitionsEnabled) {
      getConf(COALESCE_PARTITIONS_INITIAL_PARTITION_NUM).getOrElse(defaultNumShufflePartitions)
    } else {
      defaultNumShufflePartitions
    }
  }

从spark 3.0.1开始如果开启了AQE和shuffle分区合并,则用的是spark.sql.adaptive.coalescePartitions.initialPartitionNum,这在如果有多个shuffle stage的情况下,增加分区数,可以有效的增强shuffle分区合并的效果

分析3

在CoalesceShufflePartitions.scala,CoalesceShufflePartitions是一个物理计划的规则,会执行如下操作

 if (!shuffleStages.forall(_.shuffle.canChangeNumPartitions)) {
      plan
    } else {
      // `ShuffleQueryStageExec#mapStats` returns None when the input RDD has 0 partitions,
      // we should skip it when calculating the `partitionStartIndices`.
      val validMetrics = shuffleStages.flatMap(_.mapStats)

      // We may have different pre-shuffle partition numbers, don't reduce shuffle partition number
      // in that case. For example when we union fully aggregated data (data is arranged to a single
      // partition) and a result of a SortMergeJoin (multiple partitions).
      val distinctNumPreShufflePartitions =
        validMetrics.map(stats => stats.bytesByPartitionId.length).distinct
      if (validMetrics.nonEmpty && distinctNumPreShufflePartitions.length == 1) {
        // We fall back to Spark default parallelism if the minimum number of coalesced partitions
        // is not set, so to avoid perf regressions compared to no coalescing.
        val minPartitionNum = conf.getConf(SQLConf.COALESCE_PARTITIONS_MIN_PARTITION_NUM)
          .getOrElse(session.sparkContext.defaultParallelism)
        val partitionSpecs = ShufflePartitionsUtil.coalescePartitions(
          validMetrics.toArray,
          advisoryTargetSize = conf.getConf(SQLConf.ADVISORY_PARTITION_SIZE_IN_BYTES),
          minNumPartitions = minPartitionNum)
        // This transformation adds new nodes, so we must use `transformUp` here.
        val stageIds = shuffleStages.map(_.id).toSet
        plan.transformUp {
          // even for shuffle exchange whose input RDD has 0 partition, we should still update its
          // `partitionStartIndices`, so that all the leaf shuffles in a stage have the same
          // number of output partitions.
          case stage: ShuffleQueryStageExec if stageIds.contains(stage.id) =>
            CustomShuffleReaderExec(stage, partitionSpecs, COALESCED_SHUFFLE_READER_DESCRIPTION)
        }
      } else {
        plan
      }
    }
  }

也就是说:

  1. 如果是用户自己指定的分区操作,如repartition操作,spark.sql.adaptive.coalescePartitions.minPartitionNum无效,且跳过分区合并优化
  2. 如果多个task进行shuffle,且task有不同的分区数的话,spark.sql.adaptive.coalescePartitions.minPartitionNum无效,且跳过分区合并优化
  3. 见ShufflePartitionsUtil.coalescePartition分析

分析4

在OptimizeSkewedJoin.scala中,我们看到

/**
   * A partition is considered as a skewed partition if its size is larger than the median
   * partition size * ADAPTIVE_EXECUTION_SKEWED_PARTITION_FACTOR and also larger than
   * ADVISORY_PARTITION_SIZE_IN_BYTES.
   */
  private def isSkewed(size: Long, medianSize: Long): Boolean = {
    size > medianSize * conf.getConf(SQLConf.SKEW_JOIN_SKEWED_PARTITION_FACTOR) &&
      size > conf.getConf(SQLConf.SKEW_JOIN_SKEWED_PARTITION_THRESHOLD)
  }
  1. OptimizeSkewedJoin是个物理计划的规则,会根据isSkewed来判断是否数据数据有倾斜,而且必须是满足SKEW_JOIN_SKEWED_PARTITION_FACTOR和SKEW_JOIN_SKEWED_PARTITION_THRESHOLD才会判断为数据倾斜了
  2. medianSize为task的分区大小的中位值

分析5

在AdaptiveSparkPlanExec方法getFinalPhysicalPlan中调用了reOptimize方法,而reOptimize方法则会执行逻辑计划的优化操作:

private def reOptimize(logicalPlan: LogicalPlan): (SparkPlan, LogicalPlan) = {
    logicalPlan.invalidateStatsCache()
    val optimized = optimizer.execute(logicalPlan)
    val sparkPlan = context.session.sessionState.planner.plan(ReturnAnswer(optimized)).next()
    val newPlan = applyPhysicalRules(sparkPlan, preprocessingRules ++ queryStagePreparationRules)
    (newPlan, optimized)
  }

而optimizer 中有个DemoteBroadcastHashJoin规则:

@transient private val optimizer = new RuleExecutor[LogicalPlan] {
    // TODO add more optimization rules
    override protected def batches: Seq[Batch] = Seq(
      Batch("Demote BroadcastHashJoin", Once, DemoteBroadcastHashJoin(conf))
    )
  }

而对于DemoteBroadcastHashJoin则有对是否broadcastjoin的判断:

case class DemoteBroadcastHashJoin(conf: SQLConf) extends Rule[LogicalPlan] {

  private def shouldDemote(plan: LogicalPlan): Boolean = plan match {
    case LogicalQueryStage(_, stage: ShuffleQueryStageExec) if stage.resultOption.isDefined
      && stage.mapStats.isDefined =>
      val mapStats = stage.mapStats.get
      val partitionCnt = mapStats.bytesByPartitionId.length
      val nonZeroCnt = mapStats.bytesByPartitionId.count(_ > 0)
      partitionCnt > 0 && nonZeroCnt > 0 &&
        (nonZeroCnt * 1.0 / partitionCnt) < conf.nonEmptyPartitionRatioForBroadcastJoin
    case _ => false
  }

  def apply(plan: LogicalPlan): LogicalPlan = plan.transformDown {
    case j @ Join(left, right, _, _, hint) =>
      var newHint = hint
      if (!hint.leftHint.exists(_.strategy.isDefined) && shouldDemote(left)) {
        newHint = newHint.copy(leftHint =
          Some(hint.leftHint.getOrElse(HintInfo()).copy(strategy = Some(NO_BROADCAST_HASH))))
      }
      if (!hint.rightHint.exists(_.strategy.isDefined) && shouldDemote(right)) {
        newHint = newHint.copy(rightHint =
          Some(hint.rightHint.getOrElse(HintInfo()).copy(strategy = Some(NO_BROADCAST_HASH))))
      }
      if (newHint.ne(hint)) {
        j.copy(hint = newHint)
      } else {
        j
      }
  }
}

shouldDemote就是对是否进行broadcastjoin的判断:

  1. 首先得是ShuffleQueryStageExec操作
  2. 如果非空分区比列大于nonEmptyPartitionRatioForBroadcastJoin,也就是spark.sql.adaptive.nonEmptyPartitionRatioForBroadcastJoin,则不会把mergehashjoin转换为broadcastJoin
  3. 这在sql中先join在groupby的场景中比较容易出现

ShufflePartitionsUtil.coalescePartition分析(合并分区的核心代码)

见coalescePartition如示:

def coalescePartitions(
      mapOutputStatistics: Array[MapOutputStatistics],
      advisoryTargetSize: Long,
      minNumPartitions: Int): Seq[ShufflePartitionSpec] = {
    // If `minNumPartitions` is very large, it is possible that we need to use a value less than
    // `advisoryTargetSize` as the target size of a coalesced task.
    val totalPostShuffleInputSize = mapOutputStatistics.map(_.bytesByPartitionId.sum).sum
    // The max at here is to make sure that when we have an empty table, we only have a single
    // coalesced partition.
    // There is no particular reason that we pick 16. We just need a number to prevent
    // `maxTargetSize` from being set to 0.
    val maxTargetSize = math.max(
      math.ceil(totalPostShuffleInputSize / minNumPartitions.toDouble).toLong, 16)
    val targetSize = math.min(maxTargetSize, advisoryTargetSize)

    val shuffleIds = mapOutputStatistics.map(_.shuffleId).mkString(", ")
    logInfo(s"For shuffle($shuffleIds), advisory target size: $advisoryTargetSize, " +
      s"actual target size $targetSize.")

    // Make sure these shuffles have the same number of partitions.
    val distinctNumShufflePartitions =
      mapOutputStatistics.map(stats => stats.bytesByPartitionId.length).distinct
    // The reason that we are expecting a single value of the number of shuffle partitions
    // is that when we add Exchanges, we set the number of shuffle partitions
    // (i.e. map output partitions) using a static setting, which is the value of
    // `spark.sql.shuffle.partitions`. Even if two input RDDs are having different
    // number of partitions, they will have the same number of shuffle partitions
    // (i.e. map output partitions).
    assert(
      distinctNumShufflePartitions.length == 1,
      "There should be only one distinct value of the number of shuffle partitions " +
        "among registered Exchange operators.")

    val numPartitions = distinctNumShufflePartitions.head
    val partitionSpecs = ArrayBuffer[CoalescedPartitionSpec]()
    var latestSplitPoint = 0
    var coalescedSize = 0L
    var i = 0
    while (i < numPartitions) {
      // We calculate the total size of i-th shuffle partitions from all shuffles.
      var totalSizeOfCurrentPartition = 0L
      var j = 0
      while (j < mapOutputStatistics.length) {
        totalSizeOfCurrentPartition += mapOutputStatistics(j).bytesByPartitionId(i)
        j += 1
      }

      // If including the `totalSizeOfCurrentPartition` would exceed the target size, then start a
      // new coalesced partition.
      if (i > latestSplitPoint && coalescedSize + totalSizeOfCurrentPartition > targetSize) {
        partitionSpecs += CoalescedPartitionSpec(latestSplitPoint, i)
        latestSplitPoint = i
        // reset postShuffleInputSize.
        coalescedSize = totalSizeOfCurrentPartition
      } else {
        coalescedSize += totalSizeOfCurrentPartition
      }
      i += 1
    }
    partitionSpecs += CoalescedPartitionSpec(latestSplitPoint, numPartitions)

    partitionSpecs
  }
  1. totalPostShuffleInputSize 先计算出总的shuffle的数据大小
  2. maxTargetSize取max(totalPostShuffleInputSize/minNumPartitions,16)的最大值,minNumPartitions也就是spark.sql.adaptive.coalescePartitions.minPartitionNum的值
  3. targetSize取min(maxTargetSize,advisoryTargetSize),advisoryTargetSize也就是spark.sql.adaptive.advisoryPartitionSizeInBytes的值,所以说该值只是建议值,不一定是targetSize
  4. while循环就是取相邻的分区合并,对于每个task中的每个相邻分区合并,直到不大于targetSize

OptimizeSkewedJoin.optimizeSkewJoin分析(数据倾斜优化的核心代码)

见optimizeSkewJoin如示:

def optimizeSkewJoin(plan: SparkPlan): SparkPlan = plan.transformUp {
    case smj @ SortMergeJoinExec(_, _, joinType, _,
        s1 @ SortExec(_, _, ShuffleStage(left: ShuffleStageInfo), _),
        s2 @ SortExec(_, _, ShuffleStage(right: ShuffleStageInfo), _), _)
        if supportedJoinTypes.contains(joinType) =>
      assert(left.partitionsWithSizes.length == right.partitionsWithSizes.length)
      val numPartitions = left.partitionsWithSizes.length
      // Use the median size of the actual (coalesced) partition sizes to detect skewed partitions.
      val leftMedSize = medianSize(left.partitionsWithSizes.map(_._2))
      val rightMedSize = medianSize(right.partitionsWithSizes.map(_._2))
      logDebug(
        s"""
          |Optimizing skewed join.
          |Left side partitions size info:
          |${getSizeInfo(leftMedSize, left.partitionsWithSizes.map(_._2))}
          |Right side partitions size info:
          |${getSizeInfo(rightMedSize, right.partitionsWithSizes.map(_._2))}
        """.stripMargin)
      val canSplitLeft = canSplitLeftSide(joinType)
      val canSplitRight = canSplitRightSide(joinType)
      // We use the actual partition sizes (may be coalesced) to calculate target size, so that
      // the final data distribution is even (coalesced partitions + split partitions).
      val leftActualSizes = left.partitionsWithSizes.map(_._2)
      val rightActualSizes = right.partitionsWithSizes.map(_._2)
      val leftTargetSize = targetSize(leftActualSizes, leftMedSize)
      val rightTargetSize = targetSize(rightActualSizes, rightMedSize)

      val leftSidePartitions = mutable.ArrayBuffer.empty[ShufflePartitionSpec]
      val rightSidePartitions = mutable.ArrayBuffer.empty[ShufflePartitionSpec]
      val leftSkewDesc = new SkewDesc
      val rightSkewDesc = new SkewDesc
      for (partitionIndex <- 0 until numPartitions) {
        val isLeftSkew = isSkewed(leftActualSizes(partitionIndex), leftMedSize) && canSplitLeft
        val leftPartSpec = left.partitionsWithSizes(partitionIndex)._1
        val isLeftCoalesced = leftPartSpec.startReducerIndex + 1 < leftPartSpec.endReducerIndex

        val isRightSkew = isSkewed(rightActualSizes(partitionIndex), rightMedSize) && canSplitRight
        val rightPartSpec = right.partitionsWithSizes(partitionIndex)._1
        val isRightCoalesced = rightPartSpec.startReducerIndex + 1 < rightPartSpec.endReducerIndex

        // A skewed partition should never be coalesced, but skip it here just to be safe.
        val leftParts = if (isLeftSkew && !isLeftCoalesced) {
          val reducerId = leftPartSpec.startReducerIndex
          val skewSpecs = createSkewPartitionSpecs(
            left.mapStats.shuffleId, reducerId, leftTargetSize)
          if (skewSpecs.isDefined) {
            logDebug(s"Left side partition $partitionIndex is skewed, split it into " +
              s"${skewSpecs.get.length} parts.")
            leftSkewDesc.addPartitionSize(leftActualSizes(partitionIndex))
          }
          skewSpecs.getOrElse(Seq(leftPartSpec))
        } else {
          Seq(leftPartSpec)
        }

        // A skewed partition should never be coalesced, but skip it here just to be safe.
        val rightParts = if (isRightSkew && !isRightCoalesced) {
          val reducerId = rightPartSpec.startReducerIndex
          val skewSpecs = createSkewPartitionSpecs(
            right.mapStats.shuffleId, reducerId, rightTargetSize)
          if (skewSpecs.isDefined) {
            logDebug(s"Right side partition $partitionIndex is skewed, split it into " +
              s"${skewSpecs.get.length} parts.")
            rightSkewDesc.addPartitionSize(rightActualSizes(partitionIndex))
          }
          skewSpecs.getOrElse(Seq(rightPartSpec))
        } else {
          Seq(rightPartSpec)
        }

        for {
          leftSidePartition <- leftParts
          rightSidePartition <- rightParts
        } {
          leftSidePartitions += leftSidePartition
          rightSidePartitions += rightSidePartition
        }
      }

      logDebug("number of skewed partitions: " +
        s"left ${leftSkewDesc.numPartitions}, right ${rightSkewDesc.numPartitions}")
      if (leftSkewDesc.numPartitions > 0 || rightSkewDesc.numPartitions > 0) {
        val newLeft = CustomShuffleReaderExec(
          left.shuffleStage, leftSidePartitions, leftSkewDesc.toString)
        val newRight = CustomShuffleReaderExec(
          right.shuffleStage, rightSidePartitions, rightSkewDesc.toString)
        smj.copy(
          left = s1.copy(child = newLeft), right = s2.copy(child = newRight), isSkewJoin = true)
      } else {
        smj
      }
  }
  1. SortMergeJoinExec说明适用于sort merge join
  2. assert(left.partitionsWithSizes.length == right.partitionsWithSizes.length)保证进行join的两个task的分区数相等
  3. 分别计算进行join的task的分区中位数的大小leftMedSize和rightMedSize
  4. 分别计算进行join的task的分区的targetzise大小leftTargetSize和rightTargetSize
  5. 循环判断两个task的每个分区的是否存在倾斜,如果倾斜且满足没有进行过shuffle分区合并,则进行倾斜分区处理,否则不处理
  6. createSkewPartitionSpecs方法为:
    1.获取每个join的task的对应分区的数据大小
    2.根据targetSize分成多个slice
  7. 如果存在数据倾斜,则构造包装成CustomShuffleReaderExec,进行后续任务的运行,最最终调用ShuffledRowRDD的compute方法 匹配case PartialMapperPartitionSpec进行数据的读取,其中还会自动开启“spark.sql.adaptive.fetchShuffleBlocksInBatch”批量fetch减少io

OptimizeSkewedJoin/CoalesceShufflePartitions 在哪里被调用

如:AdaptiveSparkPlanExec

@transient private val queryStageOptimizerRules: Seq[Rule[SparkPlan]] = Seq(
    ReuseAdaptiveSubquery(conf, context.subqueryCache),
    CoalesceShufflePartitions(context.session),
    // The following two rules need to make use of 'CustomShuffleReaderExec.partitionSpecs'
    // added by `CoalesceShufflePartitions`. So they must be executed after it.
    OptimizeSkewedJoin(conf),
    OptimizeLocalShuffleReader(conf)
  )

可见在AdaptiveSparkPlanExec中被调用 ,且CoalesceShufflePartitions先于OptimizeSkewedJoin,
而AdaptiveSparkPlanExec在InsertAdaptiveSparkPlan中被调用
,而InsertAdaptiveSparkPlan在QueryExecution中被调用

而在InsertAdaptiveSparkPlan.shouldApplyAQE方法和supportAdaptive中我们看到

private def shouldApplyAQE(plan: SparkPlan, isSubquery: Boolean): Boolean = {
    conf.getConf(SQLConf.ADAPTIVE_EXECUTION_FORCE_APPLY) || isSubquery || {
      plan.find {
        case _: Exchange => true
        case p if !p.requiredChildDistribution.forall(_ == UnspecifiedDistribution) => true
        case p => p.expressions.exists(_.find {
          case _: SubqueryExpression => true
          case _ => false
        }.isDefined)
      }.isDefined
    }
  }

private def supportAdaptive(plan: SparkPlan): Boolean = {
    // TODO migrate dynamic-partition-pruning onto adaptive execution.
    sanityCheck(plan) &&
      !plan.logicalLink.exists(_.isStreaming) &&
      !plan.expressions.exists(_.find(_.isInstanceOf[DynamicPruningSubquery]).isDefined) &&
    plan.children.forall(supportAdaptive)
  }

如果不满足以上条件也是不会开启AQE的,如果要强制开启,也可以配置spark.sql.adaptive.forceApply 为true(文档中提示是内部配置)

注意:

在spark 3.0.1中已经废弃了如下的配置:

spark.sql.adaptive.skewedPartitionMaxSplits    
spark.sql.adaptive.skewedPartitionRowCountThreshold    
spark.sql.adaptive.skewedPartitionSizeThreshold   

本文部分参考:
https://mp.weixin.qq.com/s?__biz=MzA5MTc0NTMwNQ==&mid=2650718363&idx=1&sn=d20fffebafdd2bed6939eaeb39f5e6e3
https://mp.weixin.qq.com/s/RvFpXWpV8APcGTHhftS6NQ

sparksql参数调优 所以就会出现,即使你设置了spark.sql.files.ignoreMissingFiles的情况下,仍然报FileNotFoundException的情况,异常栈如下, 可以看到这里面走到了HadoopRDD,而且后面是org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrappe可见是查询一张hive表。上面的两个参数在分区表情况下是针对分区路径存在的情况下,分区路径下面的文件不存在或者损坏的处理。 阅读详情

相关推荐

spark sql query 剖析

生成 unresolved LogicalPlan // abstract class AbstractSqlParser` /** Creates LogicalPlan for a given SQL string. */ override def parsePlan(sqlText: String): LogicalPlan = parse(sqlText) { parser =&amp;amp;amp;amp;amp;amp;amp;amp;amp;gt...

zhixingheyi_tian的博客 654

Spark3 AQE之自动合并Shuffle partition源码解读

Branch:spark-3.0 有不对的地方欢迎各位大佬批评指正! 相关参数: spark.sql.adaptive.enabled AQE是否开启 spark.sql.adaptive.coalescePartitions.enabled 分区合并是否开启 spark.sql.adaptive.coalescePartitions.minPartitionNum 合并后最小的分区数,下文我们简称为minPartitionNum spark.sql.adap

weixin_45162906的博客 3226

ads传统功放设计笔记(3)

传统功放设计的流程一般包括以下这些步骤: 1、明确设计目标 2、选择功放管子 3、确定静态工作点 4、设计偏置电路 5、判断稳定性 6、负载与源牵引 7、输入输出阻抗匹配 8、原理图优化 9、大信号仿真 10、版图设计 11、版图联合仿真 12、pcb设计 第二节我们设计仿真了功放的原理图。之后我们需要对版图进行设计和仿真。 10、版图设计 首先将S参数控件都打叉,还要去掉电源、源负载阻抗、地、芯片、电容。在原先电容和芯片的位置加上gap,设置好长度和间隔。之后在电源引脚、输入输出引脚和隔离的两端加上端口。

qq_40995480的博客 6030

[SPARK][SQL] 面试问题之Spark AQE新特性

学习过 Shuffle 的工作原理之后,我们知道,每个 Map Task 都会输出以 data 为后缀的数据文件,还有以 index 为结尾的索引文件,这些文件统称为中间文件。如上图所示,N个task用于处理表A的偏斜分区0,每个task只读取表A的少数mapper的shuffle输出,并与表B的分区0进行join,将这N个task的结果合并得到最终的join结果. 为了实现这一点,我们更新了 shuffle read API 以允许仅从几个映射器而不是全部读取分区。了解一个功能,先来了解其面临的问题。

张伟的专栏 1259

全网最详细Seurat v5 流程复现(拿来即用)

本文是基于seurat v5版本的教程复现,主要依据来源于官网seurat v5建议大家有时间的话可以去官网进行详细阅读学习!本节内容在之前seurat v4的教程都很多都有提到,文章所用到的数据及代码已经给各位放在这里了,有需要自取!请点击这里哦提取码:bui1。

ZxVSaccount的博客 1万+

std::map看来可以被淘汰了?可以用vector来替代,而且更好

vector的优势在于内存连续,可以先std::sort,然后std::lower_bound,或者std::binary_search。而std::map,insert的时候,需要new出内存。

tumu_C的博客 1021

spark调优参数

CPU spark.cores.max 集群范围内满配CPU核数 spark.executor.cores 单个Executor内CPU核数 spark.task.cpus 单个任务消耗的CPU核数 spark.default.parallelism 未指定分区数时的默认并行度 s

epitomizelu的专栏 834

《QGIS快速入门与应用基础》239:指北针样式选择(预设/自定义)

系统解析指北针设计的核心规范与实操方法。文章首先强调行业规范红线,包括强制要求N字标注、箭头辨识度等关键要素,通过正误示例对比强化合规意识。其次详细讲解QGIS内置预设样式的分类应用,提供极简箭头、标准三角箭头等4类样式的选型对照表,并配图说明适用场景。最后介绍从基础微调到SVG导入的进阶自定义方法,帮助用户实现从合规到专业的样式设计。全文贯穿"合规优先、可读为本、美观为辅"原则,配有详细操作截图与规范对照表,适合从GIS新手到专业制图人员参考。

LuckyHanMo的博客 534

spark优化(二)--参数调优

spark参数

YYLong0的博客 1686

ZYNQ PL通过EMIO ETHE1外接PHY由GMII转RGMII

1,设置EMIO方式引出ETHE1 2,从GMII过度到RGMII GMII: 发送 gmii_tx_clk gmii_tx_d[7:0] gmii_tx_en gmii_tx_er 接收 gmii_rx_clk gmii_rx_d[7:0] gmii_rx_dv gmii_rx_er RGMII: 发送 tx_clk tx_d[3:0] tx_ctrl 接收 rx_clk rx_d[3:0] rx_ctrl 首先数据总线从时钟单边沿采样8bit转变为了双边沿采样4bit,从88E1

wangjie36的博客 1万+

自动驾驶算法-滤波器系列(三)——不同运动模型(CV、CA、CTRV、CTRA)的建模和推导

CV & CA & CTRV & CTRA1. CV模型2. CA模型3. CTRV模型4. CTRA模型 1. CV模型 2. CA模型 3. CTRV模型 4. CTRA模型

weixin_41480034的博客 2万+

Spark3.0核心调优参数小总结

点击上方蓝色字体,选择“设为星标”回复”面试“获取更多惊喜本文已经加入「大数据成神之路PDF版」中提供下载。你可以关注公众号,后台回复:「PDF」即可获取。更多PDF下载可以参考:《重磅...

微信搜:import_bigdata,大数据领域硬核原创作者 3073

全网最详细的 Ubuntu 18.04 安装Livox mid-360驱动,测试 fast_lio2

Livox mid-360需要使用Livox-SDK2,而非Livox-SDK,以及对应的。并需要修改FAST_LIO中部分代码。Faster—LIO;Faster-LIO:快速激光IMU里程计。Faster-LIO:快速激光IMU里程计PDF。大部分Lidar SLAM都是基于loam进行改造的,基本上是通过线特征edge和面特征(面点)分割进行配准得到里程计的,而fast-lio2是通过一个迭代扩展卡尔曼滤波器去优化帧内位姿,然后通过输入点云与地图进行配准得到里程计,精度和鲁棒性都不错。

努力努力努力 6241

Spark AQE源码探索

Spark AQE特性介绍和AQE源码的学习和分享

1863

JS逆向-7881游戏平台Lb-Sign参数和Lb-Timestamp参数

本文是该专栏的第79篇,后面会持续分享python爬虫干货知识。本文以7881游戏平台为例,通过JS逆向获取它的lb-sign参数和lb-timestamp参数生成规律。针对lb-sign和lb-timestamp两个参数具体的“逆向”思路逻辑,笔者将在本文进行详细说明,并且将在正文结合“完整代码”来详细介绍。废话不多说,具体细节部分以及详细思路逻辑,跟着笔者直接往下看正文部分。(附带完整代码)

Python进阶专栏《爬虫实战进阶》,《Pyppeteer从入门到精通》原创作者 1223

android 蓝牙通讯demo

android 下蓝牙通讯demo,实现了发现蓝牙,使能蓝牙,通过蓝牙发送消息。

Spark3自适应查询计划(Adaptive Query Execution,AQE)

动态合并shuffle分区(Dynamically coalescing shuffle partitions) 动态调整join策略(Dynamically switching join strategies) 动态优化数据倾斜join(Dynamically optimizing skew joins) 参数:spark.sql.adaptive.enabled 默认关闭,开启此参数后上述三种策略才会执行 1、动态优化数据倾斜(Dynamically optimizing skew joins) sp

weixin_45162906的博客 2637

NTscan扫描工具

ntscan是一个扫描器,可以扫描没有密码和弱密码的共享,先到配置文件中填写猜测的用户名和密码,扫描过程中右侧出现的是扫描日志。

SparkSQL的自适应执行-Adaptive Execution

点击上方蓝色字体,选择“设为星标”回复”资源“获取更多资源大数据技术与架构点击右侧关注,大数据开发领域最强公众号!暴走大数据点击右侧关注,暴走大数据!Spark SQL 自适应执行优化引...

微信搜:import_bigdata,大数据领域硬核原创作者 2247

温度PID控制PLC程序

温度PID稳定控制程序,适用于欧姆龙PLC,对其他PLC也具有很大参考价值

VB摄像头监控

该VB模块可对摄像头捕获的画面进行录像等,可在创建的窗口进行控制

spark-sql优化简述

对于一条SQL语句中可能多次使用到的表,可以对其进行缓存,使用SQLContext.cacheTable(TableName)或者DataFrame.cache即可,SparkSQL会用内存列存储的格式进行表的缓存,然后SparkSQL就可以仅仅扫描需要使用的列,并且自动优化压缩,来最小化内存的使用和GC的开销,SQLContext.uncacheTable(tableName)可以将表从缓存中移除,使用SQLContext.setConf()设置,可以通过。数据偏斜会严重降低联接查询的性能。

天翼云开发者社区 356

Spark AQE 配置和源码说明

网摘:AQE(Adaptive Query Execution)自适应查询,是Spark 3.0开始增加的一种机制,可以根据 Shuffle Map阶段的统计信息,基于地调整和修正的,来完成对原始查询语句的优化。

lilyjoke的专栏 988

Spark3-AQE-数据倾斜Join优化

Adaptive Query Exection(自适应查询计划)简称AQE,在最早在spark 1.6版本就已经有了AQE;到了spark 2.x版本,intel大数据团队进行了相应的原型开发和实践;到了spark 3.0时代,AQE终于面向用户可以使用了 注:以下代码分析基于Spark3.0.1版本 1 Join的自适应数据倾斜处理 代码位于sql.core模块的org.apache.spark.sql.execution.adaptive.OptimizeSkewedJoin 主要原理就是基.

z7one 1714

24讲spark AQE的三个特性怎么才能用好?

spark aqe

u011624157的专栏 1451

spark的AQE机制

AQE优化规则和策略。

qq_42438830的博客 1255

sparksql优化_Spark原理Spark SQL 自适应执行优化引擎

在本篇文章中,笔者将给大家带来 Spark SQL 中关于自适应执行引擎(Spark Adaptive Execution)的内容。在之前的文章中,笔者介绍过 Flink SQL,目前 Flink 社区在积极地更新迭代 Flink SQL 功能和优化性能,尤其 Flink 1.10.0 版本的发布,在增强流式 SQL 处理能力的同时也具备了成熟的批处理能力。但是在 SQL 功能完整性和生...

weixin_39605835的博客 511

Spark3 新特性之AQE

Spark3 新特性之AQE

Shyllin的博客 5130

sparksql源码系列 | 一文搞懂Distribution源码体系(spark3.2)

这篇文章主要介绍sparksql中Distribution的源码体系,Distribution是我们理解Physical Plan、executed Plan、shuffle、SparkSQL的AQE机制等的一个比较基础的知识点

xiaoluobutou的专栏 672

Spark AQE

Spark 3.0 添加了自适应查询执行(AQE)、动态分区剪裁(DPP)、扩展的 Join Hints。

记录点点滴滴 1266
上一篇: Data Lakehouse (湖仓一体) 到底是什么
下一篇: 【go系列3】go 的包管理工具go modules的使用
鸿乃江边鸟
博客等级 码龄6年 3746粉丝 235原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值