zhixingheyi_tian的博客

spark build

Spark Plan

Spark Parquet Decode

Parquet Meta

checkAnswer系列- pivot with null and aggregate type not supported by PivotFirst returns correct result *** FAILED ***null 由 null 变成 0，引入了 多组 c2r，r2c，问题出在ArrowWritableColumnVector读写出了问题。

allocateColumns /** * Allocates columns to store elements of each field of the schema on heap. * Capacity is the initial capacity of the vector and it will grow as necessary. Capacity is * in number of elements, not number of bytes. */ publi

以parquet 来举例基本的接口实现DataSourceV2 => Table => ScanBuilder => Scan => PartitionReaderFactory= (VectorizedParquetRecordReader )ParquetPartitionReaderFactoryParquetPartitionReaderFactory 包装了 VectorizedParquetRecordReader...

ListenerBus 是一个 trait，可以接受事件，并将事件提交到对应事件的监听器private[spark] trait ListenerBus[L <: AnyRef, E] extends Logging {

spark 递归删除目录的方法，会尝试两种做法若第一种deleteRecursivelyUsingUnixNative不成功，会立即尝试第二种// org.apache.spark.network.util.JavaUtils.java/** * Delete a file or directory and its contents recursively. * Don't fo...

InternalRow — Abstract Binary Row FormatInternalRow is also called Catalyst row or Spark SQL row.abstract class InternalRow extends SpecializedGetters with Serializable {}UnsafeRowUnsafeRow is a...

VectorizedParquetRecordReader.java{ /** * The number of rows that have been returned. */ private long rowsReturned;/** * The number of rows that have been reading, including the current...

BinaryExecNodeBinary physical operator with two child left and right physical operatorsLeafExecNodeLeaf physical operator with no childrenBy default, the set of all attributes that are produce...