spark datasourceV1和v2

原创已于 2023-02-27 19:25:57 修改 · 730 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #大数据 #分布式

于 2023-02-27 14:49:59 首次发布

spark源码同时被 2 个专栏收录

12 篇文章

订阅专栏

大数据

5 篇文章

订阅专栏

ApacheSpark的DataSourceV2因DataFrame/SQLContext的依赖、缺乏物理存储信息传播、扩展性和操作下推限制以及写入接口无事务支持等问题而诞生。DataSourceV2旨在改进这些问题，提供高性能的列读接口和更好的优化。文章讨论了如何通过DataSourceV2解决小文件问题，特别是在AWSS3上的应用。

datasourceV2

一文理解 Apache Spark DataSource V2 诞生背景及入门实战
https://zhuanlan.zhihu.com/p/83006243

2.3 Data source API v2

https://issues.apache.org/jira/browse/SPARK-15689

Because of the above limitations/issues, the built-in data source implementations (like parquet, json, etc.) inside Spark SQL are not using this public Data Source API. Instead, they use an internal/non-public interface.

https://issues.apache.org/jira/browse/SPARK-13664

动机

Since its input arguments include DataFrame/SQLContext, the data source API compatibility depends on the upper level API.
依赖 DataFrame/SQLContext, 上层api

The physical storage information (e.g., partitioning and sorting) is not propagated from the data sources, and thus, not used in the Spark optimizer.
分区排序没有传播给spark

Extensibility is not good and operator push-down capabilities are limited.
扩展的下推能力被限制

Lacking columnar read interface for high performance.
列的读接口

The write interface is so general without transaction supports.
没有事务支持

spark FileFormatWriter

org.apache.spark.sql.execution.datasources.FileFormatWriter

// We should first sort by partition columns, then bucket id, and finally sorting columns.
val requiredOrdering = partitionColumns ++ bucketIdExpression ++ sortColumns

对写入数据的要求,可以影响logical plan