spark的Datasets和DataFrames及其区别

spark的Datasets和DataFrames及其区别

A Dataset is a distributed collection of data. Dataset is a new interface added in Spark 1.6 that provides the benefits of RDDs (strong typing, ability to use powerful lambda functions) with the benefits of Spark SQL’s optimized execution engine. A Dataset can be constructed from JVM objects and then manipulated using functional transformations (map, flatMap, filter, etc.). The Dataset API is available in Scala and Java. Python does not have the support for the Dataset API. But due to Python’s dynamic nature, many of the benefits of the Dataset API are already available (i.e. you can access the field of a row by name naturally row.columnName). The case for R is similar.

Datasets是分布式数据集合。Dataset是Spark 1.6中添加的一个新接口，它提供了RDD的优势（强类型，使用强大的lambda函数的能力）以及Spark SQL优化执行引擎的优点。数据集可以从JVM对象中构造，然后使用转换函数（map，flatMap，filter等）进行操作。Datasets API在Scala和Java中可用。Python没有对Dataset API的支持。但由于Python的动态特性，Dataset API的许多好处已经可用（比如您可以通过名称自然地访问行的字段row.columnName）。 R的情况类似。

A DataFrame is a Dataset organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R/Python, but with richer optimizations under the hood. DataFrames can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing RDDs. The DataFrame API is available in Scala, Java, Python, and R. In Scala and Java, a DataFrame is represented by a Dataset of Rows. In the Scala API, DataFrame is simply a type alias of Dataset[Row]. While, in Java API, users need to use Dataset<Row> to represent a DataFrame.

DataFrames是将Datasets一列一列地通过列名组合的集合。它在概念上等同于关系数据库中的表或R / Python中的data frame，但在相关计算引擎下具有更丰富的优化。DataFrame可以从多种来源构建，例如：结构化数据文件，Hive中的表，外部数据库或现有RDD。DataFrame API在Scala，Java，Python和R中可用。在Scala和Java中，DataFrame由行数据集表示。在Scala API中，DataFrame只是Dataset[Row]的类型别名。而在Java API中，用户需要使用数据集Dataset<Row>来表示DataFrame。

参考文献：Spark SQL, DataFrames and Datasets Guide