Databricks Delta资料及使用TIPS

原创

已于 2022-05-30 10:57:59 修改 · 837 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#spark #databricks #azure #python

于 2022-05-26 17:53:33 首次发布

本文介绍了Delta Lake的特性，如ACID事务、数据可靠性等，并提供了在Azure Databricks中创建、修改Delta Table的方法。内容涵盖SQL与API创建Delta Table，读取与写入数据的技巧，强调API创建支持更多参数，以及如何处理表定义更改时的数据一致性问题。

Reference: Creating Delta Lake Tables in Azure Databricks

Delta lake is an open-source data format that provides ACID transactions, data reliability, query performance, data caching and indexing, and many other benefits. Delta lake can be thought of as an extension of existing data lakes and can be configured per the data requirements. Azure Databricks has a delta engine as one of the core components that facilitates delta lake format for data engineering and performance. Delta lake format is used to create modern data lake or lakehouse architectures. It is also used to build a combined streaming and batch architecture popularly known as lambda architecture.

Delta lake overview from aliyun: Delta Lake概述 - 开源大数据平台E-MapReduce - 阿里云

Streaming example cases: 【详谈 Delta Lake 】系列技术专题之 Streaming（流式计算）-阿里云开发者社区

Formal document: Welcome to the Delta Lake documentation — Delta Lake Documentation