ETL is dead, long live streams

Reorganized according to Infoq talk "ETL is dead, long live streams"

 // 根据 Infoq 的演讲 《ETL 结束了、流处理万岁》重新整理

 

 

 

Data and data system changed a lot over the past decade. 

 // 过去的十年里，数据和数据库发生了重大的变化。

In the past, database and datawarehouse are main location for our data.

 // 在过去，数据库和数据仓库是我们数据的重要位置。

And most of the database and datawarehouse are relational. 

 // 大部分数据库和数据仓库都是关系型的数据库。

 

The recent data trend includes:

 // 未来的趋势如下

 

1. single server databases are replaced by a myriad of distributed data platform that operated at a company-wide scale. 

// 单服务器数据库已被在公司范围内运行的无数分布式数据平台所取代。

A medium to large size company could have more than one data centre in different locations.

// 一个大中型公司可能在不同的地点有多个数据中心。

 

2. There are many more non-transactional data: logs,images,sensors etc. 

// 还有很多非事务性的数据：日志、图像、传感器等

No-sql database appeared and data blending work more handly.

// NoSQL 数据库出现了，这些数据混合的工作更加地顺手

 

3. Stream data is increasingly ubiquitous,and faster processing is needed.

// 数据的流式处理变得更普遍，也需要更快的处理

 

Therefore, the tranditional way of Extract, Transform and Load becomes a giant mess, the shortcomings are as below:

 // 因此，传统的提取、转换和加载方式变得一团糟，缺点如下：

 

1. it needs a global schema

// 它需要一个全局的约束

 

2. data cleaning and curation is manual and fundamentally error-prone

// 数据清理和管理是手动的，而且基本上容易出错

 

3. operational cost of ETL is high and resource intensive.

// ETL的运行成本高、资源密集

 

4. ETL tools were normally built to narrowly focus on connecting to databases and datawarehouse in a batch fashion.

// ETL工具的构建通常只关注以批处理方式连接到数据库和数据仓库

 

Comparably, Streaming platform will have its shiny points:

// 相比之下，流媒体平台将有其闪光点

 

1. It is able to process high volume and high diversity data

// 它能够处理大容量、高分集的数据

 

2. It is able to provide real-time processed data from ground level

// 能够提供地面实时处理数据

 

3. It enables forward-compatible data architecture.

// 它支持向前兼容的数据体系结构