Baidu's C++ Extention on Hadoop

最新推荐文章于 2025-08-21 23:36:32 发布

ustcdd

最新推荐文章于 2025-08-21 23:36:32 发布

阅读量1.3k

点赞数

文章标签： hadoop c++ function date table extension

Hadoop Map/Reduce

Hadoop C++ Extention

Created: 07/Dec/09 08:09 AM Updated: Thursday 07:38 PM

Component/s:

task

Affects Version/s:

0.20.1

Fix Version/s:

None

Time Tracking:

Not Specified

Environment:

hadoop linux

Hadoop Flags:	Incompatible change
Tags:	PIPES C++
Labels:

Description

« Hide

Hadoop C++ extension is an internal project in baidu, We start it for these reasons:
1 To provide C++ API. We mostly use Streaming before, and we also try to use PIPES, but we do not find PIPES is more efficient than Streaming. So we

think a new C++ extention is needed for us.
2 Even using PIPES or Streaming, it is hard to control memory of hadoop map/reduce Child JVM.
3 It costs so much to read/write/sort TB/PB data by Java. When using PIPES or Streaming, pipe or socket is not efficient to carry so huge data.

What we want to do:
1 We do not use map/reduce Child JVM to do any data processing, which just prepares environment, starts C++ mapper, tells mapper which split it should deal with, and reads report from mapper until that finished. The mapper will read record, ivoke user defined map, to do partition, write spill, combine and merge into file.out. We think these operations can be done by C++ code.
2 Reducer is similar to mapper, it was started after sort finished, it read from sorted files, ivoke user difined reduce, and write to user defined record writer.
3 We also intend to rewrite shuffle and sort with C++, for efficience and memory control.
at first, 1 and 2, then 3.

What's the difference with PIPES:
1 Yes, We will reuse most PIPES code.
2 And, We should do it more completely, nothing changed in scheduling and management, but everything in execution.