莱布尼兹

i choice to learn the sources from the flow-work of FS to MR,so the steps are these:a.FS(IO)b.MRc.IPC outline of API:  simple flow of hadoop:

open the bin/hadoop file,you will see the there is a config file to load:  either libexec/hadoop-config.sh or bin/hadoop-config.sh and the previor is loaded if exists,else the load the later. ...

 出于工作需要及版本更新带来的变动，现在开始再次进入源码空间－hadoop-1.0.1 这次阅读的目的有这几个：－比较全面的阅读整体代码，清楚大体的工作流程，各部件的连接与交互 ；－common的改动及主要职责；－config/shell 的启动流程；－hdfs具体的设计及实现；－mapreduce的詳細设计及实现；－ipc詳細实现－others ...

在map reduce框架中，除了常用的分布式计算外，排序也算是比较重要的一环了。这形如sql查询中的排序数据一样重要。  一。无排序当书写code 时，如果指定了mapred.reduce.tasks=0(same effect as setNumReduceTasks)。这样便达到目的。产生的效果当然是只有一个part file，而且其中的entries是unorder....

when run in cluster(pseudo),it is difficult to debug the code.i have tried to debug use some tricks to do it:a use the jdpa debug platform.add this options to hadoop opts after start and before ...

here is my summry during reading the sources,consider to  my ability and  the complexity of hadoop,and i have not read all the sources,there will be some inlogical statements in them,so if you find a ...

TODO

TODO

it is used to schedule jobs and tasks also.there are certain ones : JobQueueTaskScheduler(by default,and is FIFO algorithm); FairScheduler; CapacityScheduler;... here is the simple flow of...

"split" which is a logical concept relatives to a "block" whis is real store unit.when a client submit a job to JT,it will compute the splits by file,than the TT will generate InputSplit to map ...

as writing a file to hdfs,the client get a DistributedSystem to communicate with Namenode.and the DS will create a DFSClient to create DFSInputstream which is encasluted to FSDataInputStream. off ...

data structure transfrom file to hdfs from client

 when executes start-all.sh (or start-dfs.sh,start-mapred.sh) ,some config files listed below will be loaded:   libexec/hadoop-config.sh ( use  bin/hadoop-config.sh  if without previor)   confi...