Hadoop 作业的几个参数

最新推荐文章于 2025-09-09 09:49:36 发布

转载最新推荐文章于 2025-09-09 09:49:36 发布 · 714 阅读

文章标签：

#hadoop

Hadoop 专栏收录该内容

5 篇文章

订阅专栏

本文介绍如何在Hadoop中设置MapReduce任务的Mapper和Reducer数量，包括Hadoop 1.x和Hadoop 2.x (YARN) 的配置方法，并提供本地运行MapReduce任务的配置示例。

Number of mappers and reducers can be set like (5 mappers, 2 reducers):

-D mapred.map.tasks=5 -D mapred.reduce.tasks=2

in the command line.

In the code, one can configure JobConf variables.

job.setNumMapTasks(5); // 5 mappers
job.setNumReduceTasks(2); // 2 reducers

Note that on Hadoop 2 (YARN), the mapred.map.tasks and mapred.reduce.tasks aredeprecated and are replaced by other variables:

mapred.map.tasks     -->    mapreduce.job.maps
mapred.reduce.tasks  -->    mapreduce.job.reduces

============================================================= 

如果想在本地使用mapreduce，那么把mapreduce.framework.name和mapreduce.jobtracker.address都设置成local