Create Hadoop project in IDEA Intellij and Commit project to Hadoop platform

最新推荐文章于 2026-01-05 18:04:27 发布

原创最新推荐文章于 2026-01-05 18:04:27 发布 · 817 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #IDEA Intellij

Hadoop 专栏收录该内容

3 篇文章

订阅专栏

本文详细介绍了如何在本地环境中安装配置Java、Hadoop及IDEA Intellij，并通过具体步骤指导创建并运行MapReduce项目，包括设置运行配置、添加Hadoop依赖库以及构建项目JAR文件。

1.First ,we should set up Java ,Hadoop,and IDEA Intellij-2016.3 . I have Installed Java-1.8 , Hadoop-2.7.3 and IDEA Intellij-2016.3

2.Start-up HDFS ,YARN

3.We create new Maven Project, then click next

4.Input GroupId and ArtifactId,Then selectnext

5.Input Project name and location and click finish

6.Set Run Configuration with the optionRun ->Edit Configurations,SetProgram argumentswithinput file pathand output file path(It can be in HDFS).Then click OK.

7.Select File -> Project Structure to add hadoop jars,Which are common , yarn , mapreduce , hdfs coming from ~hadoop home/share/hadoop . Click OK.

8.Then you can run your MapReduce program in local.

9.If you want commit project to Hadoop platform, you must setartifacts to createProject Jar. In this step, you should selectFile -> Project Structureto add artifacts.Click" + " and write Name .Then Click"+ " , which is underOutput Layout. Finally modificationMain Class, which is your project' s main class.ClickOK.

10.Build artifact. Select Build ->Build Artifacts ...->MatrixMultiply (which is created by yourself in previous step).

11.Commit project to Hadoop platform. You should add code in your project as :

conf.set("mapred.jar", "./out/artifacts/MatrixMultiply/MatrixMultiply.jar");
conf.set("mapreduce.framework.name", "yarn");

12.Look at the result:According to the log , we can get that the job have committed to hadoop platform.

13.We can view the job' s information at http://localhost:8088.