hadoop use at first

最新推荐文章于 2024-06-27 11:58:56 发布

原创最新推荐文章于 2024-06-27 11:58:56 发布 · 447 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #jar #path #input #shell

Linux/Unix 专栏收录该内容

26 篇文章

订阅专栏

本文详细介绍了如何使用Hadoop核心版本0.20.203.0.jar执行WordCount任务，包括编译Java代码、打包成jar文件、修改core-site.xml配置以改变HDFS目录、格式化NameNode、运行jar文件并准备输入数据。

1. compile the WordCount.java

javac -classpath ./hadoop-core-0.20.203.0.jar -d ../hdp_test -s ../hdp_test ../hdp_test/WordCo
unt.java -Xlint:deprecation

2. jar it
jar -cvf ~/hdp_test/WordCount.jar .

3. change core-site.xml
add the dfs property,
because the format meta data of hdfs was save in the /tmp in default,
so it 's clear after reboot the machine
change the dfs proterty block:

    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/chenglun/tmp</value>
        <description>A base for other temporary directories</description>
    </property>

4.
hadoop namenode format

5. run it
5.0 prepare the input data
hadoop jar WordCount.jar org.myorg.WordCount /home/chenglun/input /home/chenglun/output

Appendix: hdfs shell:

hadoop fs -ls [path]
hadoop fs -cat [path]
hadoop fs -mkdir [dir]
hadoop fs -copyFromLocal local URI