利用Shell命令与HDFS进行交互
1.目录操作
hdfs dfs –mkdir –p /user/hadoop
该命令中表示在HDFS中创建一个“/user/hadoop”目录,“–mkdir”是创建目录的操作,“-p”表示如果是多级目录,则父目录和子目录一起创建,这里“/user/hadoop”就是一个多级目录,因此必须使用参数“-p”,否则会出错。“/user/hadoop”目录就成为hadoop用户对应的用户目录,可以使用如下命令显示HDFS中与当前用户hadoop对应的用户目录下的内容:
hdfs dfs –ls .
该命令中,“-ls”表示列出HDFS某个目录下的所有内容,“.”表示HDFS中的当前用户目录,也就是“/user/hadoop”目录,因此,上面的命令和下面的命令是等价的:
./bin/hdfs dfs –ls /user/hadoop
./bin/hdfs dfs –ls
./bin/hdfs dfs –mkdir input
./bin/hdfs dfs –mkdir /input
./bin/hdfs dfs –rm –r /input
上面命令中,“-r”参数表示如果删除“/input”目录及其子目录下的所有内容,如果要删除的一个目录包含了子目录,则必须使用“-r”参数,否则会执行失败。
2.文件操作
“/home/hadoop/”目录下创建一个文件myLocalFile.txt,里面可以随意输入一些单词,比如,输入如下三行:
Hadoop
Spark
XMU DBLAB
然后,可以使用如下命令把本地文件系统的“/home/hadoop/myLocalFile.txt”上传到HDFS中的当前用户目录的input目录下,也就是上传到HDFS的“/user/hadoop/input/”目录下:
./bin/hdfs dfs -put /home/hadoop/myLocalFile.txt input
./bin/hdfs dfs –ls input
./bin/hdfs dfs –cat input/myLocalFile.txt
./bin/hdfs dfs -get input/myLocalFile.txt /home/hadoop/下载
如何把文件从HDFS中的一个目录拷贝到HDFS中的另外一个目录。比如,如果要把HDFS的“/input/myLocalFile.txt”文件,拷贝到HDFS的另外一个目录“/input”中(注意,这个input目录位于HDFS根目录下),可以使用如下命令:
./bin/hdfs dfs -cp input/myLocalFile.txt /input
利用Java API与HDFS进行交互
- 创建项目
- 为项目添加需要用到的JAR包
- 编写Java应用程序代码
- 编译程序
- 应用程序的部署
Hadoop不同的文件系统之间通过调用Java API进行交互,上面介绍的Shell命令,本质上就是Java API的应用。下面提供了Hadoop官方的Hadoop API文档,想要深入学习Hadoop,可以访问如下网站,查看各个API的功能。
Hadoop API文档 http://hadoop.apache.org/docs/stable/api/
创建项目
在hadoop目录下新建文件夹workplace/HDFSExample
为项目添加需要用到的JAR包
为了编写一个能够与HDFS交互的Java应用程序,一般需要向Java工程中添加以下JAR包:(在文件夹workplace/HDFSExample中):
(1)”/usr/local/hadoop/share/hadoop/common”目录下的hadoop-common-2.7.1.jar和haoop-nfs-2.7.1.jar;
(2)/usr/local/hadoop/share/hadoop/common/lib”目录下的所有JAR包;
(3)“/usr/local/hadoop/share/hadoop/hdfs”目录下的haoop-hdfs-2.7.1.jar和haoop-hdfs-nfs-2.7.1.jar;
(4)“/usr/local/hadoop/share/hadoop/hdfs/lib”目录下的所有JAR包。
添加到路径(profile)
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:/app/lib/hadoop-2.7.3/share/hadoop/common/lib:/app/lib/hadoop-2.7.3/share/hadoop/hdfs/lib
[xxxycentos@xxxycentos7 HDFSExample]$ source /etc/profile
编写Java应用程序代码
创建了一个名为“HDFSFileIfExist.java”的源代码文件,请在该文件中输入以下代码:
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class HDFSFileIfExist {
public static void main(String[] args){
try{
String fileName = "test";
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://localhost:9000");
conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");
FileSystem fs = FileSystem.get(conf);
if(fs.exists(new Path(fileName))){
System.out.println("文件存在");
}else{
System.out.println("文件不存在");
}
}catch (Exception e){
e.printStackTrace();
}
}
}
编译程序
[xxxycentos@xxxycentos7 HDFSExample]$ javac -classpath hadoop-common-2.7.3.jar:hadoop-nfs-2.7.3.jar:hadoop-hdfs-2.7.3.jar:hadoop-hdfs-nfs-2.7.3.jar HDFSFileIfExist.java
应用程序的部署
[xxxycentos@xxxycentos7 HDFSExample]$ echo Main-class: HDFSFileIfExist >mainfest.txt
[xxxycentos@xxxycentos7 HDFSExample]$ cat mainfest.txt
Main-class: HDFSFileIfExist
[xxxycentos@xxxycentos7 HDFSExample]$ jar cvfm HDFS.jar mainfest.txt HDFSFileIfExist.class
added manifest
adding: HDFSFileIfExist.class(in = 1152) (out= 683)(deflated 40%)
[xxxycentos@xxxycentos7 HDFSExample]$ hadoop jar HDFS.jar
19/04/17 21:38:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
file not exists
附录
收录安装过程中遇到的一些问题
$ javac -classpath hadoop-common-2.7.3.jar:hadoop-annotations-2.7.3.jar HDFSFileIfExist.java
[xxxycentos@xxxycentos7 HDFSExample]$ java HDFSFileIfExist
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at HDFSFileIfExist.main(HDFSFileIfExist.java:8)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
JVM在编译的时候能找到调用方法或静态变量所在的类,但在运行的时候找不到此类而引发的错误。直接使用java命令是报错(如上)但打包后用命令 hadoop jar xx.jar 运行成功
java -classpath /workplace/Test
Error: Could not find or load main class workspace.HDFSExample.Test
解决 修改/etc/profile
export CLASSPATH=…
解决了
练习
写入文件
新建Chapter3.java文件
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;
public class Chapter3{
public static void main(String[] args) {
try {
Configuration conf = new Configuration();
conf.set("fs.defaultFS","hdfs://192.168.1.1xx:9000");
conf.set("fs.hdfs.impl","org.apache.hadoop.hdfs.DistributedFileSystem");
FileSystem fs = FileSystem.get(conf);
byte[] buff = "Hello world".getBytes();
String filename = "testWrite";
FSDataOutputStream os = fs.create(new Path(filename));
os.write(buff,0,buff.length);
System.out.println("Create:"+ filename);
os.close();
fs.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
编译
[xxxycentos@xxxycentos7 HDFSExample]$ javac -classpath hadoop-common-2.7.3.jar:hadoop-nfs-2.7.3.jar:hadoop-hdfs-2.7.3.jar:hadoop-hdfs-nfs-2.7.3.jar Chapter3.java
打包
[xxxycentos@xxxycentos7 HDFSExample]$ echo Main-class: Chapter3 >mainfest.txt
[xxxycentos@xxxycentos7 HDFSExample]$ cat mainfest.txt
Main-class: Chapter3
[xxxycentos@xxxycentos7 HDFSExample]$ jar cvfm Chapter3.jar mainfest.txt Chapter3.class
added manifest
adding: Chapter3.class(in = 1469) (out= 843)(deflated 42%)
运行
[xxxycentos@xxxycentos7 HDFSExample]$ hadoop jar Chapter3.jar
19/04/18 10:24:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Create:testWrite
[xxxycentos@xxxycentos7 HDFSExample]$
检查写入结果
[xxxycentos@xxxycentos7 HDFSExample]$ hadoop fs -cat /user/xxxycentos/testWrite
Hello world
读入文件
[xxxycentos@xxxycentos7 HDFSExample]$ javac -classpath hadoop-common-2.7.3.jar:hadoop-nfs-2.7.3.jar:hadoop-hdfs-2.7.3.jar:hadoop-hdfs-nfs-2.7.3.jar Chapter3Read.java
[xxxycentos@xxxycentos7 HDFSExample]$ echo Main-class: Chapter3Read >mainfest.txt
[xxxycentos@xxxycentos7 HDFSExample]$ jar cvfm Chapter3Read.jar mainfest.txt Chapter3Read.class
added manifest
adding: Chapter3Read.class(in = 1338) (out= 769)(deflated 42%)
[xxxycentos@xxxycentos7 HDFSExample]$ hadoop jar Chapter3Read.jar
Hello world