误用聪明,何若一生守拙
滥交朋友,不如终日读书
相关连接
HDFS相关知识
Hadoop集群连接
WordCount程序示例
HDFS Java API
代码下载
MyHadoop.java下载 提取码z458
具体介绍
注意:在使用Eclipse或者IntelliJ IDEA成功连接Hadoop集群后,方可进行如下操作
- 本测试类类名为MyHadoop,其包含FileSystem类的属性fs和Configuration类的属性conf
- 需要定义HDFSUtil()方法
- 需要在主函数中加入
System.setProperty(“HADOOP_USER_NAME”, ”root”);
,以解决org.apache.hadoop.security.AccessControlException:Permission denied: user=...
错误
package neu.software;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import java.io.IOException;
public class MyHadoop {
private FileSystem fs;
private Configuration conf;
public static void main(String args[]) throws IOException {
System.setProperty("HADOOP_USER_NAME","root");
MyHadoop myHadoop = new MyHadoop();
myHadoop.HDFSUtil();
}
public void HDFSUtil() throws IOException {
conf = new org.apache.hadoop.conf.Configuration();
fs = FileSystem.get(conf);
}
}
1. 在HDFS中创建目录 /data/test
方法定义
public boolean mkdir(String path) throws IOException {
Path srcPath = new Path(path);
return fs.mkdirs(srcPath);
}
方法测试
public static void main(String args[]) throws IOException {
System.setProperty("HADOOP_USER_NAME","root");
MyHadoop myHadoop = new MyHadoop();
myHadoop.HDFSUtil();
myHadoop.mkdir("/data/test");
}
结果验证(XShell命令窗口)
[root@master ~]# hadoop fs -ls /data
Found 1 items
drwxr-xr-x - root supergroup 0 2019-10-15 11:14 /data/test
2. 将本地文件夹mytest通过Java API上传到HDFS的 /data/test 目录中
方法定义
public void put(String src, String dst, boolean delSrc, boolean overwritted) throws IOException {
Path srcPath = new Path(src);
Path dstPath = new Path(dst);
//调用文件系统的文件复制函数,delSrc参数指是否删除原文件,true为删除
fs.copyFromLocalFile(delSrc, overwritted, srcPath, dstPath);
}
方法测试
public static void main(String args[]) throws IOException {
System.setProperty("HADOOP_USER_NAME","root");
MyHadoop myHadoop = new MyHadoop();
myHadoop.HDFSUtil();
//myHadoop.mkdir("/data/test"); //不能二次执行
myHadoop.put("C:\\Users\\Lenovo\\Desktop\\localfile\\mytest", "/data/test/",