JavaAPI操作HDFS文件

最新推荐文章于 2025-03-19 12:47:15 发布

孙彬不像个好程序员

最新推荐文章于 2025-03-19 12:47:15 发布

阅读量356

点赞数

分类专栏：大数据文章标签： hadoop hdfs

本文链接：https://blog.youkuaiyun.com/sunbin11220904/article/details/107041224

版权

大数据专栏收录该内容

23 篇文章

订阅专栏

这篇博客介绍了如何在Java环境下通过API进行HDFS文件系统的操作，包括创建 Maven 工程，配置 pom.xml，设置 Hadoop 配置，创建目录，上传、下载文件以及删除文件等步骤。通过HTTP访问地址可以验证操作是否成功。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

创建maven工程
new project -Maven quickstart
配置prm.xml

<dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.12</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.6.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>2.6.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>2.6.0</version>
    </dependency>
    <dependency>
      <groupId>commons-logging</groupId>
      <artifactId>commons-logging</artifactId>
      <version>1.2</version>
    </dependency>
    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>1.2.17</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.12</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-api</artifactId>
      <version>RELEASE</version>
      <scope>compile</scope>
    </dependency>
  </dependencies>

在$HADOOP_HOME/etc/hadoop/hdfs-site.xml中添加

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

1.创建目录

@Test
    public void testMkdirs() throws URISyntaxException, IOException, InterruptedException {
        //创建配置
        Configuration conf = new Configuration();
        //2获取文件系统
        FileSystem fs = FileSystem.get(new URI("hdfs://192.168.106.107:9000"), conf, "root");
        //3调用API
        fs.mkdirs(new Path("/hdfs/shell"));
        fs.close();
    }

上传本地一个文件

 @Test
    public void testCopyFromLocalFile() throws URISyntaxException, IOException, InterruptedException {
        Configuration conf = new Configuration();
        //2获取文件系统
        FileSystem fs = FileSystem.get(new URI("hdfs://192.168.106.107:9000"), conf, "root");
        fs.copyFromLocalFile(new Path("a\\aa.txt"), new Path("/user/a.txt"));
        fs.close();
    }

下载到本地一个文件

 @Test
    public void testCopyToLocalFile() throws URISyntaxException,IOException,InterruptedException{
        //1.创建配置文件
        Configuration conf=new Configuration();
        //2.获取文件系统
        FileSystem fs=FileSystem.get(new URI("hdfs://192.168.106.107:9000"),conf,"root");
        //3.调用API操作
        fs.copyToLocalFile(new Path("/hdfs"),new Path("E:\\"));
        //4.关闭资源
        fs.close();
    }

删除文件

 @Test
    public void testdelete() throws URISyntaxException, IOException, InterruptedException {
        Configuration conf = new Configuration();
        //2获取文件系统
        FileSystem fs = FileSystem.get(new URI("hdfs://192.168.106.107:9000"), conf, "root");
        fs.delete(new Path("/hdfs/aa.txt"),true);
        fs.close();
    }