Springboot连接HDFS

HDFS入门：连接与文件夹创建详解

最新推荐文章于 2024-03-29 01:55:15 发布

原创最新推荐文章于 2024-03-29 01:55:15 发布 · 1.2k 阅读

5 ·

CC 4.0 BY-SA版权

文章标签：

#hdfs #hadoop #java

Java/Spingboot开发专栏收录该内容

3 篇文章

订阅专栏

本文介绍了如何通过HDFS URI和配置创建连接，并演示了如何在Hadoop文件系统中创建目录。关键步骤包括创建URI对象、配置连接、指定用户及使用`FileSystem.get()`方法。遇到`winutils`问题时，提供了下载和环境变量设置的解决方案。

连接HDFS并创建文件夹

1.创建URI对象，参数为HDFS对应ip端口

URI uri = new URI("hdfs://192.168.109.135:9000");

2. 创建连接配置

Configuration configuration = new Configuration();
String user="hadoop";
FileSystem fs = FileSystem.get(uri, configuration,user);

3.代码

pom依赖导入：

<!-- hadoop 依赖 -->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-hdfs</artifactId>
      <version>2.7.3</version>
      <exclusions>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
        <exclusion>
          <groupId>log4j</groupId>
          <artifactId>log4j</artifactId>
        </exclusion>
        <exclusion>
          <groupId>javax.servlet</groupId>
          <artifactId>servlet-api</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.7.3</version>
      <exclusions>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
        <exclusion>
          <groupId>log4j</groupId>
          <artifactId>log4j</artifactId>
        </exclusion>
        <exclusion>
          <groupId>javax.servlet</groupId>
          <artifactId>servlet-api</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>2.7.3</version>
      <exclusions>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
        <exclusion>
          <groupId>log4j</groupId>
          <artifactId>log4j</artifactId>
        </exclusion>
        <exclusion>
          <groupId>javax.servlet</groupId>
          <artifactId>servlet-api</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>1.2.16</version>
      <scope>compile</scope>
    </dependency>

方法示例：

public String hello() throws URISyntaxException, IOException, InterruptedException {

//        URI uri = new URI("hdfs://192.168.109.135:9000");
//        Configuration configuration = new Configuration();
//        String user="hadoop";
//        FileSystem fs = FileSystem.get(uri, configuration,user);
//        Path path = new Path("/dir/test");
//        fs.mkdirs(path);
//        fs.close();

        //1.获取文件系统
        Configuration configuration = new Configuration();
        String user="hadoop";
        FileSystem fs = FileSystem.get(new URI(nameNode), configuration, user);
        //2.执行操作 创建hdfs文件夹
        Path path = new Path(filePath);
        if (!fs.exists(path)) {
            fs.mkdirs(path);
        }
        //关闭资源
        fs.close();
        System.out.println("结束！");
        return "hello";
    }

可能出现报错：Could not locate executable null \bin\winutils.exe in the hadoop binaries

解决方案：winutils下载地址：winutils

添加到系统环境变量HADOOP_HOME重启系统