读HDFS书笔记---5.2 文件读操作与输入流(5.2.1)

最新推荐文章于 2022-07-14 17:00:41 发布

乘风如水

最新推荐文章于 2022-07-14 17:00:41 发布

阅读量682

点赞数

CC 4.0 BY-SA版权

分类专栏： hadoop

本文链接：https://blog.youkuaiyun.com/weixin_39935887/article/details/84328640

5.2.1 打开文件

用户读取一个HDFS文件时，首先会调用DistributedFileSystem.open()方法打开这个文件，并获取文件对应的FSDataInputStream输入流，然后在这个FSDataInputStream对象上调用read方法读取数据。整个流程图如下：

读文件流程图

上图中的USERCODE为用户代码，调用DistributedFileSystem中的open函数，代码如下：

@Override
  public FSDataInputStream open(Path f, final int bufferSize)
      throws IOException {
    statistics.incrementReadOps(1);
    Path absF = fixRelativePart(f);
    return new FileSystemLinkResolver<FSDataInputStream>() {
      @Override
      public FSDataInputStream doCall(final Path p)
          throws IOException, UnresolvedLinkException {
        final DFSInputStream dfsis =
          dfs.open(getPathName(p), bufferSize, verifyChecksum);
        return dfs.createWrappedInputStream(dfsis);
      }
      @Override
      public FSDataInputStream next(final FileSystem fs, final Path p)
          throws IOException {
        return fs.open(p, bufferSize);
      }
    }.resolve(this, absF);
  }

resolve函数代码如下：

/**
   * Attempt calling overridden {@link #doCall(Path)} method with
   * specified {@link FileSystem} and {@link Path}. If the call fails with an
   * UnresolvedLinkException, it will try to resolve the path and retry the call
   * by calling {@link #next(FileSystem, Path)}.
   * @param filesys FileSystem with which to try call
   * @param path Path with which to try call
   * @return Generic type determined by implementation
   * @throws IOException
   */
  public T resolve(final FileSystem filesys, final Path path)
      throws IOException {
    int count = 0;
    T in = null;
    Path p = path;
    // Assumes path belongs to this FileSystem.
    // Callers validate this by passing paths through FileSystem#checkPath
    FileSystem fs = filesys;
    for (boolean isLink = true; isLink;) {
      try {
        in = doCall(p);
        isLink = false;
      } catch (UnresolvedLinkException e) {
        if (!filesys.resolveSymlinks) {
          throw new IOException("Path " + path + " contains a symlink"
              + " and symlink resolution is disabled ("
              + CommonConfigurationKeys.FS_CLIENT_RESOLVE_REMOTE_SYMLINKS_KEY
              + ").", e);
        }
        if (!FileSystem.areSymlinksEnabled()) {
          throw new IOException("Symlink resolution is disabled in" +
              " this version of Hadoop.");
        }
        if (count++ > FsConstants.MAX_PATH_LINKS) {
          throw new IOException("Possible cyclic loop while " +
                                "following symbolic link " + path);
        }
        // Resolve the first unresolved path component
        p = FSLinkResolver.qualifySymlinkTarget(fs.getUri(), p,
            fil