HDFS源码剖析-HDFS的鉴权

最新推荐文章于 2025-07-08 00:00:00 发布

原创最新推荐文章于 2025-07-08 00:00:00 发布 · 1.3k 阅读

2 ·

CC 4.0 BY-SA版权

HDFS 专栏收录该内容

4 篇文章

订阅专栏

1 HDFS权限控制和ACL

官网文档：http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html

中文翻译：https://blog.youkuaiyun.com/shubingzhuoxue/article/details/50072271

acl涉及的参数：

dfs.permissions.enabled

dfs.web.ugi

dfs.permissions.superusergroup

fs.permissions.umask-mode

dfs.cluster.administrators

dfs.namenode.acls.enabled

2 HDFS鉴权的流程

以HDFS Client为例，来分析下HDFS鉴权的过程

在hdfs client分析：hdfs dfs -ls一文中，我们已经知道client在创建出FileSystem对象后，就会去调用fs.listStatus方法去列取目录信息。listStatus底层是通过RPC调用到NameNode的listStatus方法，那么ls的鉴权肯定就是在NameNode的listStatus方法中进行的了。

进入DistributedFileSystem.listStatus方法

//DistributedFileSystem.java
@Override
  public FileStatus[] listStatus(Path p) throws IOException {
    Path absF = fixRelativePart(p);
    return new FileSystemLinkResolver<FileStatus[]>() {
      @Override
      public FileStatus[] doCall(final Path p)
          throws IOException, UnresolvedLinkException {
        return listStatusInternal(p);
      }
      @Override
      public FileStatus[] next(final FileSystem fs, final Path p)
          throws IOException {
        return fs.listStatus(p);
      }
    }.resolve(this, absF);
  }

再进入listStatusInternal(p)

//DistributedFileSystem.java
private FileStatus[] listStatusInternal(Path p) throws IOException {
    String src = getPathName(p);

    // fetch the first batch of entries in the directory
    DirectoryListing thisListing = dfs.listPaths(
        src, HdfsFileStatus.EMPTY_NAME);

    if (thisListing == null) { // the directory does not exist
      throw new FileNotFoundException("File " + p + " does not exist.");
    }
    ...
}

再进入dfs.listPaths(src, HdfsFileStatus.EMPTY_NAME)

//DFSClient.java 
public DirectoryListing listPaths(String src,  byte[] startAfter)
    throws IOException {
    return listPaths(src, startAfter, false);
  }

再进入listPaths(src, startAfter, false)

//DFSClient.java
public DirectoryListing listPaths(String src,  byte[] startAfter,
      boolean needLocation) throws IOException {
    checkOpen();
    TraceScope scope = getPathTraceScope("listPaths", src);
    try {
      return namenode.getListing(src, startAfter, needLocation);
    } catch(RemoteException re) {
      throw re.unwrapRemoteException(AccessControlException.class,
                                     FileNotFoundException.class,
                                     UnresolvedPathException.class);
    } finally {
      scope.close();
    }
  }

namenode.getListing最终是调用到了NameNodeRPCServer端了

//NameNodeRpcServer.java
public DirectoryListing getListing(String src, byte[] startAfter,
      boolean needLocation) throws IOException {
    checkNNStartup();
    DirectoryListing files = namesystem.getListing(
        src, startAfter, needLocation);
    if (files != null) {
      metrics.incrGetListingOps();
      metrics.incrFilesInGetListingOps(files.getPartialListing().length);
    }
    return files;
  }

进入namesystem.getListing

DirectoryListing getListing(String src, byte[] startAfter,
      boolean needLocation) 
      throws IOException {
    checkOperation(OperationCategory.READ);
    DirectoryListing dl = null;
    readLock();
    try {
      checkOperation(NameNode.OperationCategory.READ);
      dl = FSDirStatAndListingOp.getListingInt(dir, src, startAfter,
          needLocation);
    } catch (AccessControlException e) {
      logAuditEvent(false, "listStatus", src);
      throw e;
    } finally {
      readUnlock();
    }
    logAuditEvent(true, "listStatus", src);
    return dl;
  }

终于看到AccessControlException出现了，看来getListingInt

static DirectoryListing getListingInt(FSDirectory fsd, final String srcArg,
      byte[] startAfter, boolean needLocation) throws IOException {
    //创建出FSPermissionChecker对象，此对象包含client端的UGI信息
    FSPermissionChecker pc = fsd.getPermissionChecker();
    byte[][] pathComponents = FSDirectory
        .getPathComponentsForReservedPath(srcArg);
    final String startAfterString = new String(startAfter, Charsets.UTF_8);
    final String src = fsd.resolvePath(pc, srcArg, pathComponents);
    final INodesInPath iip = fsd.getINodesInPath(src, true);

    // Get file name when startAfter is an INodePath
    if (FSDirectory.isReservedName(startAfterString)) {
      byte[][] startAfterComponents = FSDirectory
          .getPathComponentsForReservedPath(startAfterString);
      try {
        String tmp = FSDirectory.resolvePath(src, startAfterComponents, fsd);
        byte[][] regularPath = INode.getPathComponents(tmp);
        startAfter = regularPath[regularPath.length - 1];
      } catch (IOException e) {
        // Possibly the inode is deleted
        throw new DirectoryListingStartAfterNotFoundException(
            "Can't find startAfter " + startAfterString);
      }
    }

    boolean isSuperUser = true;
    /**
     * 开启鉴权，dfs.permissions.enabled
     * 进行path的权限验证
     */
    if (fsd.isPermissionEnabled()) {
      if (iip.getLastINode() != null && iip.getLastINode().isDirectory()) {
        fsd.checkPathAccess(pc, iip, FsAction.READ_EXECUTE);
      } else {
        fsd.checkTraverse(pc, iip);
      }
      isSuperUser = pc.isSuperUser();
    }
    return getListing(fsd, iip, src, startAfter, needLocation, isSuperUser);
  }

先来看下fsd.getPermissionChecker()

FSPermissionChecker getPermissionChecker()
  throws AccessControlException {
  try {
    return getPermissionChecker(fsOwnerShortUserName, supergroup,
        NameNode.getRemoteUser());
  } catch (IOException e) {
    throw new AccessControlException(e);
  }
}

public static UserGroupInformation getRemoteUser() throws IOException {
  UserGroupInformation ugi = Server.getRemoteUser();
  return (ugi != null) ? ugi : UserGroupInformation.getCurrentUser();
}

/** Returns the RPC remote user when invoked inside an RPC.  Note this
 *  may be different than the current user if called within another doAs
 *  @return connection's UGI or null if not an RPC
 */
public static UserGroupInformation getRemoteUser() {
  Call call = CurCall.get();
  return (call != null && call.connection != null) ? call.connection.user
      : null;
}

看到remoteUser信息是从RPC连接中获取到的。

接下来看鉴权的过程fsd.isPermissionEnabled()

在NameNodeServer端获取到了remoteUGI后，就是和path的ugi信息做匹配了，很简单，就不一步步的分析了。

3 HDFS认证