1. HDFS CLIENT WRITE调试篇

HDFS写流程解析
原创 已于 2022-08-16 14:34:31 修改 · 1.5k 阅读 · 3 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#hadoop #hdfs

于 2019-12-19 16:48:05 首次发布

文章目录

知乎 https://zhuanlan.zhihu.com/p/66051354

使用测试集群使用FsShell断点

    @Test
    public void testWrite() throws Exception{
        FsShell shell = new FsShell();
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS","hdfs://hadoop1:9000");
        conf.setQuietMode(false);
        shell.setConf(conf);
        System.setProperty("HADOOP_USER_NAME", "hadoop");
        System.setProperty("HADOOP_USER_PASSWORD", "xxx");
        String local = "/Users/didi/CodeFile/proTest/hadoop-test/src/main/resources/test/core-site.xml";
        String[] args = {"-put", local, "/test/write/1219-1.xml"};   // 普通目录ok
        int res = 1;
        try {
            res = ToolRunner.run(shell, args);
        } finally {
            shell.close();
        }
        System.exit(res);
    }

在org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.<init>构造器中加断点。

DFSOutputStream

"main@1" prio=5 tid=0x1 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.<init>(DFSOutputStream.java:276)
	  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.<init>(DFSOutputStream.java:227)
	  at org.apache.hadoop.hdfs.DFSOutputStream.<init>(DFSOutputStream.java:1626)
	  at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1679)
	  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1692)
	  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1627)
	  at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:503)
	  at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:499)
	  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:514)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:442)
	  at org.apache.hadoop.fs.FilterFileSystem.create(FilterFileSystem.java:179)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:979)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:960)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:857)
	  at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.create(CommandWithDestination.java:488)
	  at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:465)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:391)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:328)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:263)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:248)
	  at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
	  at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:291)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:243)
	  at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:273)
	  at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:257)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:220)
	  at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:267)
	  at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:203)
	  at org.apache.hadoop.fs.shell.Command.run(Command.java:167)
	  at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	  at cn.whbing.hadoop.ReadWriteTest.testWrite(ReadWriteTest.java:132)

创建了临时文件如：
/test/write/1219-5.xml.COPYING

DataStreamer --> 是一个守护线程类。

1.主线程创建DFSOutputStream

static DFSOutputStream newStreamForCreate(DFSClient dfsClient, String src,
      FsPermission masked, EnumSet<CreateFlag> flag, boolean createParent,
      short replication, long blockSize, Progressable progress, int buffersize,
      DataChecksum checksum, String[] favoredNodes)

在这里插入图片描述
其中 FSOutputSummer 继承 OutputStream，主要是添加了一个计算checksum的功能。DFSOutputStream继承FSOutputSummer。

主要的三个类：

  • DFSOutputStream
  • DataStreamer
  • ResponseProcessor

2

要把读写过程细节搞明白前，你必须知道block、packet与chunk

  • block
    block是最大的一个单位，它是最终存储于DataNode上的数据粒度，由dfs.block.size参数决定。文件上传前需要分块，这个块就是block，一般为128MB，当然你可以去改，不顾不推荐。因为块太小：寻址时间占比过高。块太大：Map任务数太少，作业执行速度变慢。它是最大的一个单位。
  • packet
    packet是第二大的单位，它是client端向DataNode，或DataNode的PipLine之间传数据的基本单位，默认64KB。
  • chunk
    chunk是最小的单位，它是client向DataNode，或DataNode的PipLine之间进行数据校验的基本单位，默认512Byte，因为用作校验，故每个chunk需要带有4Byte的校验位。所以实际每个chunk写入packet的大小为516Byte。由此可见真实数据与校验值数据的比值约为128 : 1。

例如，在client端向DataNode传数据的时候，HDFSOutputStream会有一个chunk buff，写满一个chunk后，会计算校验和并写入当前的chunk。之后再把带有校验和的chunk写入packet，当一个packet写满后，packet会进入dataQueue队列，其他的DataNode就是从这个dataQueue获取client端上传的数据并存储的。同时一个DataNode成功存储一个packet后之后会返回一个ack packet，放入ack Queue中。

一个比较好的图如下：
在这里插入图片描述
对他的解释：
写过程中的三层buffer
写过程中会以chunk、packet及packet queue三个粒度做三层缓存；

  • 首先，当数据流入DFSOutputStream时，DFSOutputStream内会有一个chunk大小的buf，当数据写满这个buf（或遇到强制flush），会计算checksum值，然后填塞进packet；
  • 当一个chunk填塞进入packet后，仍然不会立即发送，而是累积到一个packet填满后，将这个packet放入dataqueue队列；
  • 进入dataQueue队列的packet会被另一线程(DataSreamer)按序取出发送到datanode，发送出去后，会将该packet放入ackQueue中；只有当ResponseProcessor线程收到下游节点传过来的ack消息后，才会将数据包从ackQueue中移除。（注：生产者消费者模型，阻塞生产者的条件是dataqueue与ackqueue之和超过一个block的packet上限）

DFSOutStream中使用 DFSPacket 来包装数据。
DFSPacket:

 /**
   * buf is pointed into like follows:
   *  (C is checksum data, D is payload data)
   *
   * [_________CCCCCCCCC________________DDDDDDDDDDDDDDDD___]
   *           ^        ^               ^               ^
   *           |        checksumPos     dataStart       dataPos
   *           checksumStart
   *
   * Right before sending, we move the checksum data to immediately precede
   * the actual data, and then insert the header into the buffer immediately
   * preceding the checksum data, so we make sure to keep enough space in
   * front of the checksum data to support the largest conceivable header.
   */

在这里插入图片描述

DataStreamer线程

在这里插入图片描述
DataStreamer将数据包通过管道发送给datanode，并在当前数据块写满时，向NN申请分配新的数据块，然后更新管道。DataStream定义了管道的状态信息：

	// 数据块数据流管道中的datanode
	private volatile DatanodeInfo[] nodes = null; // list of targets for current block
	// 在dn上保存这个块的存储类型
    private volatile StorageType[] storageTypes = null;
    // 在dn上保存这个块的存储
    private volatile String[] storageIDs = null;
    // 数据块对应的数据流管道状态
    private BlockConstructionStage stage;  // block construction stage
...
    private void setPipeline(DatanodeInfo[] nodes, StorageType[] storageTypes,
        String[] storageIDs) {
      this.nodes = nodes;
      this.storageTypes = storageTypes;
      this.storageIDs = storageIDs;
    }

1.对于新写文件操作，DataStreamer会调用nextBlockOutputStream()方法向NN申请新的数据块，然后构建这个新的数据块的数据流管道。

          // get new block from namenode.
          if (stage == BlockConstructionStage.PIPELINE_SETUP_CREATE) {
            if(DFSClient.LOG.isDebugEnabled()) {
              DFSClient.LOG.debug("Allocating new block");
            }
            setPipeline(nextBlockOutputStream()); //新写数据块时，先调用next
            initDataStreaming(); // 再传输数据
          } else if (stage == BlockConstructionStage.PIPELINE_SETUP_APPEND) {
            if(DFSClient.LOG.isDebugEnabled()) {
              DFSClient.LOG.debug("Append to block " + block);
            }
            setupPipelineForAppendOrRecovery();
            initDataStreaming();
          }

2.对于append，会返回最后一个数据块的位置。根据最后一个数据块的位置信息初始化数据流管道。
3.成功构建管道后，会调用initDataStreaming将管道状态更改为DATA_STREAMING。然后就通过数据流管道发送数据。
在这里插入图片描述

PIPELINE整理

1. [HDFS写文件过程分析]

HDFS写文件过程分析

2. [Hadoop Pipeline详解]

Hadoop Pipeline详解

3. [再议HDFS写流程之pipeline]

再议HDFS写流程之pipeline

4.[HDFS源码分析（5）：datanode数据块的读与写]

HDFS源码分析（5）：datanode数据块的读与写

5.[Hhadoop-2.7.0中HDFS写文件源码分析（二）：客户端实现之DFSPacket]

Hhadoop-2.7.0中HDFS写文件源码分析（二）：客户端实现之DFSPacket

在这里插入图片描述

开启debug模式流程分析

hadoop fs -put xxx

...
19/12/23 09:11:46 DEBUG hdfs.DFSClient: /test/write/1.txt._COPYING_: masked=rw-r--r--
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #3
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #3
19/12/23 09:11:46 DEBUG ipc.ProtobufRpcEngine: Call: create took 13ms
19/12/23 09:11:46 DEBUG hdfs.DFSClient: computePacketChunkSize: src=/test/write/1.txt._COPYING_, chunkSize=516, chunksPerPacket=126, packetSize=65016
19/12/23 09:11:46 DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_-78804544_1] with renew id 1 started
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #4
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #4
19/12/23 09:11:46 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
19/12/23 09:11:46 DEBUG hdfs.DFSClient: DFSClient writeChunk allocating new packet seqno=0, src=/test/write/1.txt._COPYING_, packetSize=65016, chunksPerPacket=126, bytesCurBlock=0
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Queued packet 0
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Queued packet 1
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Allocating new block
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Waiting for ack for: 1
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #5
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #5
19/12/23 09:11:46 DEBUG ipc.ProtobufRpcEngine: Call: addBlock took 29ms
19/12/23 09:11:46 DEBUG hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[10.179.72.122:50010,DS-508839e9-efb9-417c-8aa9-fd7b5b78b688,DISK]
19/12/23 09:11:46 DEBUG hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[10.179.72.42:50010,DS-239d3012-2d81-4f66-9b08-77e9d21397d1,DISK]
19/12/23 09:11:46 DEBUG hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[10.179.16.227:50010,DS-286d8579-fe5f-484d-bb05-ae41787734e3,DISK]
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Connecting to datanode 10.179.72.122:50010
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Send buf size 131072
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #6
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #6
19/12/23 09:11:46 DEBUG ipc.ProtobufRpcEngine: Call: getServerDefaults took 2ms
19/12/23 09:11:46 DEBUG sasl.SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.179.72.122, datanodeId = DatanodeInfoWithStorage[10.179.72.122:50010,DS-508839e9-efb9-417c-8aa9-fd7b5b78b688,DISK]
19/12/23 09:11:47 DEBUG hdfs.DFSClient: DataStreamer block BP-857948820-10.179.72.122-1560825265775:blk_1073747684_6863 sending packet packet seqno: 0 offsetInBlock: 0 lastPacketInBlock: false lastByteOffsetInBlock: 1114
19/12/23 09:11:47 DEBUG hdfs.DFSClient: DFSClient seqno: 0 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 545916 flag: 0 flag: 0 flag: 0
19/12/23 09:11:47 DEBUG hdfs.DFSClient: DataStreamer block BP-857948820-10.179.72.122-1560825265775:blk_1073747684_6863 sending packet packet seqno: 1 offsetInBlock: 1114 lastPacketInBlock: true lastByteOffsetInBlock: 1114
19/12/23 09:11:47 DEBUG hdfs.DFSClient: DFSClient seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111705 flag: 0 flag: 0 flag: 0
19/12/23 09:11:47 DEBUG hdfs.DFSClient: Closing old block BP-857948820-10.179.72.122-1560825265775:blk_1073747684_6863
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #7
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #7
19/12/23 09:11:47 DEBUG ipc.ProtobufRpcEngine: Call: complete took 11ms
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #8
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #8
19/12/23 09:11:47 DEBUG ipc.ProtobufRpcEngine: Call: rename took 13ms
19/12/23 09:11:47 DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@d737b89
19/12/23 09:11:47 DEBUG ipc.Client: removing client from cache: org.apache.hadoop.ipc.Client@d737b89
19/12/23 09:11:47 DEBUG ipc.Client: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@d737b89
19/12/23 09:11:47 DEBUG ipc.Client: Stopping client
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop: closed
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop: stopped, remaining connections 0

注：上述debug打印了关键信息，下文会顺序分析；DFSClient的log不一定在该类中，可能是其他类中的调用。

核心流程

1.fs.create获得FSDataOutputStream，实质是DFSOutputStream。
中间借助DFSClient.create()方法进入：

public DFSOutputStream create(String src, 
                             FsPermission permission,
                             EnumSet<CreateFlag> flag, 
                             boolean createParent,
                             short replication,
                             long blockSize,
                             Progressable progress,
                             int buffersize,
                             ChecksumOpt checksumOpt,
                             InetSocketAddress[] favoredNodes) throws IOException {
    checkOpen();
    if (permission == null) {
      permission = FsPermission.getFileDefault();
    }
    FsPermission masked = permission.applyUMask(dfsClientConf.uMask);
    if(LOG.isDebugEnabled()) {
      LOG.debug(src + ": masked=" + masked); // 1 该出的debug，见上述调试
    }
    final DFSOutputStream result = DFSOutputStream.newStreamForCreate(this,
        src, masked, flag, createParent, replication, blockSize, progress,
        buffersize, dfsClientConf.createChecksum(checksumOpt),
        getFavoredNodesStr(favoredNodes));  // 2 获得Stream
    beginFileLease(result.getFileId(), result);  // 3 租约有关
    return result;
  }

注：该debug见上述打印，先建一个临时文件；并获得管道stream进行复杂操作；同时启动租约。

在这里插入图片描述
2.关键入口方法：DFSoutputStream.newStreamForCreate，通过构造器得到输出流。并同时启动输出流主线程。

static DFSOutputStream newStreamForCreate(DFSClient dfsClient, String src,
      FsPermission masked, EnumSet<CreateFlag> flag, boolean createParent,
      short replication, long blockSize, Progressable progress, int buffersize,
      DataChecksum checksum, String[] favoredNodes) throws IOException {
    TraceScope scope =
        dfsClient.getPathTraceScope("newStreamForCreate", src);
    try {
      HdfsFileStatus stat = null;

      // Retry the create if we get a RetryStartFileException up to a maximum
      // number of times
      boolean shouldRetry = true;
      int retryCount = CREATE_RETRY_COUNT;
      while (shouldRetry) {
        shouldRetry = false;
        try {
          stat = dfsClient.namenode.create(src, masked, dfsClient.clientName,
              new EnumSetWritable<CreateFlag>(flag), createParent, replication,
              blockSize, SUPPORTED_CRYPTO_VERSIONS);
          break;
        } catch (RemoteException re) {
          IOException e = re.unwrapRemoteException(
              AccessControlException.class,
              DSQuotaExceededException.class,
              FileAlreadyExistsException.class,
              FileNotFoundException.class,
              ParentNotDirectoryException.class,
              NSQuotaExceededException.class,
              RetryStartFileException.class,
              SafeModeException.class,
              UnresolvedPathException.class,
              SnapshotAccessControlException.class,
              UnknownCryptoProtocolVersionException.class);
          if (e instanceof RetryStartFileException) {
            if (retryCount > 0) {
              shouldRetry = true;
              retryCount--;
            } else {
              throw new IOException("Too many retries because of encryption" +
                  " zone operations", e);
            }
          } else {
            throw e;
          }
        }
      }
      Preconditions.checkNotNull(stat, "HdfsFileStatus should not be null!");
      final DFSOutputStream out = new DFSOutputStream(dfsClient, src, stat,
          flag, progress, checksum, favoredNodes);   // 获得输出流
      out.start();  // 启动输出流注线程
      return out;
    } finally {
      scope.close();
    }
  }

上述创建输出流分为三步：
（1）向NN发送请求，调用NN的create()方法（这块暂时不研究）；
（2）通过构造器构造出输出流；
（3）启动输出流线程

后续流程均在out.start()方法中启动。即DFSOutputStream.run()方法。
根据调用栈及debuglog、代码可知，在DFSOutputStream的构造器中，调用了computePacketChunkSize。
在这里插入图片描述
上述数值固定。即每个packet有126个chunk，每个chunk大小（data 512 + checksum 4=516），header占了33个字节。(计算的值是126.9那么这里到底是用126还是127呢？)
每个packet传输的data+checksum的字节为：126*516=65016字节。
每个packet传输的有效字节为：126*512=64512字节。

在这里插入图片描述
文本如下：

19/12/23 10:35:58 DEBUG hdfs.DFSClient: computePacketChunkSize: src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, chunkSize=516, chunksPerPacket=126, packetSize=65016
19/12/23 10:35:58 DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_-1334506554_1] with renew id 1 started
19/12/23 10:35:58 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #4
19/12/23 10:35:58 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #4
19/12/23 10:35:58 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk allocating new packet seqno=0, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, packetSize=65016, chunksPerPacket=126, bytesCurBlock=0
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk packet full seqno=0, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, bytesCurBlock=64512, blockSize=134217728, appendChunk=false
19/12/23 10:35:58 DEBUG hdfs.DFSClient: Queued packet 0
19/12/23 10:35:58 DEBUG hdfs.DFSClient: computePacketChunkSize: src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, chunkSize=516, chunksPerPacket=126, packetSize=65016
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk allocating new packet seqno=1, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, packetSize=65016, chunksPerPacket=126, bytesCurBlock=64512
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk packet full seqno=1, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, bytesCurBlock=129024, blockSize=134217728, appendChunk=false
19/12/23 10:35:58 DEBUG hdfs.DFSClient: Queued packet 1
19/12/23 10:35:58 DEBUG hdfs.DFSClient: Allocating new block
19/12/23 10:35:58 DEBUG hdfs.DFSClient: computePacketChunkSize: src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, chunkSize=516, chunksPerPacket=126, packetSize=65016
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk allocating new packet seqno=2, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, packetSize=65016, chunksPerPacket=126, bytesCurBlock=129024
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk packet full seqno=2, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, bytesCurBlock=193536, blockSize=134217728, appendChunk=false

每次传64512个有效字节，然后再新分配一个packet。
继续看分配packet的代码，在writeChunkImpl后续分析。

在这里插入图片描述
那么谁在调用看下调用栈：

java.lang.Thread.State: RUNNABLE
	  at org.apache.hadoop.hdfs.DFSOutputStream.computePacketChunkSize(DFSOutputStream.java:1741)
	  at org.apache.hadoop.hdfs.DFSOutputStream.writeChunkImpl(DFSOutputStream.java:1876)
	  - locked <0xaeb> (a org.apache.hadoop.hdfs.DFSOutputStream)
	  at org.apache.hadoop.hdfs.DFSOutputStream.writeChunk(DFSOutputStream.java:1813)
	  at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:206)
	  at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:124)
	  at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:110)
	  at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
	  at java.io.DataOutputStream.write(DataOutputStream.java:107)
	  - locked <0xb29> (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream)
	  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:87)
	  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
	  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
	  at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:466)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:391)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:328)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:263)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:248)
	  at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
	  at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:291)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:243)
	  at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:273)
	  at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:257)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:220)
	  at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:267)
	  at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:203)
	  at org.apache.hadoop.fs.shell.Command.run(Command.java:167)
	  at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	  at cn.whbing.hadoop.ReadWriteTest.testWrite(ReadWriteTest.java:133)

可以看到，调用FSOutputSummer.write，然后循环调用write1()。为什么这个线程栈和上述的不一样呢？我们将两个都列出来：

  java.lang.Thread.State: RUNNABLE
	  at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1679)
	  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1692)
	  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1627)
	  at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:503)
	  at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:499)
	  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:514)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:442)
	  at org.apache.hadoop.fs.FilterFileSystem.create(FilterFileSystem.java:179)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:979)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:960)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:857)
	  at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.create(CommandWithDestination.java:488)
	  at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:465)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:391)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:328)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:263)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:248)
	  at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
	  at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:291)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:243)
	  at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:273)
	  at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:257)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:220)
	  at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:267)
	  at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:203)
	  at org.apache.hadoop.fs.shell.Command.run(Command.java:167)
	  at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	  at cn.whbing.hadoop.ReadWriteTest.testWrite(ReadWriteTest.java:133)

可以看到在 at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:465) 时，走向了不同的方法。

    void writeStreamToFile(InputStream in, PathData target,
                           boolean lazyPersist) throws IOException {
      FSDataOutputStream out = null;
      try {
        out = create(target, lazyPersist); // 1
        IOUtils.copyBytes(in, out, getConf(), true); // 2
      } finally {
        IOUtils.closeStream(out); // just in case copyBytes didn't
      }
    }

分为两步，先创建一个流，再复制到目标流。

write中初次长度131072，write1每次写4608：
在这里插入图片描述

private synchronized void writeChunkImpl(byte[] b, int offset, int len,
          byte[] checksum, int ckoff, int cklen) throws IOException {
    dfsClient.checkOpen();
    checkClosed();

    if (len > bytesPerChecksum) {
      throw new IOException("writeChunk() buffer size is " + len +
                            " is larger than supported  bytesPerChecksum " +
                            bytesPerChecksum);
    }
    if (cklen != 0 && cklen != getChecksumSize()) {
      throw new IOException("writeChunk() checksum size is supposed to be " +
                            getChecksumSize() + " but found to be " + cklen);
    }

    // 1 分配一个新的packet
    if (currentPacket == null) {
      currentPacket = createPacket(packetSize, chunksPerPacket, 
          bytesCurBlock, currentSeqno++, false);
      if (DFSClient.LOG.isDebugEnabled()) {
        DFSClient.LOG.debug("DFSClient writeChunk allocating new packet seqno=" + 
            currentPacket.getSeqno() +
            ", src=" + src +
            ", packetSize=" + packetSize +
            ", chunksPerPacket=" + chunksPerPacket +
            ", bytesCurBlock=" + bytesCurBlock);
      }
    }

    currentPacket.writeChecksum(checksum, ckoff, cklen);
    currentPacket.writeData(b, offset, len);
    currentPacket.incNumChunks();
    bytesCurBlock += len;

    // If packet is full, enqueue it for transmission
    //
    if (currentPacket.getNumChunks() == currentPacket.getMaxChunks() ||
        bytesCurBlock == blockSize) {
      if (DFSClient.LOG.isDebugEnabled()) {
        DFSClient.LOG.debug("DFSClient writeChunk packet full seqno=" +
            currentPacket.getSeqno() +
            ", src=" + src +
            ", bytesCurBlock=" + bytesCurBlock +
            ", blockSize=" + blockSize +
            ", appendChunk=" + appendChunk);
      }
      waitAndQueueCurrentPacket();

      // If the reopened file did not end at chunk boundary and the above
      // write filled up its partial chunk. Tell the summer to generate full 
      // crc chunks from now on.
      if (appendChunk && bytesCurBlock%bytesPerChecksum == 0) {
        appendChunk = false;
        resetChecksumBufSize();
      }

      if (!appendChunk) {
        int psize = Math.min((int)(blockSize-bytesCurBlock), dfsClient.getConf().writePacketSize);
        computePacketChunkSize(psize, bytesPerChecksum);
      }
      //
      // if encountering a block boundary, send an empty packet to 
      // indicate the end of block and reset bytesCurBlock.
      //
      if (bytesCurBlock == blockSize) {
        currentPacket = createPacket(0, 0, bytesCurBlock, currentSeqno++, true);
        currentPacket.setSyncBlock(shouldSyncBlock);
        waitAndQueueCurrentPacket();
        bytesCurBlock = 0;
        lastFlushOffset = 0;
      }
    }
  }

1.分配一个新的packet，createPacket
2.往这个packet中写checum、data：

	currentPacket.writeChecksum(checksum, ckoff, cklen);
    currentPacket.writeData(b, offset, len);
    currentPacket.incNumChunks();
    bytesCurBlock += len;

在这里插入图片描述

  private void waitAndQueueCurrentPacket() throws IOException {
    synchronized (dataQueue) {
      try {
      // If queue is full, then wait till we have enough space
        boolean firstWait = true;
        try {
          while (!isClosed() && dataQueue.size() + ackQueue.size() >
              dfsClient.getConf().writeMaxPackets) {
            if (firstWait) {
              Span span = Trace.currentSpan();
              if (span != null) {
                span.addTimelineAnnotation("dataQueue.wait");
              }
              firstWait = false;
            }
            try {
              dataQueue.wait();
            } catch (InterruptedException e) {
              // If we get interrupted while waiting to queue data, we still need to get rid
              // of the current packet. This is because we have an invariant that if
              // currentPacket gets full, it will get queued before the next writeChunk.
              //
              // Rather than wait around for space in the queue, we should instead try to
              // return to the caller as soon as possible, even though we slightly overrun
              // the MAX_PACKETS length.
              Thread.currentThread().interrupt();
              break;
            }
          }
        } finally {
          Span span = Trace.currentSpan();
          if ((span != null) && (!firstWait)) {
            span.addTimelineAnnotation("end.wait");
          }
        }
        checkClosed();
        queueCurrentPacket();
      } catch (ClosedChannelException e) {
      }
    }
  }
  
  private void queueCurrentPacket() {
    synchronized (dataQueue) {
      if (currentPacket == null) return;
      currentPacket.addTraceParent(Trace.currentSpan());
      dataQueue.addLast(currentPacket);
      lastQueuedSeqno = currentPacket.getSeqno();
      if (DFSClient.LOG.isDebugEnabled()) {
        DFSClient.LOG.debug("Queued packet " + currentPacket.getSeqno());
      }
      currentPacket = null;
      dataQueue.notifyAll();
    }
  }

writeMaxPackets 默认80，dataQueue + ackQueue > 80时就要等待。

在这里插入图片描述
1.使用nextBlockOutputStream向集群申请分配新的数据块。然后构造这个数据块的pipeline。
在这里插入图片描述

run(){
	...
	// get new block from namenode.
    if (stage == BlockConstructionStage.PIPELINE_SETUP_CREATE) {
          if(DFSClient.LOG.isDebugEnabled()) {
              DFSClient.LOG.debug("Allocating new block");
            }
            setPipeline(nextBlockOutputStream());
            initDataStreaming();
          } else if ...
	...
}

上述三个方法如下：

  • nextBlockOutputStream: 向NN申请分配新的数据块。
private LocatedBlock nextBlockOutputStream(){
		...
		// 故障节点
        DatanodeInfo[] excluded =
            excludedNodes.getAllPresent(excludedNodes.asMap().keySet())
            .keySet()
            .toArray(new DatanodeInfo[0]);
        ...  
        // 创建到第一个datanode的输出流 
        //
        // Connect to first DataNode in the list.
        //
        success = createBlockOutputStream(nodes, storageTypes, 0L, false);
        // 如果不成功，则放弃块
        if (!success) {
          DFSClient.LOG.info("Abandoning " + block);
          // 将dadanode放入故障节点中
          dfsClient.namenode.abandonBlock(block, fileId, src,
              dfsClient.clientName);
          block = null;
          DFSClient.LOG.info("Excluding datanode " + nodes[errorIndex]);
          excludedNodes.put(nodes[errorIndex], nodes[errorIndex]);
        }
      } while (!success && --count >= 0); // 进行重试
}

2.获取到数据管道中的第一个datanode的IO流，然后通过这个IO流发送数据包。

			// get packet to be sent.
			// 取出一个包，或者构造一个心跳包
            if (dataQueue.isEmpty()) {
              one = createHeartbeatPacket(); 
              assert one != null;
            } else {
              one = dataQueue.getFirst(); // regular data packet
              long parents[] = one.getTraceParents();
              if (parents.length > 0) {
                scope = Trace.startSpan("dataStreamer", new TraceInfo(0, parents[0]));
                // TODO: use setParents API once it's available from HTrace 3.2
//                scope = Trace.startSpan("dataStreamer", Sampler.ALWAYS);
//                scope.getSpan().setParents(parents);
              }
            }

从dataQueue中取出一个待发送的packet，如果是最后一个，则构造一个空的心跳包。
在这里插入图片描述
在这里插入图片描述
心跳包与最后一个包？
A:心跳包一定不是最后一个包。
（四）心跳包
如果长时间没有数据传输，在输出流未关闭的情况下，客户端会发送心跳包给数据节点，心跳包是DataPacket的一种特殊实现，它通过数据包序列号为-1来进行特殊标识，如下：

  /**
   * Check if this packet is a heart beat packet
   * 判断该包释放为心跳包
   *
   * @return true if the sequence number is HEART_BEAT_SEQNO
   */
  boolean isHeartbeatPacket() {
	// 心跳包的序列号均为-1
    return seqno == HEART_BEAT_SEQNO;
  }
            而心跳包的构造如下：
  /**
   * For heartbeat packets, create buffer directly by new byte[]
   * since heartbeats should not be blocked.
   */
  private DFSPacket createHeartbeatPacket() throws InterruptedIOException {
    final byte[] buf = new byte[PacketHeader.PKT_MAX_HEADER_LEN];
    return new DFSPacket(buf, 0, 0, DFSPacket.HEART_BEAT_SEQNO,
                         getChecksumSize(), false);
  }

在这里插入图片描述
在这里插入图片描述
queue到达80个的时候，不能再往队列中加了。必须要发送了。
上述写了一个块：DataStreamer block BP-857948820-10.179.72.122-1560825265775:blk_1073747689_6868

在这里插入图片描述
在这里插入图片描述

hdfs源码解析之DFSClient
上好佳的博客
06-19 851
该构造函数已废弃，接受一个Configuration对象，并调用另一个构造函数获取NameNode地址该构造函数接受一个InetSocketAddress对象和一个Configuration对象，并将InetSocketAddress 转换为URI然后调用另一个基于URI的构造函数该构造函数接受一个URI对象和一个Configuration对象，并将FileSystem.Statistics参数设置为 null，然后调用另一个更完整的构造函数。
【hadoop】Hadoop远程调试
九师兄
10-05 626
在hadoop-env.sh中添加。
windows下使用hdfs调试
hyd19931002的博客
03-04 316
配置环境变量HADOOP_HOME=hadoop安装解压目录 PATH=%HADOOP_HOME%\bin hadoop的bin目录下放置winutils.exe和hadoop.dll 2个文件,要不然报找不到winutils.exe. 2个文件下载地址.版本2.7.2https://download.csdn.net/download/hyd19931002/15563574 以下为使用的pom.xml文件 <?xml version="1.0" encoding="UTF-8"...
Hadoop学习九：hdfs源代码调试
youfashion的博客
04-13 1743
Hadoop学习九：hdfs源代码调试标签（空格分隔）： hadoopHadoop学习九hdfs源代码调试 一利用log4j调试 一，利用log4j调试1,在eclipse工程里面创建source folder 2,拷贝log4j.properties文件[hadoop002@hadoop002 hadoop]$ cp log4j.properties /home/hadoop002/worksp
2. HDFS CLINT WRITE整理版
王小禾
02-23 460
1.概述 通过fsShell或api在客户端断点调试，可以知道写文件入口DistributedFileSystem.create()。 "main@1"prio=5tid=0x1nid=NA runnable java.lang.Thread.State: RUNNABLE at org.apache.hadoop.hdfs.DFSOutputStre...
HDFS client 实例
Love-Coding
04-19 2197
package hdfs; import java.io.IOException; import java.net.URI; import java.net.URISyntaxException;import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataOutputStream; import org
python hdfs.client
qq_40771567的博客
04-28 4979
前言 HDFS主要用来存储文件系统 安装 pip install hdfs 使用 from hdfs.client import Client client = Client(HDFSHOST,root="/",timeout=900,session=False) #连接hdfs 参数说明: classhdfs.Client(url, root=None, proxy=No...
************************************************************/ 2025-11-17 11:39:42,286 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT] 2025-11-17 11:39:44,201 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data 2025-11-17 11:39:44,747 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties 2025-11-17 11:39:45,024 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2025-11-17 11:39:45,025 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started 2025-11-17 11:39:46,402 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling 2025-11-17 11:39:46,463 INFO org.apache.hadoop.hdfs.server.datanode.BlockScanner: Initialized block scanner with targetBytesPerSec 1048576 2025-11-17 11:39:46,488 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is zwk-virtual-machine 2025-11-17 11:39:46,489 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling 2025-11-17 11:39:46,500 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting DataNode with maxLockedMemory = 0 2025-11-17 11:39:46,621 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming server at /0.0.0.0:9866 2025-11-17 11:39:46,626 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwidth is 104857600 bytes/s 2025-11-17 11:39:46,626 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Number threads for balancing is 100 2025-11-17 11:39:47,035 INFO org.eclipse.jetty.util.log: Logging initialized @6372ms to org.eclipse.jetty.util.log.Slf4jLog 2025-11-17 11:39:48,027 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. Reason: Could not read signature secret file: /home/zwk/hadoop-http-auth-signature-secret 2025-11-17 11:39:48,050 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.datanode is not defined 2025-11-17 11:39:48,063 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2025-11-17 11:39:48,066 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode 2025-11-17 11:39:48,066 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2025-11-17 11:39:48,067 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2025-11-17 11:39:48,235 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 44301 2025-11-17 11:39:48,244 INFO org.eclipse.jetty.server.Server: jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 1.8.0_371-b11 2025-11-17 11:39:48,344 INFO org.eclipse.jetty.server.session: DefaultSessionIdManager workerName=node0 2025-11-17 11:39:48,344 INFO org.eclipse.jetty.server.session: No SessionScavenger set, using defaults 2025-11-17 11:39:48,353 INFO org.eclipse.jetty.server.session: node0 Scavenging every 660000ms 2025-11-17 11:39:48,404 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@71529963{logs,/logs,file:///home/zwk/hadoop/hadoop-3.3.5/logs/,AVAILABLE} 2025-11-17 11:39:48,409 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@3f270e0a{static,/static,file:///home/zwk/hadoop/hadoop-3.3.5/share/hadoop/hdfs/webapps/static/,AVAILABLE} 2025-11-17 11:39:48,765 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@aafcffa{datanode,/,file:///home/zwk/hadoop/hadoop-3.3.5/share/hadoop/hdfs/webapps/datanode/,AVAILABLE}{file:/home/zwk/hadoop/hadoop-3.3.5/share/hadoop/hdfs/webapps/datanode} 2025-11-17 11:39:48,807 INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@74c79fa2{HTTP/1.1, (http/1.1)}{localhost:44301} 2025-11-17 11:39:48,809 INFO org.eclipse.jetty.server.Server: Started @8144ms 2025-11-17 11:39:49,100 WARN org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Got null for restCsrfPreventionFilter - will not do any filtering. 2025-11-17 11:39:49,366 INFO org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Listening HTTP traffic on /0.0.0.0:9864 2025-11-17 11:39:49,417 INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor 2025-11-17 11:39:49,434 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = zwk 2025-11-17 11:39:49,434 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = supergroup 2025-11-17 11:39:49,782 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. 2025-11-17 11:39:50,113 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9867 2025-11-17 11:39:50,994 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:9867 2025-11-17 11:39:51,179 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null 2025-11-17 11:39:51,234 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default> 2025-11-17 11:39:51,319 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000 starting to offer service 2025-11-17 11:39:51,392 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2025-11-17 11:39:51,422 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9867: starting 2025-11-17 11:39:53,925 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode during handshakeBlock pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000 2025-11-17 11:39:53,966 INFO org.apache.hadoop.hdfs.server.common.Storage: Using 1 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=1, dataDirs=1) 2025-11-17 11:39:54,005 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data/in_use.lock acquired by nodename 3605@zwk-virtual-machine 2025-11-17 11:39:54,137 INFO org.apache.hadoop.hdfs.server.common.Storage: Analyzing storage directories for bpid BP-655327536-127.0.1.1-1762954302619 2025-11-17 11:39:54,138 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled for /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data/current/BP-655327536-127.0.1.1-1762954302619 2025-11-17 11:39:54,146 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Setting up storage: nsid=1281678374;bpid=BP-655327536-127.0.1.1-1762954302619;lv=-57;nsInfo=lv=-66;cid=CID-de621368-0d95-41d0-9ebe-6538d4b7b6e3;nsid=1281678374;c=1762954302619;bpid=BP-655327536-127.0.1.1-1762954302619;dnuuid=c229f5e4-2241-43fd-8c64-6086b77fa49a 2025-11-17 11:39:54,235 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: The datanode lock is a read write lock 2025-11-17 11:39:55,179 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added new volume: DS-5c441fde-88f8-4f39-a77a-075dfdc1cdf9 2025-11-17 11:39:55,179 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - [DISK]file:/home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data, StorageType: DISK 2025-11-17 11:39:55,198 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.MemoryMappableBlockLoader: Initializing cache loader: MemoryMappableBlockLoader. 2025-11-17 11:39:55,206 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Registered FSDatasetState MBean 2025-11-17 11:39:55,238 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-655327536-127.0.1.1-1762954302619 2025-11-17 11:39:55,245 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-655327536-127.0.1.1-1762954302619 on volume /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data... 2025-11-17 11:39:55,280 WARN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: dfsUsed file missing in /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data/current/BP-655327536-127.0.1.1-1762954302619/current, will proceed with Du for space computation calculation, 2025-11-17 11:39:55,334 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-655327536-127.0.1.1-1762954302619 on /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data: 88ms 2025-11-17 11:39:55,335 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-655327536-127.0.1.1-1762954302619: 96ms 2025-11-17 11:39:55,337 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding replicas to map for block pool BP-655327536-127.0.1.1-1762954302619 on volume /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data... 2025-11-17 11:39:55,337 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice: Replica Cache file: /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data/current/BP-655327536-127.0.1.1-1762954302619/current/replicas doesn't exist 2025-11-17 11:39:55,427 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-655327536-127.0.1.1-1762954302619 on volume /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data: 90ms 2025-11-17 11:39:55,428 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to add all replicas to map for block pool BP-655327536-127.0.1.1-1762954302619: 92ms 2025-11-17 11:39:55,428 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data 2025-11-17 11:39:55,447 INFO org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker: Scheduled health check for volume /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data 2025-11-17 11:39:55,479 INFO org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data, DS-5c441fde-88f8-4f39-a77a-075dfdc1cdf9): no suitable block pools found to scan. Waiting 1417960410 ms. 2025-11-17 11:39:55,486 WARN org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value above 1000 ms/sec. Assuming default value of -1 2025-11-17 11:39:55,487 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting in 3506423ms with interval of 21600000ms and throttle limit of -1ms/s 2025-11-17 11:39:55,504 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-655327536-127.0.1.1-1762954302619 (Datanode Uuid c229f5e4-2241-43fd-8c64-6086b77fa49a) service to localhost/127.0.0.1:9000 beginning handshake with NN 2025-11-17 11:39:55,682 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-655327536-127.0.1.1-1762954302619 (Datanode Uuid c229f5e4-2241-43fd-8c64-6086b77fa49a) service to localhost/127.0.0.1:9000 successfully registered with NN 2025-11-17 11:39:55,682 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode localhost/127.0.0.1:9000 using BLOCKREPORT_INTERVAL of 21600000msecs CACHEREPORT_INTERVAL of 10000msecs Initial delay: 0msecs; heartBeatInterval=3000 2025-11-17 11:39:56,204 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Successfully sent block report 0xdb29685bfe76cad with lease ID 0xf06eba0d25825652 to namenode: localhost/127.0.0.1:9000, containing 1 storage report(s), of which we sent 1. The reports had 59 total blocks and used 1 RPC(s). This took 23 msecs to generate and 224 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5. 2025-11-17 11:39:56,208 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Got finalize command for block pool BP-655327536-127.0.1.1-1762954302619 2025-11-17 11:41:22,791 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-655327536-127.0.1.1-1762954302619:blk_1073741919_1102 src: /127.0.0.1:39016 dest: /127.0.0.1:9866 2025-11-17 11:41:22,834 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:39016, dest: /127.0.0.1:9866, bytes: 202, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-433486687_1, offset: 0, srvID: c229f5e4-2241-43fd-8c64-6086b77fa49a, blockid: BP-655327536-127.0.1.1-1762954302619:blk_1073741919_1102, duration(ns): 15716338 2025-11-17 11:41:22,834 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-655327536-127.0.1.1-1762954302619:blk_1073741919_1102, type=LAST_IN_PIPELINE terminating 2025-11-17 11:41:25,717 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockRecoveryWorker: NameNode at localhost/127.0.0.1:9000 calls recoverBlock(BP-655327536-127.0.1.1-1762954302619:blk_1073741918_1101, targets=[DatanodeInfoWithStorage[127.0.0.1:9866,null,null]], newGenerationStamp=1103, newBlock=null, isStriped=false) 2025-11-17 11:41:25,719 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: initReplicaRecovery: blk_1073741918_1101, recoveryId=1103, replica=ReplicaWaitingToBeRecovered, blk_1073741918_1101, RWR getNumBytes() = 85 getBytesOnDisk() = 85 getVisibleLength()= -1 getVolume() = /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data getBlockURI() = file:/home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data/current/BP-655327536-127.0.1.1-1762954302619/current/rbw/blk_1073741918 2025-11-17 11:41:25,719 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: initReplicaRecovery: changing replica state for blk_1073741918_1101 from RWR to RUR 2025-11-17 11:41:25,721 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockRecoveryWorker: block=BP-655327536-127.0.1.1-1762954302619:blk_1073741918_1101 (length=0), isTruncateRecovery=false, syncList=[block:blk_1073741918_1101[numBytes=85,originalReplicaState=RWR] node:DatanodeInfoWithStorage[127.0.0.1:9866,null,null]] 2025-11-17 11:41:25,721 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockRecoveryWorker: block=BP-655327536-127.0.1.1-1762954302619:blk_1073741918_1101 (length=0), bestState=RWR, newBlock=BP-655327536-127.0.1.1-1762954302619:blk_1073741918_1103 (length=85), participatingList=[block:blk_1073741918_1101[numBytes=85,originalReplicaState=RWR] node:DatanodeInfoWithStorage[127.0.0.1:9866,null,null]] 2025-11-17 11:41:25,722 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: updateReplica: BP-655327536-127.0.1.1-1762954302619:blk_1073741918_1101[numBytes=85,originalReplicaState=RWR], recoveryId=1103, length=85, replica=ReplicaUnderRecovery, blk_1073741918_1101, RUR getNumBytes() = 85 getBytesOnDisk() = 85 getVisibleLength()= -1 getVolume() = /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data getBlockURI() = file:/home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data/current/BP-655327536-127.0.1.1-1762954302619/current/rbw/blk_1073741918 recoveryId=1103 original=ReplicaWaitingToBeRecovered, blk_1073741918_1101, RWR getNumBytes() = 85 getBytesOnDisk() = 85 getVisibleLength()= -1 getVolume() = /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data getBlockURI() = file:/home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data/current/BP-655327536-127.0.1.1-1762954302619/current/rbw/blk_1073741918 2025-11-17 11:41:27,636 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-655327536-127.0.1.1-1762954302619:blk_1073741920_1104 src: /127.0.0.1:39028 dest: /127.0.0.1:9866 2025-11-17 11:41:28,738 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741911_1091 replica FinalizedReplica, blk_1073741911_1091, FINALIZED getNumBytes() = 202 getBytesOnDisk() = 202 getVisibleLength()= 202 getVolume() = /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data getBlockURI() = file:/home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data/current/BP-655327536-127.0.1.1-1762954302619/current/finalized/subdir0/subdir0/blk_1073741911 for deletion 2025-11-17 11:41:28,742 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-655327536-127.0.1.1-1762954302619 blk_1073741911_1091 URI file:/home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data/current/BP-655327536-127.0.1.1-1762954302619/current/finalized/subdir0/subdir0/blk_1073741911 2025-11-17 11:41:31,695 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741918_1103 replica FinalizedReplica, blk_1073741918_1103, FINALIZED getNumBytes() = 85 getBytesOnDisk() = 85 getVisibleLength()= 85 getVolume() = /home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data getBlockURI() = file:/home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data/current/BP-655327536-127.0.1.1-1762954302619/current/finalized/subdir0/subdir0/blk_1073741918 for deletion 2025-11-17 11:41:31,696 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-655327536-127.0.1.1-1762954302619 blk_1073741918_1103 URI file:/home/zwk/hadoop/hadoop-3.3.5/tmp/dfs/data/current/BP-655327536-127.0.1.1-1762954302619/current/finalized/subdir0/subdir0/blk_1073741918 2025-11-17 11:41:33,777 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-655327536-127.0.1.1-1762954302619:blk_1073741921_1105 src: /127.0.0.1:47140 dest: /127.0.0.1:9866 2025-11-17 11:41:34,429 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-655327536-127.0.1.1-1762954302619:blk_1073741922_1106 src: /127.0.0.1:47168 dest: /127.0.0.1:9866 2025-11-17 11:49:36,751 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-655327536-127.0.1.1-1762954302619:blk_1073741923_1107 src: /127.0.0.1:53574 dest: /127.0.0.1:9866 2025-11-17 11:49:36,852 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:53574, dest: /127.0.0.1:9866, bytes: 7988, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1974153692_1, offset: 0, srvID: c229f5e4-2241-43fd-8c64-6086b77fa49a, blockid: BP-655327536-127.0.1.1-1762954302619:blk_1073741923_1107, duration(ns): 22420015 2025-11-17 11:49:36,852 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-655327536-127.0.1.1-1762954302619:blk_1073741923_1107, type=LAST_IN_PIPELINE terminating提交任务没有output
最新发布
11-18
<value>file:///disk1/hdfs/data,file:///disk2/hdfs/data ``` 2. **定期健康检查脚本** ```bash #!/bin/bash hdfs dfsadmin -report | grep "Live" | wc -l hdfs fsck / | grep "Under replicated" | awk ...
eclipse操作HDFS时异常的解决: Permission denied: user=administrator, access=WRITE,
ForestCat的专栏
12-20 907
在虚拟机上搭建了一个Hadoop的环境（4节点），在用windows上的Java客户端来操作集群中的HDFS文件时出现了如下的认证错误 org.apache.hadoop.security.AccessControlException: Permission denied: user=Administrator, access=WRITE, inode="/input4":root:...
大数据实验（二）HDFS API的使用（Python 3.7）
while0l1的博客
12-15 1736
前言 书本第六章的主要内容就是讲了HDFS的一些操作指令，还有非常简略的Java调用HDFS API代码的示例。虽然据说用Java的运行效率会高很多，可是我也没有系统地学过Java，而且看样子实现做起来比较繁琐，所以我还是选择了Python来实现HDFS API的调用。用Python特别好的地方就是后面的数据处理和可视化会方便很多，人生苦短，我用Python，哈哈哈哈哈哈哈哈。 HDFS API的...
D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\bin\java.exe "-javaagent:D:\Java\idea\IDEA 2023.1\IDEA neo\lib\idea_rt.jar=11999:D:\Java\idea\IDEA 2023.1\IDEA neo\bin" -Dfile.encoding=UTF-8 -classpath D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\charsets.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\deploy.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\access-bridge-64.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\cldrdata.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\dnsns.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\jaccess.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\jfxrt.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\localedata.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\nashorn.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunec.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunjce_provider.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunmscapi.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\sunpkcs11.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\ext\zipfs.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\javaws.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jce.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jfr.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jfxswt.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\jsse.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\management-agent.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\plugin.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\resources.jar;D:\Java\jdk1.8\jdk1.8\jdk1.8.0_181\jre\lib\rt.jar;D:\Java\Hadoop_code\out\production\Hadoop_code;D:\443zrs\lib2\fst-2.50.jar;D:\443zrs\lib2\re2j-1.1.jar;D:\443zrs\lib2\asm-5.0.4.jar;D:\443zrs\lib2\guice-4.0.jar;D:\443zrs\lib2\jna-5.2.0.jar;D:\443zrs\lib2\avro-1.7.7.jar;D:\443zrs\lib2\gson-2.8.9.jar;D:\443zrs\lib2\okio-2.8.0.jar;D:\443zrs\lib2\jline-3.9.0.jar;D:\443zrs\lib2\jsch-0.1.55.jar;D:\443zrs\lib2\asm-tree-9.1.jar;D:\443zrs\lib2\jettison-1.1.jar;D:\443zrs\lib2\jsr305-3.0.2.jar;D:\443zrs\lib2\log4j-1.2.17.jar;D:\443zrs\lib2\okhttp-4.9.3.jar;D:\443zrs\lib2\dnsjava-2.1.7.jar;D:\443zrs\lib2\ehcache-3.3.1.jar;D:\443zrs\lib2\json-io-2.5.1.jar;D:\443zrs\lib2\objenesis-2.6.jar;D:\443zrs\lib2\paranamer-2.3.jar;D:\443zrs\lib2\guava-27.0-jre.jar;D:\443zrs\lib2\javax.inject-1.jar;D:\443zrs\lib2\snakeyaml-1.26.jar;D:\443zrs\lib2\aopalliance-1.0.jar;D:\443zrs\lib2\asm-commons-9.1.jar;D:\443zrs\lib2\commons-cli-1.2.jar;D:\443zrs\lib2\commons-net-3.6.jar;D:\443zrs\lib2\httpcore-4.4.13.jar;D:\443zrs\lib2\java-util-1.9.0.jar;D:\443zrs\lib2\jaxb-api-2.2.11.jar;D:\443zrs\lib2\kerb-core-1.0.1.jar;D:\443zrs\lib2\kerb-util-1.0.1.jar;D:\443zrs\lib2\kerby-xdr-1.0.1.jar;D:\443zrs\lib2\reload4j-1.2.22.jar;D:\443zrs\lib2\stax2-api-4.2.1.jar;D:\443zrs\lib2\zookeeper-3.5.6.jar;D:\443zrs\lib2\asm-analysis-9.1.jar;D:\443zrs\lib2\commons-io-2.8.0.jar;D:\443zrs\lib2\commons-text-1.4.jar;D:\443zrs\lib2\hadoop-kms-3.3.4.jar;D:\443zrs\lib2\hadoop-nfs-3.3.4.jar;D:\443zrs\lib2\jersey-core-1.19.jar;D:\443zrs\lib2\jersey-json-1.19.jar;D:\443zrs\lib2\json-smart-2.4.7.jar;D:\443zrs\lib2\jsr311-api-1.1.1.jar;D:\443zrs\lib2\kerb-admin-1.0.1.jar;D:\443zrs\lib2\kerby-asn1-1.0.1.jar;D:\443zrs\lib2\kerby-pkix-1.0.1.jar;D:\443zrs\lib2\kerby-util-1.0.1.jar;D:\443zrs\lib2\slf4j-api-1.7.36.jar;D:\443zrs\lib2\failureaccess-1.0.jar;D:\443zrs\lib2\guice-servlet-4.0.jar;D:\443zrs\lib2\hadoop-auth-3.3.4.jar;D:\443zrs\lib2\hadoop-hdfs-3.3.4.jar;D:\443zrs\lib2\httpclient-4.5.13.jar;D:\443zrs\lib2\jackson-xc-1.9.13.jar;D:\443zrs\lib2\jaxb-impl-2.2.3-1.jar;D:\443zrs\lib2\jersey-guice-1.19.jar;D:\443zrs\lib2\json-simple-1.1.1.jar;D:\443zrs\lib2\kerb-client-1.0.1.jar;D:\443zrs\lib2\kerb-common-1.0.1.jar;D:\443zrs\lib2\kerb-crypto-1.0.1.jar;D:\443zrs\lib2\kerb-server-1.0.1.jar;D:\443zrs\lib2\checker-qual-2.5.2.jar;D:\443zrs\lib2\commons-codec-1.15.jar;D:\443zrs\lib2\jersey-client-1.19.jar;D:\443zrs\lib2\jersey-server-1.19.jar;D:\443zrs\lib2\kerby-config-1.0.1.jar;D:\443zrs\lib2\leveldbjni-all-1.8.jar;D:\443zrs\lib2\metrics-core-3.2.4.jar;D:\443zrs\lib2\netty-3.10.6.Final.jar;D:\443zrs\lib2\bcpkix-jdk15on-1.60.jar;D:\443zrs\lib2\bcprov-jdk15on-1.60.jar;D:\443zrs\lib2\commons-math3-3.1.1.jar;D:\443zrs\lib2\hadoop-common-3.3.4.jar;D:\443zrs\lib2\jackson-core-2.12.7.jar;D:\443zrs\lib2\jersey-servlet-1.19.jar;D:\443zrs\lib2\kerb-identity-1.0.1.jar;D:\443zrs\lib2\protobuf-java-2.5.0.jar;D:\443zrs\lib2\snappy-java-1.1.8.2.jar;D:\443zrs\lib2\woodstox-core-5.3.0.jar;D:\443zrs\lib2\commons-lang3-3.12.0.jar;D:\443zrs\lib2\curator-client-4.2.0.jar;D:\443zrs\lib2\jackson-jaxrs-1.9.13.jar;D:\443zrs\lib2\kerb-simplekdc-1.0.1.jar;D:\443zrs\lib2\kotlin-stdlib-1.4.10.jar;D:\443zrs\lib2\token-provider-1.0.1.jar;D:\443zrs\lib2\zookeeper-jute-3.5.6.jar;D:\443zrs\lib2\accessors-smart-2.4.7.jar;D:\443zrs\lib2\commons-compress-1.21.jar;D:\443zrs\lib2\commons-daemon-1.0.13.jar;D:\443zrs\lib2\commons-logging-1.1.3.jar;D:\443zrs\lib2\curator-recipes-4.2.0.jar;D:\443zrs\lib2\hadoop-hdfs-nfs-3.3.4.jar;D:\443zrs\lib2\hadoop-hdfs-rbf-3.3.4.jar;D:\443zrs\lib2\hadoop-registry-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-api-3.3.4.jar;D:\443zrs\lib2\HikariCP-java7-2.4.12.jar;D:\443zrs\lib2\mssql-jdbc-6.2.1.jre7.jar;D:\443zrs\lib2\nimbus-jose-jwt-9.8.1.jar;D:\443zrs\lib2\slf4j-reload4j-1.7.36.jar;D:\443zrs\lib2\j2objc-annotations-1.1.jar;D:\443zrs\lib2\jcip-annotations-1.0-1.jar;D:\443zrs\lib2\netty-all-4.1.77.Final.jar;D:\443zrs\lib2\commons-beanutils-1.9.4.jar;D:\443zrs\lib2\curator-framework-4.2.0.jar;D:\443zrs\lib2\hadoop-hdfs-3.3.4-tests.jar;D:\443zrs\lib2\jackson-core-asl-1.9.13.jar;D:\443zrs\lib2\jackson-databind-2.12.7.jar;D:\443zrs\lib2\javax.servlet-api-3.1.0.jar;D:\443zrs\lib2\javax.websocket-api-1.0.jar;D:\443zrs\lib2\hadoop-annotations-3.3.4.jar;D:\443zrs\lib2\hadoop-hdfs-client-3.3.4.jar;D:\443zrs\lib2\hadoop-hdfs-httpfs-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-client-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-common-3.3.4.jar;D:\443zrs\lib2\netty-codec-4.1.77.Final.jar;D:\443zrs\lib2\commons-collections-3.2.2.jar;D:\443zrs\lib2\hadoop-common-3.3.4-tests.jar;D:\443zrs\lib2\hadoop-shaded-guava-1.1.1.jar;D:\443zrs\lib2\jackson-jaxrs-base-2.12.7.jar;D:\443zrs\lib2\jackson-mapper-asl-1.9.13.jar;D:\443zrs\lib2\jetty-io-9.4.43.v20210629.jar;D:\443zrs\lib2\netty-buffer-4.1.77.Final.jar;D:\443zrs\lib2\netty-common-4.1.77.Final.jar;D:\443zrs\lib2\swagger-annotations-1.5.4.jar;D:\443zrs\lib2\audience-annotations-0.5.0.jar;D:\443zrs\lib2\hadoop-yarn-registry-3.3.4.jar;D:\443zrs\lib2\jackson-annotations-2.12.7.jar;D:\443zrs\lib2\jakarta.xml.bind-api-2.3.2.jar;D:\443zrs\lib2\jetty-xml-9.4.43.v20210629.jar;D:\443zrs\lib2\netty-handler-4.1.77.Final.jar;D:\443zrs\lib2\hadoop-hdfs-rbf-3.3.4-tests.jar;D:\443zrs\lib2\jetty-http-9.4.43.v20210629.jar;D:\443zrs\lib2\jetty-jndi-9.4.43.v20210629.jar;D:\443zrs\lib2\jetty-plus-9.4.43.v20210629.jar;D:\443zrs\lib2\jetty-util-9.4.43.v20210629.jar;D:\443zrs\lib2\kotlin-stdlib-common-1.4.10.jar;D:\443zrs\lib2\netty-resolver-4.1.77.Final.jar;D:\443zrs\lib2\commons-configuration2-2.1.1.jar;D:\443zrs\lib2\jakarta.activation-api-1.2.1.jar;D:\443zrs\lib2\netty-codec-dns-4.1.77.Final.jar;D:\443zrs\lib2\netty-codec-xml-4.1.77.Final.jar;D:\443zrs\lib2\netty-transport-4.1.77.Final.jar;D:\443zrs\lib2\jetty-client-9.4.43.v20210629.jar;D:\443zrs\lib2\jetty-server-9.4.43.v20210629.jar;D:\443zrs\lib2\jetty-webapp-9.4.43.v20210629.jar;D:\443zrs\lib2\netty-codec-http-4.1.77.Final.jar;D:\443zrs\lib2\netty-codec-mqtt-4.1.77.Final.jar;D:\443zrs\lib2\netty-codec-smtp-4.1.77.Final.jar;D:\443zrs\lib2\hadoop-hdfs-client-3.3.4-tests.jar;D:\443zrs\lib2\hadoop-yarn-server-tests-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-services-api-3.3.4.jar;D:\443zrs\lib2\javax.websocket-client-api-1.0.jar;D:\443zrs\lib2\jetty-servlet-9.4.43.v20210629.jar;D:\443zrs\lib2\netty-codec-http2-4.1.77.Final.jar;D:\443zrs\lib2\netty-codec-redis-4.1.77.Final.jar;D:\443zrs\lib2\netty-codec-socks-4.1.77.Final.jar;D:\443zrs\lib2\netty-codec-stomp-4.1.77.Final.jar;D:\443zrs\lib2\websocket-api-9.4.43.v20210629.jar;D:\443zrs\lib2\animal-sniffer-annotations-1.17.jar;D:\443zrs\lib2\hadoop-hdfs-native-client-3.3.4.jar;D:\443zrs\lib2\hadoop-mapreduce-examples-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-server-common-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-server-router-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-services-core-3.3.4.jar;D:\443zrs\lib2\jetty-security-9.4.43.v20210629.jar;D:\443zrs\lib2\netty-resolver-dns-4.1.77.Final.jar;D:\443zrs\lib2\hadoop-mapreduce-client-hs-3.3.4.jar;D:\443zrs\lib2\hadoop-shaded-protobuf_3_7-1.1.1.jar;D:\443zrs\lib2\jetty-util-ajax-9.4.43.v20210629.jar;D:\443zrs\lib2\netty-codec-haproxy-4.1.77.Final.jar;D:\443zrs\lib2\netty-handler-proxy-4.1.77.Final.jar;D:\443zrs\lib2\netty-transport-udt-4.1.77.Final.jar;D:\443zrs\lib2\hadoop-mapreduce-client-app-3.3.4.jar;D:\443zrs\lib2\netty-codec-memcache-4.1.77.Final.jar;D:\443zrs\lib2\netty-transport-rxtx-4.1.77.Final.jar;D:\443zrs\lib2\netty-transport-sctp-4.1.77.Final.jar;D:\443zrs\lib2\websocket-client-9.4.43.v20210629.jar;D:\443zrs\lib2\websocket-common-9.4.43.v20210629.jar;D:\443zrs\lib2\websocket-server-9.4.43.v20210629.jar;D:\443zrs\lib2\hadoop-mapreduce-client-core-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-server-web-proxy-3.3.4.jar;D:\443zrs\lib2\jackson-jaxrs-json-provider-2.12.7.jar;D:\443zrs\lib2\jetty-annotations-9.4.43.v20210629.jar;D:\443zrs\lib2\websocket-servlet-9.4.43.v20210629.jar;D:\443zrs\lib2\geronimo-jcache_1.0_spec-1.0-alpha-1.jar;D:\443zrs\lib2\hadoop-mapreduce-client-common-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-server-nodemanager-3.3.4.jar;D:\443zrs\lib2\hadoop-hdfs-native-client-3.3.4-tests.jar;D:\443zrs\lib2\hadoop-mapreduce-client-shuffle-3.3.4.jar;D:\443zrs\lib2\hadoop-mapreduce-client-uploader-3.3.4.jar;D:\443zrs\lib2\jackson-module-jaxb-annotations-2.12.7.jar;D:\443zrs\lib2\hadoop-mapreduce-client-jobclient-3.3.4.jar;D:\443zrs\lib2\hadoop-mapreduce-client-hs-plugins-3.3.4.jar;D:\443zrs\lib2\hadoop-mapreduce-client-nativetask-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-applications-mawo-core-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-server-resourcemanager-3.3.4.jar;D:\443zrs\lib2\netty-transport-classes-epoll-4.1.77.Final.jar;D:\443zrs\lib2\hadoop-yarn-server-sharedcachemanager-3.3.4.jar;D:\443zrs\lib2\netty-transport-classes-kqueue-4.1.77.Final.jar;D:\443zrs\lib2\javax-websocket-client-impl-9.4.43.v20210629.jar;D:\443zrs\lib2\javax-websocket-server-impl-9.4.43.v20210629.jar;D:\443zrs\lib2\hadoop-mapreduce-client-jobclient-3.3.4-tests.jar;D:\443zrs\lib2\netty-resolver-dns-classes-macos-4.1.77.Final.jar;D:\443zrs\lib2\hadoop-yarn-applications-distributedshell-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-server-timeline-pluginstorage-3.3.4.jar;D:\443zrs\lib2\netty-transport-native-unix-common-4.1.77.Final.jar;D:\443zrs\lib2\hadoop-yarn-server-applicationhistoryservice-3.3.4.jar;D:\443zrs\lib2\hadoop-yarn-applications-unmanaged-am-launcher-3.3.4.jar;D:\443zrs\lib2\netty-transport-native-kqueue-4.1.77.Final-osx-x86_64.jar;D:\443zrs\lib2\netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar;D:\443zrs\lib2\netty-resolver-dns-native-macos-4.1.77.Final-osx-x86_64.jar;D:\443zrs\lib2\netty-transport-native-kqueue-4.1.77.Final-osx-aarch_64.jar;D:\443zrs\lib2\netty-transport-native-epoll-4.1.77.Final-linux-aarch_64.jar;D:\443zrs\lib2\netty-resolver-dns-native-macos-4.1.77.Final-osx-aarch_64.jar;D:\443zrs\lib2\listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar fuirt_sale 2025-11-04 02:02:45,349 WARN [main] util.Shell (Shell.java:<clinit>(692)) - Did not find winutils.exe: {} java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:547) at org.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:568) at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:591) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:688) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1712) at org.apache.hadoop.security.SecurityUtil.setConfigurationInternal(SecurityUtil.java:99) at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:88) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:312) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:300) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:575) at org.apache.hadoop.mapreduce.task.JobContextImpl.<init>(JobContextImpl.java:72) at org.apache.hadoop.mapreduce.Job.<init>(Job.java:152) at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:195) at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:214) at fuirt_sale.main(fuirt_sale.java:18) Caused by: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:467) at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:438) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:515) ... 12 more 2025-11-04 02:02:50,887 WARN [main] impl.MetricsConfig (MetricsConfig.java:loadFirst(136)) - Cannot locate configuration: tried hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties 2025-11-04 02:02:50,917 INFO [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(378)) - Scheduled Metric snapshot period at 10 second(s). 2025-11-04 02:02:50,917 INFO [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:start(191)) - JobTracker metrics system started 2025-11-04 02:02:51,257 WARN [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadResourcesInternal(149)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2025-11-04 02:02:51,271 WARN [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadJobJar(482)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2025-11-04 02:02:51,296 INFO [main] input.FileInputFormat (FileInputFormat.java:listStatus(300)) - Total input files to process : 1 2025-11-04 02:02:51,337 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(202)) - number of splits:1 2025-11-04 02:02:51,386 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(298)) - Submitting tokens for job: job_local1602603729_0001 2025-11-04 02:02:51,386 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(299)) - Executing with tokens: [] 2025-11-04 02:02:51,449 INFO [main] mapreduce.Job (Job.java:submit(1575)) - The url to track the job: http://localhost:8080/ 2025-11-04 02:02:51,449 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1620)) - Running job: job_local1602603729_0001 2025-11-04 02:02:51,449 INFO [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(501)) - OutputCommitter set in config null 2025-11-04 02:02:51,453 INFO [Thread-6] output.FileOutputCommitter (FileOutputCommitter.java:<init>(142)) - File Output Committer Algorithm version is 2 2025-11-04 02:02:51,453 INFO [Thread-6] output.FileOutputCommitter (FileOutputCommitter.java:<init>(157)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 2025-11-04 02:02:51,453 INFO [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(519)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2025-11-04 02:02:51,467 WARN [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local1602603729_0001 org.apache.hadoop.security.AccessControlException: Permission denied: user=zrs, access=WRITE, inode="/user/root/data2":root:supergroup:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:506) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:346) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermissionWithContext(FSPermissionChecker.java:370) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:240) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1943) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1927) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1886) at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:60) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3405) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1159) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:740) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:604) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2509) at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2483) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1485) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1482) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1499) at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1474) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2388) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:356) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:541) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=zrs, access=WRITE, inode="/user/root/data2":root:supergroup:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:506) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:346) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermissionWithContext(FSPermissionChecker.java:370) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:240) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1943) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1927) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1886) at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:60) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3405) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1159) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:740) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:604) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:572) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1612) at org.apache.hadoop.ipc.Client.call(Client.java:1558) at org.apache.hadoop.ipc.Client.call(Client.java:1455) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:129) at com.sun.proxy.$Proxy9.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:674) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy10.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2507) ... 9 more 2025-11-04 02:02:52,464 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1641)) - Job job_local1602603729_0001 running in uber mode : false 2025-11-04 02:02:52,466 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1648)) - map 0% reduce 0% 2025-11-04 02:02:52,469 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1661)) - Job job_local1602603729_0001 failed with state FAILED due to: NA 2025-11-04 02:02:52,483 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1666)) - Counters: 0 进程已结束,退出代码1
11-05
1. **Hadoop 权限控制机制**：HDFS 使用 POSIX 类似权限模型（用户/组/其他 + 读写执行），写入需 WRITE 权限，否则抛出 `AccessControlException`。 2. **Windows 下 Hadoop 运行依赖 winutils.exe**：Hadoop 原生...
HDFS的java客户端操作
weixin_45102492的博客
02-18 432
HDFS的Java客户端操作 HDFS客户端环境准备 详见： https://blog.csdn.net/weixin_45102492/article/details/103008541 创建一个Maven工程HdfsClientDemo,导入相应的依赖坐标+日志添加 <dependencies> <dependency> <groupId>juni...
hdfs client api
qq_38250124的博客
03-23 381
public class HdfsClient { FileSystem fs = null; @Before public void init() throws Exception { Configuration conf = new Configuration(); // conf.set("fs.defaultFS","hdfs://m
HDFS client写文件过程 源码分析
thaddeuslu的博客
10-12 416
HDFS client写文件过程 源码分析 HDFS写入文件的重要概念 HDFS一个文件由多个block构成。HDFS在进行block读写的时候是以packet(默认每个packet为64K)为单位进行的。每一个packet由若干个chunk（默认512Byte）组成。Chunk是进行数据校验的基本单位，对每一个chunk生成一个校验和(默认4Byte)并将校验和进行存储。 在写入一个block的...
ESP8266作数据转发client.write的使用坑
yuanqia4079的博客
04-05 3997
ESP8266接收从STM32送来的串口浮点数据，转发给服务器。使用这一段开启client。（ticker的detach可以解除关联的函数。让函数不再定时运行。ticker是一个实例对象，它只能唯一关联一个函数。所以可以这么操作）while (!client.connect(host, httpPort)) {   ticker.detach();//常亮不闪说明没连上服务器   digitalW...
log4j打印sql
Y_JY
09-28 555
log4j.rootLogger=DEBUG,stdout   log4j.logger.java.sql.PreparedStatement=DEBUG,stdout      log4j.appender.stdout=org.apache.log4j.ConsoleAppender   log4j.appender.stdout.layout=org.apache.log4j.Pat
使用python的hdfs包操作HDFS
热门推荐
jiede1的博客
04-05 1万+
项目需要利用python访问hdfs，查到可以利用hdfs包实现这一点，总结了下使用方法。 1.安装 我在Linux环境下安装 pip install hdfs 2.连接hdfs from hdfs.client import Client # 50070: NameNode web管理端口 client = Client("http://localhost:50070") ...
HDFS中不起作用的buffer size
渴死的火烈鸟
05-12 404
/** * Create a new output stream to the given DataNode. * @see ClientProtocol#create(String, FsPermission, String, boolean, short, long) */ DFSOutputStream(String src, FsPermis...
HDFS租约解析
半日闲的博客
03-28 4376
转自：http://bigdatadecode.club/HDFS%E7%A7%9F%E7%BA%A6%E8%A7%A3%E6%9E%90.html 租约(Lease)是一种广泛应用与分布式系统领域的协议，主要用来维护分布式系统的一致性。 租约是在解决缓存一致性时被提出的。所谓租约，其实就是一个合同，即服务器给予客户端在一定期限内可以控制修改操作的权力。如果服务器要修改数据，首先要征求拥有
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值