1. HDFS CLIENT WRITE调试篇

已于 2022-08-16 14:34:31 修改
阅读量1.5k 收藏 3
点赞数 1
分类专栏： # 读写 文章标签： hadoop hdfs
于 2019-12-19 16:48:05 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/answer100answer/article/details/103612088
版权

文章目录

知乎 https://zhuanlan.zhihu.com/p/66051354

使用测试集群使用FsShell断点

    @Test
    public void testWrite() throws Exception{
        FsShell shell = new FsShell();
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS","hdfs://hadoop1:9000");
        conf.setQuietMode(false);
        shell.setConf(conf);
        System.setProperty("HADOOP_USER_NAME", "hadoop");
        System.setProperty("HADOOP_USER_PASSWORD", "xxx");
        String local = "/Users/didi/CodeFile/proTest/hadoop-test/src/main/resources/test/core-site.xml";
        String[] args = {"-put", local, "/test/write/1219-1.xml"};   // 普通目录ok
        int res = 1;
        try {
            res = ToolRunner.run(shell, args);
        } finally {
            shell.close();
        }
        System.exit(res);
    }

在org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.<init>构造器中加断点。

DFSOutputStream

"main@1" prio=5 tid=0x1 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.<init>(DFSOutputStream.java:276)
	  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.<init>(DFSOutputStream.java:227)
	  at org.apache.hadoop.hdfs.DFSOutputStream.<init>(DFSOutputStream.java:1626)
	  at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1679)
	  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1692)
	  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1627)
	  at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:503)
	  at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:499)
	  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:514)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:442)
	  at org.apache.hadoop.fs.FilterFileSystem.create(FilterFileSystem.java:179)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:979)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:960)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:857)
	  at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.create(CommandWithDestination.java:488)
	  at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:465)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:391)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:328)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:263)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:248)
	  at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
	  at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:291)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:243)
	  at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:273)
	  at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:257)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:220)
	  at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:267)
	  at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:203)
	  at org.apache.hadoop.fs.shell.Command.run(Command.java:167)
	  at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	  at cn.whbing.hadoop.ReadWriteTest.testWrite(ReadWriteTest.java:132)

创建了临时文件如：
/test/write/1219-5.xml.COPYING

DataStreamer --> 是一个守护线程类。

1.主线程创建DFSOutputStream

static DFSOutputStream newStreamForCreate(DFSClient dfsClient, String src,
      FsPermission masked, EnumSet<CreateFlag> flag, boolean createParent,
      short replication, long blockSize, Progressable progress, int buffersize,
      DataChecksum checksum, String[] favoredNodes)

在这里插入图片描述
其中 FSOutputSummer 继承 OutputStream，主要是添加了一个计算checksum的功能。DFSOutputStream继承FSOutputSummer。

主要的三个类：

  • DFSOutputStream
  • DataStreamer
  • ResponseProcessor

2

要把读写过程细节搞明白前，你必须知道block、packet与chunk

  • block
    block是最大的一个单位，它是最终存储于DataNode上的数据粒度，由dfs.block.size参数决定。文件上传前需要分块，这个块就是block，一般为128MB，当然你可以去改，不顾不推荐。因为块太小：寻址时间占比过高。块太大：Map任务数太少，作业执行速度变慢。它是最大的一个单位。
  • packet
    packet是第二大的单位，它是client端向DataNode，或DataNode的PipLine之间传数据的基本单位，默认64KB。
  • chunk
    chunk是最小的单位，它是client向DataNode，或DataNode的PipLine之间进行数据校验的基本单位，默认512Byte，因为用作校验，故每个chunk需要带有4Byte的校验位。所以实际每个chunk写入packet的大小为516Byte。由此可见真实数据与校验值数据的比值约为128 : 1。

例如，在client端向DataNode传数据的时候，HDFSOutputStream会有一个chunk buff，写满一个chunk后，会计算校验和并写入当前的chunk。之后再把带有校验和的chunk写入packet，当一个packet写满后，packet会进入dataQueue队列，其他的DataNode就是从这个dataQueue获取client端上传的数据并存储的。同时一个DataNode成功存储一个packet后之后会返回一个ack packet，放入ack Queue中。

一个比较好的图如下：
在这里插入图片描述
对他的解释：
写过程中的三层buffer
写过程中会以chunk、packet及packet queue三个粒度做三层缓存；

  • 首先，当数据流入DFSOutputStream时，DFSOutputStream内会有一个chunk大小的buf，当数据写满这个buf（或遇到强制flush），会计算checksum值，然后填塞进packet；
  • 当一个chunk填塞进入packet后，仍然不会立即发送，而是累积到一个packet填满后，将这个packet放入dataqueue队列；
  • 进入dataQueue队列的packet会被另一线程(DataSreamer)按序取出发送到datanode，发送出去后，会将该packet放入ackQueue中；只有当ResponseProcessor线程收到下游节点传过来的ack消息后，才会将数据包从ackQueue中移除。（注：生产者消费者模型，阻塞生产者的条件是dataqueue与ackqueue之和超过一个block的packet上限）

DFSOutStream中使用 DFSPacket 来包装数据。
DFSPacket:

 /**
   * buf is pointed into like follows:
   *  (C is checksum data, D is payload data)
   *
   * [_________CCCCCCCCC________________DDDDDDDDDDDDDDDD___]
   *           ^        ^               ^               ^
   *           |        checksumPos     dataStart       dataPos
   *           checksumStart
   *
   * Right before sending, we move the checksum data to immediately precede
   * the actual data, and then insert the header into the buffer immediately
   * preceding the checksum data, so we make sure to keep enough space in
   * front of the checksum data to support the largest conceivable header.
   */

在这里插入图片描述

DataStreamer线程

在这里插入图片描述
DataStreamer将数据包通过管道发送给datanode，并在当前数据块写满时，向NN申请分配新的数据块，然后更新管道。DataStream定义了管道的状态信息：

	// 数据块数据流管道中的datanode
	private volatile DatanodeInfo[] nodes = null; // list of targets for current block
	// 在dn上保存这个块的存储类型
    private volatile StorageType[] storageTypes = null;
    // 在dn上保存这个块的存储
    private volatile String[] storageIDs = null;
    // 数据块对应的数据流管道状态
    private BlockConstructionStage stage;  // block construction stage
...
    private void setPipeline(DatanodeInfo[] nodes, StorageType[] storageTypes,
        String[] storageIDs) {
      this.nodes = nodes;
      this.storageTypes = storageTypes;
      this.storageIDs = storageIDs;
    }

1.对于新写文件操作，DataStreamer会调用nextBlockOutputStream()方法向NN申请新的数据块，然后构建这个新的数据块的数据流管道。

          // get new block from namenode.
          if (stage == BlockConstructionStage.PIPELINE_SETUP_CREATE) {
            if(DFSClient.LOG.isDebugEnabled()) {
              DFSClient.LOG.debug("Allocating new block");
            }
            setPipeline(nextBlockOutputStream()); //新写数据块时，先调用next
            initDataStreaming(); // 再传输数据
          } else if (stage == BlockConstructionStage.PIPELINE_SETUP_APPEND) {
            if(DFSClient.LOG.isDebugEnabled()) {
              DFSClient.LOG.debug("Append to block " + block);
            }
            setupPipelineForAppendOrRecovery();
            initDataStreaming();
          }

2.对于append，会返回最后一个数据块的位置。根据最后一个数据块的位置信息初始化数据流管道。
3.成功构建管道后，会调用initDataStreaming将管道状态更改为DATA_STREAMING。然后就通过数据流管道发送数据。
在这里插入图片描述

PIPELINE整理

1. [HDFS写文件过程分析]

HDFS写文件过程分析

2. [Hadoop Pipeline详解]

Hadoop Pipeline详解

3. [再议HDFS写流程之pipeline]

再议HDFS写流程之pipeline

4.[HDFS源码分析（5）：datanode数据块的读与写]

HDFS源码分析（5）：datanode数据块的读与写

5.[Hhadoop-2.7.0中HDFS写文件源码分析（二）：客户端实现之DFSPacket]

Hhadoop-2.7.0中HDFS写文件源码分析（二）：客户端实现之DFSPacket

在这里插入图片描述

开启debug模式流程分析

hadoop fs -put xxx

...
19/12/23 09:11:46 DEBUG hdfs.DFSClient: /test/write/1.txt._COPYING_: masked=rw-r--r--
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #3
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #3
19/12/23 09:11:46 DEBUG ipc.ProtobufRpcEngine: Call: create took 13ms
19/12/23 09:11:46 DEBUG hdfs.DFSClient: computePacketChunkSize: src=/test/write/1.txt._COPYING_, chunkSize=516, chunksPerPacket=126, packetSize=65016
19/12/23 09:11:46 DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_-78804544_1] with renew id 1 started
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #4
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #4
19/12/23 09:11:46 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
19/12/23 09:11:46 DEBUG hdfs.DFSClient: DFSClient writeChunk allocating new packet seqno=0, src=/test/write/1.txt._COPYING_, packetSize=65016, chunksPerPacket=126, bytesCurBlock=0
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Queued packet 0
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Queued packet 1
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Allocating new block
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Waiting for ack for: 1
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #5
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #5
19/12/23 09:11:46 DEBUG ipc.ProtobufRpcEngine: Call: addBlock took 29ms
19/12/23 09:11:46 DEBUG hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[10.179.72.122:50010,DS-508839e9-efb9-417c-8aa9-fd7b5b78b688,DISK]
19/12/23 09:11:46 DEBUG hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[10.179.72.42:50010,DS-239d3012-2d81-4f66-9b08-77e9d21397d1,DISK]
19/12/23 09:11:46 DEBUG hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[10.179.16.227:50010,DS-286d8579-fe5f-484d-bb05-ae41787734e3,DISK]
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Connecting to datanode 10.179.72.122:50010
19/12/23 09:11:46 DEBUG hdfs.DFSClient: Send buf size 131072
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #6
19/12/23 09:11:46 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #6
19/12/23 09:11:46 DEBUG ipc.ProtobufRpcEngine: Call: getServerDefaults took 2ms
19/12/23 09:11:46 DEBUG sasl.SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /10.179.72.122, datanodeId = DatanodeInfoWithStorage[10.179.72.122:50010,DS-508839e9-efb9-417c-8aa9-fd7b5b78b688,DISK]
19/12/23 09:11:47 DEBUG hdfs.DFSClient: DataStreamer block BP-857948820-10.179.72.122-1560825265775:blk_1073747684_6863 sending packet packet seqno: 0 offsetInBlock: 0 lastPacketInBlock: false lastByteOffsetInBlock: 1114
19/12/23 09:11:47 DEBUG hdfs.DFSClient: DFSClient seqno: 0 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 545916 flag: 0 flag: 0 flag: 0
19/12/23 09:11:47 DEBUG hdfs.DFSClient: DataStreamer block BP-857948820-10.179.72.122-1560825265775:blk_1073747684_6863 sending packet packet seqno: 1 offsetInBlock: 1114 lastPacketInBlock: true lastByteOffsetInBlock: 1114
19/12/23 09:11:47 DEBUG hdfs.DFSClient: DFSClient seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 1111705 flag: 0 flag: 0 flag: 0
19/12/23 09:11:47 DEBUG hdfs.DFSClient: Closing old block BP-857948820-10.179.72.122-1560825265775:blk_1073747684_6863
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #7
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #7
19/12/23 09:11:47 DEBUG ipc.ProtobufRpcEngine: Call: complete took 11ms
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #8
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #8
19/12/23 09:11:47 DEBUG ipc.ProtobufRpcEngine: Call: rename took 13ms
19/12/23 09:11:47 DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@d737b89
19/12/23 09:11:47 DEBUG ipc.Client: removing client from cache: org.apache.hadoop.ipc.Client@d737b89
19/12/23 09:11:47 DEBUG ipc.Client: stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@d737b89
19/12/23 09:11:47 DEBUG ipc.Client: Stopping client
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop: closed
19/12/23 09:11:47 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop: stopped, remaining connections 0

注：上述debug打印了关键信息，下文会顺序分析；DFSClient的log不一定在该类中，可能是其他类中的调用。

核心流程

1.fs.create获得FSDataOutputStream，实质是DFSOutputStream。
中间借助DFSClient.create()方法进入：

public DFSOutputStream create(String src, 
                             FsPermission permission,
                             EnumSet<CreateFlag> flag, 
                             boolean createParent,
                             short replication,
                             long blockSize,
                             Progressable progress,
                             int buffersize,
                             ChecksumOpt checksumOpt,
                             InetSocketAddress[] favoredNodes) throws IOException {
    checkOpen();
    if (permission == null) {
      permission = FsPermission.getFileDefault();
    }
    FsPermission masked = permission.applyUMask(dfsClientConf.uMask);
    if(LOG.isDebugEnabled()) {
      LOG.debug(src + ": masked=" + masked); // 1 该出的debug，见上述调试
    }
    final DFSOutputStream result = DFSOutputStream.newStreamForCreate(this,
        src, masked, flag, createParent, replication, blockSize, progress,
        buffersize, dfsClientConf.createChecksum(checksumOpt),
        getFavoredNodesStr(favoredNodes));  // 2 获得Stream
    beginFileLease(result.getFileId(), result);  // 3 租约有关
    return result;
  }

注：该debug见上述打印，先建一个临时文件；并获得管道stream进行复杂操作；同时启动租约。

在这里插入图片描述
2.关键入口方法：DFSoutputStream.newStreamForCreate，通过构造器得到输出流。并同时启动输出流主线程。

static DFSOutputStream newStreamForCreate(DFSClient dfsClient, String src,
      FsPermission masked, EnumSet<CreateFlag> flag, boolean createParent,
      short replication, long blockSize, Progressable progress, int buffersize,
      DataChecksum checksum, String[] favoredNodes) throws IOException {
    TraceScope scope =
        dfsClient.getPathTraceScope("newStreamForCreate", src);
    try {
      HdfsFileStatus stat = null;

      // Retry the create if we get a RetryStartFileException up to a maximum
      // number of times
      boolean shouldRetry = true;
      int retryCount = CREATE_RETRY_COUNT;
      while (shouldRetry) {
        shouldRetry = false;
        try {
          stat = dfsClient.namenode.create(src, masked, dfsClient.clientName,
              new EnumSetWritable<CreateFlag>(flag), createParent, replication,
              blockSize, SUPPORTED_CRYPTO_VERSIONS);
          break;
        } catch (RemoteException re) {
          IOException e = re.unwrapRemoteException(
              AccessControlException.class,
              DSQuotaExceededException.class,
              FileAlreadyExistsException.class,
              FileNotFoundException.class,
              ParentNotDirectoryException.class,
              NSQuotaExceededException.class,
              RetryStartFileException.class,
              SafeModeException.class,
              UnresolvedPathException.class,
              SnapshotAccessControlException.class,
              UnknownCryptoProtocolVersionException.class);
          if (e instanceof RetryStartFileException) {
            if (retryCount > 0) {
              shouldRetry = true;
              retryCount--;
            } else {
              throw new IOException("Too many retries because of encryption" +
                  " zone operations", e);
            }
          } else {
            throw e;
          }
        }
      }
      Preconditions.checkNotNull(stat, "HdfsFileStatus should not be null!");
      final DFSOutputStream out = new DFSOutputStream(dfsClient, src, stat,
          flag, progress, checksum, favoredNodes);   // 获得输出流
      out.start();  // 启动输出流注线程
      return out;
    } finally {
      scope.close();
    }
  }

上述创建输出流分为三步：
（1）向NN发送请求，调用NN的create()方法（这块暂时不研究）；
（2）通过构造器构造出输出流；
（3）启动输出流线程

后续流程均在out.start()方法中启动。即DFSOutputStream.run()方法。
根据调用栈及debuglog、代码可知，在DFSOutputStream的构造器中，调用了computePacketChunkSize。
在这里插入图片描述
上述数值固定。即每个packet有126个chunk，每个chunk大小（data 512 + checksum 4=516），header占了33个字节。(计算的值是126.9那么这里到底是用126还是127呢？)
每个packet传输的data+checksum的字节为：126*516=65016字节。
每个packet传输的有效字节为：126*512=64512字节。

在这里插入图片描述
文本如下：

19/12/23 10:35:58 DEBUG hdfs.DFSClient: computePacketChunkSize: src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, chunkSize=516, chunksPerPacket=126, packetSize=65016
19/12/23 10:35:58 DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_-1334506554_1] with renew id 1 started
19/12/23 10:35:58 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop sending #4
19/12/23 10:35:58 DEBUG ipc.Client: IPC Client (1537471098) connection to cluster-host1/10.179.72.122:9000 from hadoop got value #4
19/12/23 10:35:58 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk allocating new packet seqno=0, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, packetSize=65016, chunksPerPacket=126, bytesCurBlock=0
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk packet full seqno=0, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, bytesCurBlock=64512, blockSize=134217728, appendChunk=false
19/12/23 10:35:58 DEBUG hdfs.DFSClient: Queued packet 0
19/12/23 10:35:58 DEBUG hdfs.DFSClient: computePacketChunkSize: src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, chunkSize=516, chunksPerPacket=126, packetSize=65016
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk allocating new packet seqno=1, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, packetSize=65016, chunksPerPacket=126, bytesCurBlock=64512
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk packet full seqno=1, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, bytesCurBlock=129024, blockSize=134217728, appendChunk=false
19/12/23 10:35:58 DEBUG hdfs.DFSClient: Queued packet 1
19/12/23 10:35:58 DEBUG hdfs.DFSClient: Allocating new block
19/12/23 10:35:58 DEBUG hdfs.DFSClient: computePacketChunkSize: src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, chunkSize=516, chunksPerPacket=126, packetSize=65016
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk allocating new packet seqno=2, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, packetSize=65016, chunksPerPacket=126, bytesCurBlock=129024
19/12/23 10:35:58 DEBUG hdfs.DFSClient: DFSClient writeChunk packet full seqno=2, src=/test/write/hadoop-2.7.6-2.tar.gz._COPYING_, bytesCurBlock=193536, blockSize=134217728, appendChunk=false

每次传64512个有效字节，然后再新分配一个packet。
继续看分配packet的代码，在writeChunkImpl后续分析。

在这里插入图片描述
那么谁在调用看下调用栈：

java.lang.Thread.State: RUNNABLE
	  at org.apache.hadoop.hdfs.DFSOutputStream.computePacketChunkSize(DFSOutputStream.java:1741)
	  at org.apache.hadoop.hdfs.DFSOutputStream.writeChunkImpl(DFSOutputStream.java:1876)
	  - locked <0xaeb> (a org.apache.hadoop.hdfs.DFSOutputStream)
	  at org.apache.hadoop.hdfs.DFSOutputStream.writeChunk(DFSOutputStream.java:1813)
	  at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:206)
	  at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:124)
	  at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:110)
	  at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
	  at java.io.DataOutputStream.write(DataOutputStream.java:107)
	  - locked <0xb29> (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream)
	  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:87)
	  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
	  at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
	  at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:466)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:391)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:328)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:263)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:248)
	  at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
	  at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:291)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:243)
	  at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:273)
	  at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:257)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:220)
	  at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:267)
	  at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:203)
	  at org.apache.hadoop.fs.shell.Command.run(Command.java:167)
	  at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	  at cn.whbing.hadoop.ReadWriteTest.testWrite(ReadWriteTest.java:133)

可以看到，调用FSOutputSummer.write，然后循环调用write1()。为什么这个线程栈和上述的不一样呢？我们将两个都列出来：

  java.lang.Thread.State: RUNNABLE
	  at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1679)
	  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1692)
	  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1627)
	  at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:503)
	  at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:499)
	  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:514)
	  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:442)
	  at org.apache.hadoop.fs.FilterFileSystem.create(FilterFileSystem.java:179)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:979)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:960)
	  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:857)
	  at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.create(CommandWithDestination.java:488)
	  at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:465)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:391)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:328)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:263)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:248)
	  at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
	  at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:291)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:243)
	  at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:273)
	  at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:257)
	  at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:220)
	  at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:267)
	  at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:203)
	  at org.apache.hadoop.fs.shell.Command.run(Command.java:167)
	  at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	  at cn.whbing.hadoop.ReadWriteTest.testWrite(ReadWriteTest.java:133)

可以看到在 at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:465) 时，走向了不同的方法。

    void writeStreamToFile(InputStream in, PathData target,
                           boolean lazyPersist) throws IOException {
      FSDataOutputStream out = null;
      try {
        out = create(target, lazyPersist); // 1
        IOUtils.copyBytes(in, out, getConf(), true); // 2
      } finally {
        IOUtils.closeStream(out); // just in case copyBytes didn't
      }
    }

分为两步，先创建一个流，再复制到目标流。

write中初次长度131072，write1每次写4608：
在这里插入图片描述

private synchronized void writeChunkImpl(byte[] b, int offset, int len,
          byte[] checksum, int ckoff, int cklen) throws IOException {
    dfsClient.checkOpen();
    checkClosed();

    if (len > bytesPerChecksum) {
      throw new IOException("writeChunk() buffer size is " + len +
                            " is larger than supported  bytesPerChecksum " +
                            bytesPerChecksum);
    }
    if (cklen != 0 && cklen != getChecksumSize()) {
      throw new IOException("writeChunk() checksum size is supposed to be " +
                            getChecksumSize() + " but found to be " + cklen);
    }

    // 1 分配一个新的packet
    if (currentPacket == null) {
      currentPacket = createPacket(packetSize, chunksPerPacket, 
          bytesCurBlock, currentSeqno++, false);
      if (DFSClient.LOG.isDebugEnabled()) {
        DFSClient.LOG.debug("DFSClient writeChunk allocating new packet seqno=" + 
            currentPacket.getSeqno() +
            ", src=" + src +
            ", packetSize=" + packetSize +
            ", chunksPerPacket=" + chunksPerPacket +
            ", bytesCurBlock=" + bytesCurBlock);
      }
    }

    currentPacket.writeChecksum(checksum, ckoff, cklen);
    currentPacket.writeData(b, offset, len);
    currentPacket.incNumChunks();
    bytesCurBlock += len;

    // If packet is full, enqueue it for transmission
    //
    if (currentPacket.getNumChunks() == currentPacket.getMaxChunks() ||
        bytesCurBlock == blockSize) {
      if (DFSClient.LOG.isDebugEnabled()) {
        DFSClient.LOG.debug("DFSClient writeChunk packet full seqno=" +
            currentPacket.getSeqno() +
            ", src=" + src +
            ", bytesCurBlock=" + bytesCurBlock +
            ", blockSize=" + blockSize +
            ", appendChunk=" + appendChunk);
      }
      waitAndQueueCurrentPacket();

      // If the reopened file did not end at chunk boundary and the above
      // write filled up its partial chunk. Tell the summer to generate full 
      // crc chunks from now on.
      if (appendChunk && bytesCurBlock%bytesPerChecksum == 0) {
        appendChunk = false;
        resetChecksumBufSize();
      }

      if (!appendChunk) {
        int psize = Math.min((int)(blockSize-bytesCurBlock), dfsClient.getConf().writePacketSize);
        computePacketChunkSize(psize, bytesPerChecksum);
      }
      //
      // if encountering a block boundary, send an empty packet to 
      // indicate the end of block and reset bytesCurBlock.
      //
      if (bytesCurBlock == blockSize) {
        currentPacket = createPacket(0, 0, bytesCurBlock, currentSeqno++, true);
        currentPacket.setSyncBlock(shouldSyncBlock);
        waitAndQueueCurrentPacket();
        bytesCurBlock = 0;
        lastFlushOffset = 0;
      }
    }
  }

1.分配一个新的packet，createPacket
2.往这个packet中写checum、data：

	currentPacket.writeChecksum(checksum, ckoff, cklen);
    currentPacket.writeData(b, offset, len);
    currentPacket.incNumChunks();
    bytesCurBlock += len;

在这里插入图片描述

  private void waitAndQueueCurrentPacket() throws IOException {
    synchronized (dataQueue) {
      try {
      // If queue is full, then wait till we have enough space
        boolean firstWait = true;
        try {
          while (!isClosed() && dataQueue.size() + ackQueue.size() >
              dfsClient.getConf().writeMaxPackets) {
            if (firstWait) {
              Span span = Trace.currentSpan();
              if (span != null) {
                span.addTimelineAnnotation("dataQueue.wait");
              }
              firstWait = false;
            }
            try {
              dataQueue.wait();
            } catch (InterruptedException e) {
              // If we get interrupted while waiting to queue data, we still need to get rid
              // of the current packet. This is because we have an invariant that if
              // currentPacket gets full, it will get queued before the next writeChunk.
              //
              // Rather than wait around for space in the queue, we should instead try to
              // return to the caller as soon as possible, even though we slightly overrun
              // the MAX_PACKETS length.
              Thread.currentThread().interrupt();
              break;
            }
          }
        } finally {
          Span span = Trace.currentSpan();
          if ((span != null) && (!firstWait)) {
            span.addTimelineAnnotation("end.wait");
          }
        }
        checkClosed();
        queueCurrentPacket();
      } catch (ClosedChannelException e) {
      }
    }
  }
  
  private void queueCurrentPacket() {
    synchronized (dataQueue) {
      if (currentPacket == null) return;
      currentPacket.addTraceParent(Trace.currentSpan());
      dataQueue.addLast(currentPacket);
      lastQueuedSeqno = currentPacket.getSeqno();
      if (DFSClient.LOG.isDebugEnabled()) {
        DFSClient.LOG.debug("Queued packet " + currentPacket.getSeqno());
      }
      currentPacket = null;
      dataQueue.notifyAll();
    }
  }

writeMaxPackets 默认80，dataQueue + ackQueue > 80时就要等待。

在这里插入图片描述
1.使用nextBlockOutputStream向集群申请分配新的数据块。然后构造这个数据块的pipeline。
在这里插入图片描述

run(){
	...
	// get new block from namenode.
    if (stage == BlockConstructionStage.PIPELINE_SETUP_CREATE) {
          if(DFSClient.LOG.isDebugEnabled()) {
              DFSClient.LOG.debug("Allocating new block");
            }
            setPipeline(nextBlockOutputStream());
            initDataStreaming();
          } else if ...
	...
}

上述三个方法如下：

  • nextBlockOutputStream: 向NN申请分配新的数据块。
private LocatedBlock nextBlockOutputStream(){
		...
		// 故障节点
        DatanodeInfo[] excluded =
            excludedNodes.getAllPresent(excludedNodes.asMap().keySet())
            .keySet()
            .toArray(new DatanodeInfo[0]);
        ...  
        // 创建到第一个datanode的输出流 
        //
        // Connect to first DataNode in the list.
        //
        success = createBlockOutputStream(nodes, storageTypes, 0L, false);
        // 如果不成功，则放弃块
        if (!success) {
          DFSClient.LOG.info("Abandoning " + block);
          // 将dadanode放入故障节点中
          dfsClient.namenode.abandonBlock(block, fileId, src,
              dfsClient.clientName);
          block = null;
          DFSClient.LOG.info("Excluding datanode " + nodes[errorIndex]);
          excludedNodes.put(nodes[errorIndex], nodes[errorIndex]);
        }
      } while (!success && --count >= 0); // 进行重试
}

2.获取到数据管道中的第一个datanode的IO流，然后通过这个IO流发送数据包。

			// get packet to be sent.
			// 取出一个包，或者构造一个心跳包
            if (dataQueue.isEmpty()) {
              one = createHeartbeatPacket(); 
              assert one != null;
            } else {
              one = dataQueue.getFirst(); // regular data packet
              long parents[] = one.getTraceParents();
              if (parents.length > 0) {
                scope = Trace.startSpan("dataStreamer", new TraceInfo(0, parents[0]));
                // TODO: use setParents API once it's available from HTrace 3.2
//                scope = Trace.startSpan("dataStreamer", Sampler.ALWAYS);
//                scope.getSpan().setParents(parents);
              }
            }

从dataQueue中取出一个待发送的packet，如果是最后一个，则构造一个空的心跳包。
在这里插入图片描述
在这里插入图片描述
心跳包与最后一个包？
A:心跳包一定不是最后一个包。
（四）心跳包
如果长时间没有数据传输，在输出流未关闭的情况下，客户端会发送心跳包给数据节点，心跳包是DataPacket的一种特殊实现，它通过数据包序列号为-1来进行特殊标识，如下：

  /**
   * Check if this packet is a heart beat packet
   * 判断该包释放为心跳包
   *
   * @return true if the sequence number is HEART_BEAT_SEQNO
   */
  boolean isHeartbeatPacket() {
	// 心跳包的序列号均为-1
    return seqno == HEART_BEAT_SEQNO;
  }
            而心跳包的构造如下：
  /**
   * For heartbeat packets, create buffer directly by new byte[]
   * since heartbeats should not be blocked.
   */
  private DFSPacket createHeartbeatPacket() throws InterruptedIOException {
    final byte[] buf = new byte[PacketHeader.PKT_MAX_HEADER_LEN];
    return new DFSPacket(buf, 0, 0, DFSPacket.HEART_BEAT_SEQNO,
                         getChecksumSize(), false);
  }

在这里插入图片描述
在这里插入图片描述
queue到达80个的时候，不能再往队列中加了。必须要发送了。
上述写了一个块：DataStreamer block BP-857948820-10.179.72.122-1560825265775:blk_1073747689_6868

在这里插入图片描述
在这里插入图片描述

hdfs源码解析之DFSClient
上好佳的博客
06-19 769
该构造函数已废弃，接受一个Configuration对象，并调用另一个构造函数获取NameNode地址该构造函数接受一个InetSocketAddress对象和一个Configuration对象，并将InetSocketAddress 转换为URI然后调用另一个基于URI的构造函数该构造函数接受一个URI对象和一个Configuration对象，并将FileSystem.Statistics参数设置为 null，然后调用另一个更完整的构造函数。
eclipse操作HDFS时异常的解决: Permission denied: user=administrator, access=WRITE,
ForestCat的专栏
12-20 808
在虚拟机上搭建了一个Hadoop的环境（4节点），在用windows上的Java客户端来操作集群中的HDFS文件时出现了如下的认证错误 org.apache.hadoop.security.AccessControlException: Permission denied: user=Administrator, access=WRITE, inode="/input4":root:...
Hadoop学习九：hdfs源代码调试
youfashion的博客
04-13 1727
Hadoop学习九：hdfs源代码调试标签（空格分隔）： hadoopHadoop学习九hdfs源代码调试 一利用log4j调试 一，利用log4j调试1,在eclipse工程里面创建source folder 2,拷贝log4j.properties文件[hadoop002@hadoop002 hadoop]$ cp log4j.properties /home/hadoop002/worksp
windows下使用hdfs调试
hyd19931002的博客
03-04 292
配置环境变量HADOOP_HOME=hadoop安装解压目录 PATH=%HADOOP_HOME%\bin hadoop的bin目录下放置winutils.exe和hadoop.dll 2个文件,要不然报找不到winutils.exe. 2个文件下载地址.版本2.7.2https://download.csdn.net/download/hyd19931002/15563574 以下为使用的pom.xml文件 <?xml version="1.0" encoding="UTF-8"...
HDFS 客户端操作教程
最新发布
xiaoyu070321的博客
05-06 525
类代码，详细说明 HDFS 文件系统常见操作。代码通过 Hadoop 的。API 实现文件上传、下载、元数据查看等功能。路径权限不足或本地目录不存在时会抛出。释放连接资源，推荐使用。
2. HDFS CLINT WRITE整理版
王小禾
02-23 430
1.概述 通过fsShell或api在客户端断点调试，可以知道写文件入口DistributedFileSystem.create()。 "main@1"prio=5tid=0x1nid=NA runnable java.lang.Thread.State: RUNNABLE at org.apache.hadoop.hdfs.DFSOutputStre...
HDFS client 实例
_Hope_
04-19 2114
package hdfs; import java.io.IOException; import java.net.URI; import java.net.URISyntaxException;import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataOutputStream; import org
python hdfs.client
qq_40771567的博客
04-28 4945
前言 HDFS主要用来存储文件系统 安装 pip install hdfs 使用 from hdfs.client import Client client = Client(HDFSHOST,root="/",timeout=900,session=False) #连接hdfs 参数说明: classhdfs.Client(url, root=None, proxy=No...
大数据实验（二）HDFS API的使用（Python 3.7）
while0l1的博客
12-15 1665
前言 书本第六章的主要内容就是讲了HDFS的一些操作指令，还有非常简略的Java调用HDFS API代码的示例。虽然据说用Java的运行效率会高很多，可是我也没有系统地学过Java，而且看样子实现做起来比较繁琐，所以我还是选择了Python来实现HDFS API的调用。用Python特别好的地方就是后面的数据处理和可视化会方便很多，人生苦短，我用Python，哈哈哈哈哈哈哈哈。 HDFS API的...
HDFS的优缺点和读、写数据流程详解
weixin_43230682的博客
08-06 1926
目录 一、hadoop的运行模式 1. 本地运行模式 2. 伪分布式运行模式 3. 完全分布式运行模式（开发重点） 二、hdfs的优缺点 1. hdfs的优点 2. hdfs的缺点 三、hdfs的读写流程 1. hdfs的写入流程 2. hdfs的读取流程 一、hadoop的运行模式 1. 本地运行模式 无需任何守护进程，所有的程序都运行在同一个JVM上执行。在独立模式下调试MR程序非常高效方便。所以一般该模式主要是在学习或者开发阶段调试使用 2. 伪分布式运行模式
在eclipse中编写HDFS的Java程序
qq_44240521的博客
05-26 1498
1.开始我们需要配置jdk+eclipse+maven,文章参考链接https://blog.csdn.net/qq_44240521/article/details/90574089 2.在Eclipse里新建Maven Project 【file】-【new】-【Maven project】 如下图自动生成Maven项目，新建包hdfs.files，包里新建class：4个java文件。 ...
HDFS的java客户端操作
weixin_45102492的博客
02-18 415
HDFS的Java客户端操作 HDFS客户端环境准备 详见： https://blog.csdn.net/weixin_45102492/article/details/103008541 创建一个Maven工程HdfsClientDemo,导入相应的依赖坐标+日志添加 <dependencies> <dependency> <groupId>juni...
hdfs client api
qq_38250124的博客
03-23 360
public class HdfsClient { FileSystem fs = null; @Before public void init() throws Exception { Configuration conf = new Configuration(); // conf.set("fs.defaultFS","hdfs://m
HDFS client写文件过程 源码分析
thaddeuslu的博客
10-12 395
HDFS client写文件过程 源码分析 HDFS写入文件的重要概念 HDFS一个文件由多个block构成。HDFS在进行block读写的时候是以packet(默认每个packet为64K)为单位进行的。每一个packet由若干个chunk（默认512Byte）组成。Chunk是进行数据校验的基本单位，对每一个chunk生成一个校验和(默认4Byte)并将校验和进行存储。 在写入一个block的...
ESP8266作数据转发client.write的使用坑
yuanqia4079的博客
04-05 3950
ESP8266接收从STM32送来的串口浮点数据，转发给服务器。使用这一段开启client。（ticker的detach可以解除关联的函数。让函数不再定时运行。ticker是一个实例对象，它只能唯一关联一个函数。所以可以这么操作）while (!client.connect(host, httpPort)) {   ticker.detach();//常亮不闪说明没连上服务器   digitalW...
log4j打印sql
Y_JY
09-28 544
log4j.rootLogger=DEBUG,stdout   log4j.logger.java.sql.PreparedStatement=DEBUG,stdout      log4j.appender.stdout=org.apache.log4j.ConsoleAppender   log4j.appender.stdout.layout=org.apache.log4j.Pat
使用python的hdfs包操作HDFS
热门推荐
jiede1的博客
04-05 1万+
项目需要利用python访问hdfs，查到可以利用hdfs包实现这一点，总结了下使用方法。 1.安装 我在Linux环境下安装 pip install hdfs 2.连接hdfs from hdfs.client import Client # 50070: NameNode web管理端口 client = Client("http://localhost:50070") ...
HDFS中不起作用的buffer size
渴死的火烈鸟
05-12 387
/** * Create a new output stream to the given DataNode. * @see ClientProtocol#create(String, FsPermission, String, boolean, short, long) */ DFSOutputStream(String src, FsPermis...
HDFS租约解析
半日闲的博客
03-28 4269
转自：http://bigdatadecode.club/HDFS%E7%A7%9F%E7%BA%A6%E8%A7%A3%E6%9E%90.html 租约(Lease)是一种广泛应用与分布式系统领域的协议，主要用来维护分布式系统的一致性。 租约是在解决缓存一致性时被提出的。所谓租约，其实就是一个合同，即服务器给予客户端在一定期限内可以控制修改操作的权力。如果服务器要修改数据，首先要征求拥有
pycharm安装hdfs
02-25
这一步骤确保了开发环境中有足够的工具来与 HDFS 进行交互[^1]。 #### 配置 PyCharm 的解释器设置 打开 PyCharm 后，在项目设置中的 "Project Interpreter" 下找到当前使用的 Python 解释器，并确认刚才安装的 `...
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值