TestDFSIO测试集群性能

最新推荐文章于 2024-07-25 11:42:13 发布
原创 最新推荐文章于 2024-07-25 11:42:13 发布 · 7k 阅读 · 1 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。

本文记录了使用Hadoop进行不同大小文件的读写性能测试过程及结果，包括100MB和1GB的数据处理情况，并详细展示了测试过程中涉及的各项性能指标。

hadoop jar   hadoop-mapreduce-client-jobclient-2.2.0-tests.jar   TestDFSIO -write -size 108KB


hadoop jar   hadoop-mapreduce-client-jobclient-2.2.0-tests.jar   TestDFSIO -read -size 100B


16/03/05 21:19:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/03/05 21:19:46 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 1 files
16/03/05 21:19:46 INFO fs.TestDFSIO: created control files for: 1 files
16/03/05 21:19:46 INFO client.RMProxy: Connecting to ResourceManager at /192.168.0.123:8032
16/03/05 21:19:46 INFO client.RMProxy: Connecting to ResourceManager at /192.168.0.123:8032
16/03/05 21:19:47 INFO mapred.FileInputFormat: Total input paths to process : 1
16/03/05 21:19:47 INFO mapreduce.JobSubmitter: number of splits:1
16/03/05 21:19:47 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
16/03/05 21:19:47 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
16/03/05 21:19:47 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
16/03/05 21:19:47 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
16/03/05 21:19:47 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
16/03/05 21:19:47 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/03/05 21:19:47 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
16/03/05 21:19:47 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
16/03/05 21:19:47 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
16/03/05 21:19:47 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
16/03/05 21:19:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1457182542695_0004
16/03/05 21:19:47 INFO impl.YarnClientImpl: Submitted application application_1457182542695_0004 to ResourceManager at /192.168.0.123:8032
16/03/05 21:19:47 INFO mapreduce.Job: The url to track the job: http://192.168.0.123:8088/proxy/application_1457182542695_0004/
16/03/05 21:19:47 INFO mapreduce.Job: Running job: job_1457182542695_0004
16/03/05 21:19:59 INFO mapreduce.Job: Job job_1457182542695_0004 running in uber mode : false
16/03/05 21:19:59 INFO mapreduce.Job:  map 0% reduce 0%
16/03/05 21:20:05 INFO mapreduce.Job:  map 100% reduce 0%
16/03/05 21:20:12 INFO mapreduce.Job:  map 100% reduce 100%
16/03/05 21:20:18 INFO mapreduce.Job: Job job_1457182542695_0004 completed successfully
16/03/05 21:20:18 INFO mapreduce.Job: Counters: 43
        File System Counters
                FILE: Number of bytes read=90
                FILE: Number of bytes written=160769
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=239
                HDFS: Number of bytes written=104857674
                HDFS: Number of read operations=7
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=3
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=4794
                Total time spent by all reduces in occupied slots (ms)=2916
        Map-Reduce Framework
                Map input records=1
                Map output records=5
                Map output bytes=74
                Map output materialized bytes=90
                Input split bytes=127
                Combine input records=0
                Combine output records=0
                Reduce input groups=5
                Reduce shuffle bytes=90
                Reduce input records=5
                Reduce output records=5
                Spilled Records=10
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=53
                CPU time spent (ms)=3380
                Physical memory (bytes) snapshot=595660800
                Virtual memory (bytes) snapshot=2552078336
                Total committed heap usage (bytes)=1008730112
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=112
        File Output Format Counters 
                Bytes Written=74
16/03/05 21:20:18 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/03/05 21:20:18 INFO fs.TestDFSIO:            Date & time: Sat Mar 05 21:20:18 CST 2016
16/03/05 21:20:18 INFO fs.TestDFSIO:        Number of files: 1
16/03/05 21:20:18 INFO fs.TestDFSIO: Total MBytes processed: 100.0
16/03/05 21:20:18 INFO fs.TestDFSIO:      Throughput mb/sec: 55.370985603543744
16/03/05 21:20:18 INFO fs.TestDFSIO: Average IO rate mb/sec: 55.37098693847656
16/03/05 21:20:18 INFO fs.TestDFSIO:  IO rate std deviation: 0.009722219803739806

16/03/05 21:20:18 INFO fs.TestDFSIO:     Test exec time sec: 31.687


以上是写100MB


以下是读100MB


uard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
16/03/05 21:24:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/03/05 21:24:42 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 1 files
16/03/05 21:24:42 INFO fs.TestDFSIO: created control files for: 1 files
16/03/05 21:24:42 INFO client.RMProxy: Connecting to ResourceManager at /192.168.0.123:8032
16/03/05 21:24:43 INFO client.RMProxy: Connecting to ResourceManager at /192.168.0.123:8032
16/03/05 21:24:43 INFO mapred.FileInputFormat: Total input paths to process : 1
16/03/05 21:24:43 INFO mapreduce.JobSubmitter: number of splits:1
16/03/05 21:24:43 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
16/03/05 21:24:43 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
16/03/05 21:24:43 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
16/03/05 21:24:43 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
16/03/05 21:24:43 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
16/03/05 21:24:43 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/03/05 21:24:43 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
16/03/05 21:24:43 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
16/03/05 21:24:43 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
16/03/05 21:24:43 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
16/03/05 21:24:43 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1457182542695_0006
16/03/05 21:24:44 INFO impl.YarnClientImpl: Submitted application application_1457182542695_0006 to ResourceManager at /192.168.0.123:8032
16/03/05 21:24:44 INFO mapreduce.Job: The url to track the job: http://192.168.0.123:8088/proxy/application_1457182542695_0006/
16/03/05 21:24:44 INFO mapreduce.Job: Running job: job_1457182542695_0006
16/03/05 21:24:51 INFO mapreduce.Job: Job job_1457182542695_0006 running in uber mode : false
16/03/05 21:24:51 INFO mapreduce.Job:  map 0% reduce 0%
16/03/05 21:24:57 INFO mapreduce.Job:  map 100% reduce 0%
16/03/05 21:25:03 INFO mapreduce.Job:  map 100% reduce 100%
16/03/05 21:25:04 INFO mapreduce.Job: Job job_1457182542695_0006 completed successfully
16/03/05 21:25:04 INFO mapreduce.Job: Counters: 43
        File System Counters
                FILE: Number of bytes read=90
                FILE: Number of bytes written=160765
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=104857839
                HDFS: Number of bytes written=74
                HDFS: Number of read operations=8
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Rack-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=4716
                Total time spent by all reduces in occupied slots (ms)=2953
        Map-Reduce Framework
                Map input records=1
                Map output records=5
                Map output bytes=74
                Map output materialized bytes=90
                Input split bytes=127
                Combine input records=0
                Combine output records=0
                Reduce input groups=5
                Reduce shuffle bytes=90
                Reduce input records=5
                Reduce output records=5
                Spilled Records=10
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=28
                CPU time spent (ms)=2210
                Physical memory (bytes) snapshot=578883584
                Virtual memory (bytes) snapshot=2543214592
                Total committed heap usage (bytes)=1008730112
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=112
        File Output Format Counters 
                Bytes Written=74
16/03/05 21:25:04 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
16/03/05 21:25:04 INFO fs.TestDFSIO:            Date & time: Sat Mar 05 21:25:04 CST 2016
16/03/05 21:25:04 INFO fs.TestDFSIO:        Number of files: 1
16/03/05 21:25:04 INFO fs.TestDFSIO: Total MBytes processed: 100.0
16/03/05 21:25:04 INFO fs.TestDFSIO:      Throughput mb/sec: 99.30486593843098
16/03/05 21:25:04 INFO fs.TestDFSIO: Average IO rate mb/sec: 99.30486297607422
16/03/05 21:25:04 INFO fs.TestDFSIO:  IO rate std deviation: 0.01562019922593262
16/03/05 21:25:04 INFO fs.TestDFSIO:     Test exec time sec: 21.8
16/03/05 21:25:04 INFO fs.TestDFSIO: 




读1GB数据:



16/03/05 21:25:55 INFO mapreduce.Job: Job job_1457182542695_0007 running in uber mode : false
16/03/05 21:25:55 INFO mapreduce.Job:  map 0% reduce 0%
16/03/05 21:26:02 INFO mapreduce.Job:  map 100% reduce 0%
16/03/05 21:26:09 INFO mapreduce.Job:  map 100% reduce 100%
16/03/05 21:26:14 INFO mapreduce.Job: Job job_1457182542695_0007 completed successfully
16/03/05 21:26:14 INFO mapreduce.Job: Counters: 43
        File System Counters
                FILE: Number of bytes read=91
                FILE: Number of bytes written=160767
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=104857839
                HDFS: Number of bytes written=75
                HDFS: Number of read operations=8
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Rack-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=3849
                Total time spent by all reduces in occupied slots (ms)=3647
        Map-Reduce Framework
                Map input records=1
                Map output records=5
                Map output bytes=75
                Map output materialized bytes=91
                Input split bytes=127
                Combine input records=0
                Combine output records=0
                Reduce input groups=5
                Reduce shuffle bytes=91
                Reduce input records=5
                Reduce output records=5
                Spilled Records=10
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=26
                CPU time spent (ms)=2190
                Physical memory (bytes) snapshot=572350464
                Virtual memory (bytes) snapshot=2537316352
                Total committed heap usage (bytes)=1008730112
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=112
        File Output Format Counters 
                Bytes Written=75
16/03/05 21:26:14 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
16/03/05 21:26:14 INFO fs.TestDFSIO:            Date & time: Sat Mar 05 21:26:14 CST 2016
16/03/05 21:26:14 INFO fs.TestDFSIO:        Number of files: 1
16/03/05 21:26:14 INFO fs.TestDFSIO: Total MBytes processed: 100.0
16/03/05 21:26:14 INFO fs.TestDFSIO:      Throughput mb/sec: 101.01010101010101
16/03/05 21:26:14 INFO fs.TestDFSIO: Average IO rate mb/sec: 101.01010131835938
16/03/05 21:26:14 INFO fs.TestDFSIO:  IO rate std deviation: 0.02114898989674338
16/03/05 21:26:14 INFO fs.TestDFSIO:     Test exec time sec: 27.846



写1GB数据:


It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
16/03/05 21:27:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/03/05 21:27:38 INFO fs.TestDFSIO: creating control file: 1048576000 bytes, 1 files
16/03/05 21:27:39 INFO fs.TestDFSIO: created control files for: 1 files
16/03/05 21:27:39 INFO client.RMProxy: Connecting to ResourceManager at /192.168.0.123:8032
16/03/05 21:27:39 INFO client.RMProxy: Connecting to ResourceManager at /192.168.0.123:8032
16/03/05 21:27:39 INFO mapred.FileInputFormat: Total input paths to process : 1
16/03/05 21:27:40 INFO mapreduce.JobSubmitter: number of splits:1
16/03/05 21:27:40 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
16/03/05 21:27:40 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
16/03/05 21:27:40 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
16/03/05 21:27:40 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
16/03/05 21:27:40 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
16/03/05 21:27:40 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/03/05 21:27:40 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
16/03/05 21:27:40 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
16/03/05 21:27:40 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
16/03/05 21:27:40 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
16/03/05 21:27:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1457182542695_0008
16/03/05 21:27:40 INFO impl.YarnClientImpl: Submitted application application_1457182542695_0008 to ResourceManager at /192.168.0.123:8032
16/03/05 21:27:40 INFO mapreduce.Job: The url to track the job: http://192.168.0.123:8088/proxy/application_1457182542695_0008/
16/03/05 21:27:40 INFO mapreduce.Job: Running job: job_1457182542695_0008
16/03/05 21:27:46 INFO mapreduce.Job: Job job_1457182542695_0008 running in uber mode : false
16/03/05 21:27:46 INFO mapreduce.Job:  map 0% reduce 0%
16/03/05 21:27:57 INFO mapreduce.Job:  map 67% reduce 0%
16/03/05 21:28:07 INFO mapreduce.Job:  map 100% reduce 0%
16/03/05 21:28:15 INFO mapreduce.Job:  map 100% reduce 100%
16/03/05 21:28:20 INFO mapreduce.Job: Job job_1457182542695_0008 completed successfully
16/03/05 21:28:20 INFO mapreduce.Job: Counters: 43
        File System Counters
                FILE: Number of bytes read=93
                FILE: Number of bytes written=160775
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=239
                HDFS: Number of bytes written=1048576077
                HDFS: Number of read operations=7
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=3
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Rack-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=18431
                Total time spent by all reduces in occupied slots (ms)=3719
        Map-Reduce Framework
                Map input records=1
                Map output records=5
                Map output bytes=77
                Map output materialized bytes=93
                Input split bytes=127
                Combine input records=0
                Combine output records=0
                Reduce input groups=5
                Reduce shuffle bytes=93
                Reduce input records=5
                Reduce output records=5
                Spilled Records=10
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=133
                CPU time spent (ms)=15410
                Physical memory (bytes) snapshot=570277888
                Virtual memory (bytes) snapshot=2536431616
                Total committed heap usage (bytes)=979369984
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=112
        File Output Format Counters 
                Bytes Written=77
16/03/05 21:28:20 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
16/03/05 21:28:20 INFO fs.TestDFSIO:            Date & time: Sat Mar 05 21:28:20 CST 2016
16/03/05 21:28:20 INFO fs.TestDFSIO:        Number of files: 1
16/03/05 21:28:20 INFO fs.TestDFSIO: Total MBytes processed: 1000.0
16/03/05 21:28:20 INFO fs.TestDFSIO:      Throughput mb/sec: 63.94270733422853
16/03/05 21:28:20 INFO fs.TestDFSIO: Average IO rate mb/sec: 63.94270706176758
16/03/05 21:28:20 INFO fs.TestDFSIO:  IO rate std deviation: 0.010423658009744381
16/03/05 21:28:20 INFO fs.TestDFSIO:     Test exec time sec: 40.966


飞火流云
关注
Hadoop【环境搭建 05】【hadoop-3.1.3 单机版基准测试 TestDFSIO + mrbench + nnbench + Terasort + sort 举例】
シ風
08-15 1558
利用hadoop自带基准测试工具包进行集群性能测试，目录 /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar 使用TestDFSIO、mrbench、nnbench、Terasort 、sort 几个使用较广的基准测试程序 # 不带参数运行,会显示示例说明 hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hado
【Hadoop-HDFS压测】针对HDFS进行读写性能测试
weixin_53543905的博客
03-10 1705
目录下，如果是CDH版本安装的Hadoop需要自己去对应目录下找。1、如果是Apache版本安装的Hadoop默认在。：总数据量（文件数量×单个文件大小）：总数据量（文件数量×单个文件大小），所以需要在服务器中配置。2、此工具测试需要执行。Hadoop环境变量。【Hadoop-HDFS压测】针对HDFS进行读写性能测试
压测实操--TestDFSIO压测hdfs读写方案
07-25 722
在集群硬件资源能给到最大条件下（比如namenode最大能给到8G，再大就会影响其他组件的内存使用），结合并行文件和文件大小对hdfs的i/o进行测试，需要重点关注io速率、rpc、namenode同步和集群是否告警情况。也可以将执行命令得到的io rate和throughput进行整理，形成曲线图查看趋势。更多技术信息请查看云掣官网t=yrgw。
TestDFSIO基准测试HDFS
weixin_30355437的博客
05-23 2276
TestDFSIO用来测试HDFS的I/O性能，它通过使用MapReduce作业来完成测试作为并行读写文件的便捷方法。以下命令写了10个文件，每个文件10MB:hadoop jar /opt/app/hadoop-2.3.0-cdh5.1.0/share/hadoop/mapreduce/hadoopmapreduce-client-jobclient-2.3.0-cdh5.1.0-tests.j...
HDFS源码解析---TestDFSIO
热门推荐
ssjdoudou的博客
12-09 1万+
这个工具怎么用没啥好说的，里面的原理简单说一下。 MapReduce 这个工具会起mapreduce 任务在yarn的各个节点（客户端）发起读写请求。TestDFSIO.java的run方法会调用mapreduce作业任务。 @Override // Tool public int run(String[] args) throws IOException { TestType testType = null; int buff...
CDH6.0.1环境Hadoop集群性能测试
L, there!
07-31 5073
本次主要是对新集群的性能测试，测试环境： 系统 环境 台数 CPU 内存 硬盘 Centos7.3 CDH6.0.1 3台 32 64G 16T Hadoop自带的测试工具：Hadoop基准测试 首先，对服务器做一下测试，主要是磁盘I/O和网络I/O。 磁盘I/O 读测试 hdparm用于查看硬盘的相关信息或对硬...
用TestDFSIO基准测试，来测试HDFS的IO性能
最新发布
03-11
好的，我需要帮助用户使用TestDFSIO来测试HDFS的IO性能。首先，我得回忆一下TestDFSIO的基本用法和相关参数。根据用户提供的引用内容，引用[1]中提到了写入测试的命令，包括指定文件数量、每个文件的大小以及输出的...
Hadoop 分布式部署配置及基准性能测试
12-08
本文将详细介绍如何在Hadoop v3.3.3版本下进行分布式集群的部署，并进行DFSIO和Tersort基准测试，以评估系统的性能。 首先，我们来看硬件环境的要求。为了建立一个高效的Hadoop集群，至少需要两台服务器，这里分别...
Hadoop3.0分布式集群性能测试与分析
"这篇文档是关于使用Hadoop 3.0构建完全分布式集群的实践报告，涉及Hadoop的性能测试，特别是在湖北大学的背景下。实验使用4台华为云服务器，运行CentOS7.4操作系统，Java 1.8.0_191，以及Hadoop 3.0.0版本。集群...
hadoop使用内置包进行性能测试TestDFSIO、NNBench、MRBench、SliveTest
qq_44229840的博客
12-14 3362
TestDFSIO/NNBench/MRBench/SliveTest
Hadoop之TestDFSIO
smallnetvisitor的博客
04-13 3297
概述： TestDFSIO为测试HDFS的IO性能，通过MR进行并发读写并做结果的统计。主要涉及，读，随机读，写，追加写，清空测试环境 使用： 清空： bin/hadoop jar share/hadoop/mapreduceX/hadoop-test-XXX.jar TestDFSIO -clean  写： bin/hadoop jar share/hadoop/mapreduc...
TestDFSIO性能测试工具使用方法
Erik_ly的博客
06-03 4479
本文主要介绍 HDFS 性能测试工具的功能、参数说明、读写性能测试使用方法及结果分析。
【Hadoop集群系列-4】使用TestDFSIO测试Hadoop-2.7.7集群性能
lynne_cat的博客
11-19 960
一、目的： 使用TestDFSIO测试hadoop-2.7.7集群性能。 二、环境： 2台虚拟机，CentOS Linux release 7.5.1804 (Core)，内存3G，硬盘45G。 192.168.10.156 hmaster156 192.168.10.162 hslave162 三、测试过程： 3.1 测试读写性能 在任意节点（hmaster156、hslave1...
HADOOP集群吞吐性能测试
sky924102334的博客
03-22 371
性能测试前，请注意以下事项。
集群测试工具
qq_17200461的博客
01-21 9970
DFSIO测试集群I/O性能 DFSIO 用法 hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.X-tests.jar TestDFSIO 15/05/22 19:50:22 INFO fs.TestDFSIO: TestDFSIO.1.8 Missing arguments. Usage: TestDFSIO [genericOptions] -read [-random |
hdfs读写性能测试
yy的博客
03-18 801
HDFS的IO吞吐性能可以通过TestDFSIO进行测试。TestDFSIO是一个分布式任务，存在任务调度及结果汇总阶段，计算集群吞吐均值时会略低于HDFS吞吐限速。在进行顺序读与随机读的测试之前需要确保HDFS之上已有指定的待测数据，如果没有待测数据请使用顺序写命令生成。-nrFiles表示文件数量，-fileSize表示文件大小，-random表示随机读写。
hadoop集群的一些测试
allen的博客
06-08 4057
hadoop的一些测试，做一个记录：一下操作都是在这个目录下面执行：/opt/cloudera/parcels/CDH/jars 1.HDFS 写数据IO测试 Throughput 平均值： 86.73306 Average IO rate 平均值： 89.33704 IO rate std deviation 平均值： 16.62124 Test exec t
hadoop性能测试
jinhong_lu的博客
06-16 8595
一、hadoop自带的性能基准评测工具 （一）TestDFSIO 1、测试写性能 （1）若有必要，先删除历史数据 $hadoop jar /home/hadoop/hadoop/share/hadoop/mapreduce2/hadoop-mapreduce-client-jobclient-2.3.0-cdh5.1.2-tests.jar TestDFSIO -clean （
Hadoop--Hadoop基准测试(读写)
web15085599741的博客
08-16 794
版权声明：本文为优快云博主「Polaris-zlf」的原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接及本声明。向HDFS文件系统中写入数据，10个文件，每个文件10MB，文件存放到/benchmarks/TestDFSIO/io_data下面。这些例子从多个角度对Hadoop进行测试，其中 TestDFSIO、mrbench和nnbench是三个广泛被使用的测试。nnbench用于测试NameNode的负载，它会生成很多与HDFS相关的请求，给NameNode施加较大的压力。.
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
表情包 代码片
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值