《Hadoop The Definitive Guide》ch10 Administering Hadoop

Hadoop集群测试与管理

最新推荐文章于 2023-07-26 00:52:14 发布

原创最新推荐文章于 2023-07-26 00:52:14 发布 · 647 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop

Hadoop 专栏收录该内容

14 篇文章

订阅专栏

本文介绍了Hadoop集群上的基准测试程序，包括读写性能测试、文件系统一致性检查等，并展示了如何通过命令行操作安全模式及启动负载均衡器。同时提供了Hadoop相关组件的Web界面监控地址。

1. 集群基准测试程序

c01s02h01ate1:nomad2 # hadoop jar /local/hadoop/hadoop-0.20.203.0/hadoop-test-0.20.203.0.jar 
An example program must be given as the first argument.
Valid program names are:
  DFSCIOTest: Distributed i/o benchmark of libhdfs.
  DistributedFSCheck: Distributed checkup of the file system consistency.
  MRReliabilityTest: A program that tests the reliability of the MR framework by injecting faults/failures
  TestDFSIO: Distributed i/o benchmark.
  dfsthroughput: measure hdfs throughput
  filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)
  loadgen: Generic map/reduce load generator
  mapredtest: A map/reduce test check.
  mrbench: A map/reduce benchmark that can create many small jobs
  nnbench: A benchmark that stresses the namenode.
  testarrayfile: A test for flat files of binary key/value pairs.
  testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does identity map/reduce
  testfilesystem: A test for FileSystem read/write.
  testipc: A test for ipc.
  testmapredsort: A map/reduce program that validates the map-reduce framework's sort.
  testrpc: A test for rpc.
  testsequencefile: A test for flat files of binary key value pairs.
  testsequencefileinputformat: A test for sequence file input format.
  testsetfile: A test for flat files of binary key/value pairs.
  testtextinputformat: A test for text input format.
  threadedmapbench: A map/reduce benchmark that compares the performance of maps with multiple spills over maps with 1 spill

c01s02h01ate1:nomad2 # hadoop jar /local/hadoop/hadoop-0.20.203.0/hadoop-test-0.20.203.0.jar DFSCIOTest
TestDFSIO.0.0.4
Usage: TestDFSIO -read | -write | -clean [-nrFiles N] [-fileSize MB] [-resFile resultFileName] [-bufferSize Bytes]

2. safe mode

c01s02h01ate1:nomad2 # hadoop dfsadmin -safemode get
Safe mode is OFF

c01s02h01ate1:nomad2 # hadoop dfsadmin -refreshNodes

3. web界面

http://c01s0201ate2:50075/blockScannerReport

http://c01s0201ate1:50030/logLevel?log=org.apache.hadoop.mapred.JobTracker

http://c01s0201ate1:50030/stacks

4. 负载平衡

c01s02h01ate1:nomad2 # start-balancer.sh 
starting balancer, logging to /local/hadoop/hadoop-0.20.203.0/bin/../logs/hadoop-nomad2-balancer-c01s02h01ate1.out
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
The cluster is balanced. Exiting...
Balancing took 413.0 milliseconds