Hadoop初试

最新推荐文章于 2025-08-13 09:08:53 发布
最新推荐文章于 2025-08-13 09:08:53 发布 · 126 阅读 · 0
文章标签：

#Java #Linux #Apache #Hadoop #分布式

blog迁移至 :http://www.micmiu.com

[size=medium]Hadoop一个分布式系统基础架构，是Apache基金下的一个子项目。用户可以在不了解分布式底层细节的情况下，开发分布式程序。充分利用集群的威力高速运算和存储。Hadoop实现了一个分布式文件系统（Hadoop Distributed File System），简称HDFS。HDFS有着高容错性的特点，并且设计用来部署在低廉的（low-cost）硬件上。而且它提供高传输率（high throughput）来访问应用程序的数据，适合那些有着超大数据集（large data set）的应用程序。HDFS放宽了（relax）POSIX的要求（requirements）这样可以流的形式访问（streaming access）文件系统中的数据。

Hadoop官网：[url=http://hadoop.apache.org/]http://hadoop.apache.org/[/url][/size]

[size=medium][color=red]本文详细介绍如何搭建一个Hadoop的测试环境，将分别从单机、伪分布式等逐个讲解，并讲述在不同操作系统（Centos、Ubuntu）搭建过程中可能碰到的问题及解决方法，所有的测试过程本人在Centos、Ubuntu下均全部测试成功，全文的目录结构：[/color]
[color=blue][list]
[*]实验环境
[*]准备工作
[*]单机演示
[*]伪分布式演示[/list][/size][/color]
[color=blue][size=large]一、实验环境：[/size][/color]
[list]
[*]Windows Vista
[*]VirtualBox + Ubuntu10.10（OpenSSH 安装并启动）
[*]jdk 版本：1.6.0_20；安装路径为：/opt/jdk1.6（Hadoop要求jdk1.6.x）
[*]hadoop-0.20.203.0rc1.tar.gz（目前最新的稳定版本）
[/list]
以Ubuntu中的用户 michael为例：
[color=blue][size=large]二、前期准备[/size][/color]
首先将hadoop-0.20.203.0rc1.tar.gz解压到/home/michael/下

$tar -zxvf hadoop-0.20.203.0rc1.tar.gz -C /home/michael/
$mv hadoop-0.20.203.0 hadoop

修改hadoop-env.sh中JAVA_HOME的配置,找到如下信息：
[quote]# The java implementation to use. Required.
# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
[/quote]
修改为：
[quote]# The java implementation to use. Required.
#当前系统的JDK的路径
export JAVA_HOME=/opt/jdk1.6
[/quote]

[color=blue][size=large]三、单机演示（Standalone Operation）[/size][/color]
所涉及的到的操作命令如下：

$ cd /home/michael/hadoop
$ mkdir input 
$ cp conf/*.xml input 
$ bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+' 
$ cat output/*

详细过程如下：
[quote]michael@michael-VirtualBox:~/hadoop$ [color=red]mkdir input[/color]
michael@michael-VirtualBox:~/hadoop$ [color=red]cp conf/*.xml input[/color]
michael@michael-VirtualBox:~/hadoop$ [color=red]bin/hadoop jar hadoop-examples-0.20.203.0.jar grep input output 'dfs[a-z.]+'[/color]
11/07/16 10:06:48 INFO mapred.FileInputFormat: Total input paths to process : 6
11/07/16 10:06:48 INFO mapred.JobClient: Running job: job_local_0001
11/07/16 10:06:48 INFO mapred.MapTask: numReduceTasks: 1
11/07/16 10:06:48 INFO mapred.MapTask: io.sort.mb = 100
11/07/16 10:06:49 INFO mapred.MapTask: data buffer = 79691776/99614720
11/07/16 10:06:49 INFO mapred.MapTask: record buffer = 262144/327680
11/07/16 10:06:49 INFO mapred.MapTask: Starting flush of map output
11/07/16 10:06:49 INFO mapred.JobClient: map 0% reduce 0%
11/07/16 10:06:49 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
11/07/16 10:06:51 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/input/capacity-scheduler.xml:0+7457
11/07/16 10:06:51 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
11/07/16 10:06:51 INFO mapred.MapTask: numReduceTasks: 1
11/07/16 10:06:51 INFO mapred.MapTask: io.sort.mb = 100
11/07/16 10:06:51 INFO mapred.MapTask: data buffer = 79691776/99614720
11/07/16 10:06:51 INFO mapred.MapTask: record buffer = 262144/327680
11/07/16 10:06:51 INFO mapred.MapTask: Starting flush of map output
11/07/16 10:06:52 INFO mapred.MapTask: Finished spill 0
11/07/16 10:06:52 INFO mapred.Task: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
11/07/16 10:06:52 INFO mapred.JobClient: map 100% reduce 0%
11/07/16 10:06:54 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/input/hadoop-policy.xml:0+4644
11/07/16 10:06:54 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/input/hadoop-policy.xml:0+4644
11/07/16 10:06:54 INFO mapred.Task: Task 'attempt_local_0001_m_000001_0' done.
11/07/16 10:06:54 INFO mapred.MapTask: numReduceTasks: 1
11/07/16 10:06:54 INFO mapred.MapTask: io.sort.mb = 100
11/07/16 10:06:55 INFO mapred.MapTask: data buffer = 79691776/99614720
11/07/16 10:06:55 INFO mapred.MapTask: record buffer = 262144/327680
11/07/16 10:06:55 INFO mapred.MapTask: Starting flush of map output
11/07/16 10:06:55 INFO mapred.Task: Task:attempt_local_0001_m_000002_0 is done. And is in the process of commiting
11/07/16 10:06:57 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/input/mapred-queue-acls.xml:0+2033
11/07/16 10:06:57 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/input/mapred-queue-acls.xml:0+2033
11/07/16 10:06:57 INFO mapred.Task: Task 'attempt_local_0001_m_000002_0' done.
11/07/16 10:06:57 INFO mapred.MapTask: numReduceTasks: 1
11/07/16 10:06:57 INFO mapred.MapTask: io.sort.mb = 100
11/07/16 10:06:58 INFO mapred.MapTask: data buffer = 79691776/99614720
11/07/16 10:06:58 INFO mapred.MapTask: record buffer = 262144/327680
11/07/16 10:06:58 INFO mapred.MapTask: Starting flush of map output
11/07/16 10:06:58 INFO mapred.Task: Task:attempt_local_0001_m_000003_0 is done. And is in the process of commiting
11/07/16 10:07:00 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/input/mapred-site.xml:0+178
11/07/16 10:07:00 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/input/mapred-site.xml:0+178
11/07/16 10:07:00 INFO mapred.Task: Task 'attempt_local_0001_m_000003_0' done.
11/07/16 10:07:00 INFO mapred.MapTask: numReduceTasks: 1
11/07/16 10:07:00 INFO mapred.MapTask: io.sort.mb = 100
11/07/16 10:07:01 INFO mapred.MapTask: data buffer = 79691776/99614720
11/07/16 10:07:01 INFO mapred.MapTask: record buffer = 262144/327680
11/07/16 10:07:01 INFO mapred.MapTask: Starting flush of map output
11/07/16 10:07:01 INFO mapred.Task: Task:attempt_local_0001_m_000004_0 is done. And is in the process of commiting
11/07/16 10:07:04 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/input/core-site.xml:0+178
11/07/16 10:07:04 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/input/core-site.xml:0+178
11/07/16 10:07:04 INFO mapred.Task: Task 'attempt_local_0001_m_000004_0' done.
11/07/16 10:07:04 INFO mapred.MapTask: numReduceTasks: 1
11/07/16 10:07:04 INFO mapred.MapTask: io.sort.mb = 100
11/07/16 10:07:04 INFO mapred.MapTask: data buffer = 79691776/99614720
11/07/16 10:07:04 INFO mapred.MapTask: record buffer = 262144/327680
11/07/16 10:07:04 INFO mapred.MapTask: Starting flush of map output
11/07/16 10:07:04 INFO mapred.Task: Task:attempt_local_0001_m_000005_0 is done. And is in the process of commiting
11/07/16 10:07:07 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/input/hdfs-site.xml:0+178
11/07/16 10:07:07 INFO mapred.Task: Task 'attempt_local_0001_m_000005_0' done.
11/07/16 10:07:07 INFO mapred.LocalJobRunner:
11/07/16 10:07:07 INFO mapred.Merger: Merging 6 sorted segments
11/07/16 10:07:07 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 21 bytes
11/07/16 10:07:07 INFO mapred.LocalJobRunner:
11/07/16 10:07:07 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
11/07/16 10:07:07 INFO mapred.LocalJobRunner:
11/07/16 10:07:07 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
11/07/16 10:07:07 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/home/michael/hadoop/grep-temp-1267281521
11/07/16 10:07:10 INFO mapred.LocalJobRunner: reduce > reduce
11/07/16 10:07:10 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
11/07/16 10:07:10 INFO mapred.JobClient: map 100% reduce 100%
11/07/16 10:07:10 INFO mapred.JobClient: Job complete: job_local_0001
11/07/16 10:07:10 INFO mapred.JobClient: Counters: 17
11/07/16 10:07:10 INFO mapred.JobClient: File Input Format Counters
11/07/16 10:07:10 INFO mapred.JobClient: Bytes Read=14668
11/07/16 10:07:10 INFO mapred.JobClient: File Output Format Counters
11/07/16 10:07:10 INFO mapred.JobClient: Bytes Written=123
11/07/16 10:07:10 INFO mapred.JobClient: FileSystemCounters
11/07/16 10:07:10 INFO mapred.JobClient: FILE_BYTES_READ=1106074
11/07/16 10:07:10 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1231779
11/07/16 10:07:10 INFO mapred.JobClient: Map-Reduce Framework
11/07/16 10:07:10 INFO mapred.JobClient: Map output materialized bytes=55
11/07/16 10:07:10 INFO mapred.JobClient: Map input records=357
11/07/16 10:07:10 INFO mapred.JobClient: Reduce shuffle bytes=0
11/07/16 10:07:10 INFO mapred.JobClient: Spilled Records=2
11/07/16 10:07:10 INFO mapred.JobClient: Map output bytes=17
11/07/16 10:07:10 INFO mapred.JobClient: Map input bytes=14668
11/07/16 10:07:10 INFO mapred.JobClient: SPLIT_RAW_BYTES=611
11/07/16 10:07:10 INFO mapred.JobClient: Combine input records=1
11/07/16 10:07:10 INFO mapred.JobClient: Reduce input records=1
11/07/16 10:07:10 INFO mapred.JobClient: Reduce input groups=1
11/07/16 10:07:10 INFO mapred.JobClient: Combine output records=1
11/07/16 10:07:10 INFO mapred.JobClient: Reduce output records=1
11/07/16 10:07:10 INFO mapred.JobClient: Map output records=1
11/07/16 10:07:10 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
11/07/16 10:07:10 INFO mapred.FileInputFormat: Total input paths to process : 1
11/07/16 10:07:10 INFO mapred.JobClient: Running job: job_local_0002
11/07/16 10:07:10 INFO mapred.MapTask: numReduceTasks: 1
11/07/16 10:07:10 INFO mapred.MapTask: io.sort.mb = 100
11/07/16 10:07:11 INFO mapred.MapTask: data buffer = 79691776/99614720
11/07/16 10:07:11 INFO mapred.MapTask: record buffer = 262144/327680
11/07/16 10:07:11 INFO mapred.MapTask: Starting flush of map output
11/07/16 10:07:11 INFO mapred.MapTask: Finished spill 0
11/07/16 10:07:11 INFO mapred.Task: Task:attempt_local_0002_m_000000_0 is done. And is in the process of commiting
11/07/16 10:07:11 INFO mapred.JobClient: map 0% reduce 0%
11/07/16 10:07:13 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/grep-temp-1267281521/part-00000:0+111
11/07/16 10:07:13 INFO mapred.LocalJobRunner: file:/home/michael/hadoop/grep-temp-1267281521/part-00000:0+111
11/07/16 10:07:13 INFO mapred.Task: Task 'attempt_local_0002_m_000000_0' done.
11/07/16 10:07:13 INFO mapred.LocalJobRunner:
11/07/16 10:07:13 INFO mapred.Merger: Merging 1 sorted segments
11/07/16 10:07:13 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 21 bytes
11/07/16 10:07:13 INFO mapred.LocalJobRunner:
11/07/16 10:07:13 INFO mapred.Task: Task:attempt_local_0002_r_000000_0 is done. And is in the process of commiting
11/07/16 10:07:13 INFO mapred.LocalJobRunner:
11/07/16 10:07:13 INFO mapred.Task: Task attempt_local_0002_r_000000_0 is allowed to commit now
11/07/16 10:07:13 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0002_r_000000_0' to file:/home/michael/hadoop/output
11/07/16 10:07:14 INFO mapred.JobClient: map 100% reduce 0%
11/07/16 10:07:16 INFO mapred.LocalJobRunner: reduce > reduce
11/07/16 10:07:16 INFO mapred.Task: Task 'attempt_local_0002_r_000000_0' done.
11/07/16 10:07:17 INFO mapred.JobClient: map 100% reduce 100%
11/07/16 10:07:17 INFO mapred.JobClient: Job complete: job_local_0002
11/07/16 10:07:17 INFO mapred.JobClient: Counters: 17
11/07/16 10:07:17 INFO mapred.JobClient: File Input Format Counters
11/07/16 10:07:17 INFO mapred.JobClient: Bytes Read=123
11/07/16 10:07:17 INFO mapred.JobClient: File Output Format Counters
11/07/16 10:07:17 INFO mapred.JobClient: Bytes Written=23
11/07/16 10:07:17 INFO mapred.JobClient: FileSystemCounters
11/07/16 10:07:17 INFO mapred.JobClient: FILE_BYTES_READ=606737
11/07/16 10:07:17 INFO mapred.JobClient: FILE_BYTES_WRITTEN=700981
11/07/16 10:07:17 INFO mapred.JobClient: Map-Reduce Framework
11/07/16 10:07:17 INFO mapred.JobClient: Map output materialized bytes=25
11/07/16 10:07:17 INFO mapred.JobClient: Map input records=1
11/07/16 10:07:17 INFO mapred.JobClient: Reduce shuffle bytes=0
11/07/16 10:07:17 INFO mapred.JobClient: Spilled Records=2
11/07/16 10:07:17 INFO mapred.JobClient: Map output bytes=17
11/07/16 10:07:17 INFO mapred.JobClient: Map input bytes=25
11/07/16 10:07:17 INFO mapred.JobClient: SPLIT_RAW_BYTES=110
11/07/16 10:07:17 INFO mapred.JobClient: Combine input records=0
11/07/16 10:07:17 INFO mapred.JobClient: Reduce input records=1
11/07/16 10:07:17 INFO mapred.JobClient: Reduce input groups=1
11/07/16 10:07:17 INFO mapred.JobClient: Combine output records=0
11/07/16 10:07:17 INFO mapred.JobClient: Reduce output records=1
11/07/16 10:07:17 INFO mapred.JobClient: Map output records=1
michael@michael-VirtualBox:~/hadoop$

michael@michael-VirtualBox:~/hadoop$[color=red] cat output/* [/color]
[color=blue]1 dfsadmin[/color]
michael@michael-VirtualBox:~/hadoop$ [/quote]
到此单机演示成功。

[color=blue][size=large]四、伪分布式演示（Pseudo-Distributed Operation）[/size][/color]

[size=medium]1. 修改配置文件：[/size]
conf/core-site.xml:
<configuration>
     <property>
         <name>fs.default.name</name>
         <value>hdfs://localhost:9000</value>
     </property>
</configuration>

conf/hdfs-site.xml:
<configuration>
     <property>
         <name>dfs.replication</name>
         <value>1</value>
     </property>
</configuration>

conf/mapred-site.xml:

<configuration>
     <property>
         <name>mapred.job.tracker</name>
         <value>localhost:9001</value>
     </property>
</configuration>


[size=medium]2.设置SSH无密码登陆[/size]

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa 
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

ps：如果是Centos系统，有关SSH无密码的详细设置请看：[url=http://sjsky.iteye.com/blog/1123184]Linux(Centos)配置OpenSSH无密码登陆[/url]

[size=medium]3.测试：[/size]
[color=blue]相关操作的基本命令如下：[/color]

#Format a new distributed-filesystem:
$ bin/hadoop namenode -format 

#Start the hadoop daemons:
$ bin/start-all.sh 
Copy the input files into the distributed filesystem:
$ bin/hadoop fs -put conf input 

#Run some of the examples provided:
$ bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+' 

#Copy the output files from the distributed filesystem to the local filesytem and examine them:
$ bin/hadoop fs -get output output 

$ cat output/output/*


以上的测试操作在Centos5中测试顺利成功，而在Ubuntu10.10系统中却失败了，在执行命令：bin/hadoop fs -put conf input 时出错，错误信息类似[color=red]“could only be replicated to 0 nodes, instead of 1”[/color]，引起该错误信息的原因有多种(详见：[url=http://sjsky.iteye.com/blog/1124545]http://sjsky.iteye.com/blog/1124545[/url])，但此处的出错的原因是由于hadoop.tmp.dir默认配置指向/tmp/hadoop-${user.name},而在Ubuntu系统中，/tmp目录下的文件系统的类型往往是Hadoop不支持的。

解决的办法是重新定义hadoop.tmp.dir指向，修改配置文件conf/core-site.xml如下：

<configuration>
	<property>
		<name>fs.default.name</name>
		<value>hdfs://localhost:9000</value>
	</property>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/home/michael/hadooptmp/hadoop-${user.name}</value>
		<description>
			A base for other temporary directories.
		</description>
	</property>
</configuration>

再次进行测试，成功运行。
整个测试过程的详细信息如下：
[quote]michael@michael-VirtualBox:~/hadoop$ [color=red]ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa [/color]
Generating public/private dsa key pair.
Your identification has been saved in /home/michael/.ssh/id_dsa.
Your public key has been saved in /home/michael/.ssh/id_dsa.pub.
The key fingerprint is:
2a:47:e3:3a:c8:80:ab:97:d1:c6:68:54:9a:45:9f:59 michael@michael-VirtualBox
The key's randomart image is:
+--[ DSA 1024]----+
| .. E |
| o. + |
| = + |
| + |
|.. + o S |
|o + +o o |
| = =. + |
|. = .+ |
|o. .. |
+-----------------+
michael@michael-VirtualBox:~/hadoop$ [color=red]cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys [/color]
michael@michael-VirtualBox:~/hadoop$ ssh localhost
Linux michael-VirtualBox 2.6.35-22-generic #33-Ubuntu SMP Sun Sep 19 20:34:50 UTC 2010 i686 GNU/Linux
Ubuntu 10.10

[color=red]Welcome to Ubuntu![/color]
* Documentation: https://help.ubuntu.com/

71 packages can be updated.
71 updates are security updates.

New release 'natty' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Wed Jul 15 15:56:17 2011 from shnap.local
michael@michael-VirtualBox:~$ exit
注销
Connection to localhost closed.


michael@michael-VirtualBox:~/hadoop$ [color=red]bin/hadoop namenode -format[/color]
11/07/16 12:43:45 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = michael-VirtualBox/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.203.0
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
************************************************************/
11/07/16 12:43:46 INFO util.GSet: VM type = 32-bit
11/07/16 12:43:46 INFO util.GSet: 2% max memory = 19.33375 MB
11/07/16 12:43:46 INFO util.GSet: capacity = 2^22 = 4194304 entries
11/07/16 12:43:46 INFO util.GSet: recommended=4194304, actual=4194304
11/07/16 12:43:46 INFO namenode.FSNamesystem: fsOwner=michael
11/07/16 12:43:46 INFO namenode.FSNamesystem: supergroup=supergroup
11/07/16 12:43:46 INFO namenode.FSNamesystem: isPermissionEnabled=true
11/07/16 12:43:46 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
11/07/16 12:43:46 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
11/07/16 12:43:46 INFO namenode.NameNode: Caching file names occuring more than 10 times
11/07/16 12:43:47 INFO common.Storage: Image file of size 113 saved in 0 seconds.
11/07/16 12:43:47 INFO common.Storage: Storage directory /home/michael/hadooptmp/hadoop-michael/dfs/name has been successfully formatted.
11/07/16 12:43:47 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at michael-VirtualBox/127.0.1.1
************************************************************/
michael@michael-VirtualBox:~/hadoop$[color=red] bin/start-all.sh [/color]
starting namenode, logging to /home/michael/hadoop/bin/../logs/hadoop-michael-namenode-michael-VirtualBox.out
localhost: starting datanode, logging to /home/michael/hadoop/bin/../logs/hadoop-michael-datanode-michael-VirtualBox.out
localhost: starting secondarynamenode, logging to /home/michael/hadoop/bin/../logs/hadoop-michael-secondarynamenode-michael-VirtualBox.out
starting jobtracker, logging to /home/michael/hadoop/bin/../logs/hadoop-michael-jobtracker-michael-VirtualBox.out
localhost: starting tasktracker, logging to /home/michael/hadoop/bin/../logs/hadoop-michael-tasktracker-michael-VirtualBox.out

michael@michael-VirtualBox:~/hadoop$ jps
[color=red]7948 SecondaryNameNode
8033 JobTracker
8887 Jps
7627 NameNode
7781 DataNode
8190 TaskTracker[/color]

michael@michael-VirtualBox:~/hadoop$ [color=red]bin/hadoop fs -put conf input[/color]
michael@michael-VirtualBox:~/hadoop$ [color=red]bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'[/color]
11/07/16 12:46:21 INFO mapred.FileInputFormat: Total input paths to process : 15
11/07/16 12:46:21 INFO mapred.JobClient: Running job: job_201107161244_0001
11/07/16 12:46:22 INFO mapred.JobClient: map 0% reduce 0%
11/07/16 12:47:09 INFO mapred.JobClient: map 13% reduce 0%
11/07/16 12:47:33 INFO mapred.JobClient: map 26% reduce 0%
11/07/16 12:47:45 INFO mapred.JobClient: map 26% reduce 8%
11/07/16 12:47:54 INFO mapred.JobClient: map 40% reduce 8%
11/07/16 12:48:07 INFO mapred.JobClient: map 53% reduce 13%
11/07/16 12:48:16 INFO mapred.JobClient: map 53% reduce 17%
11/07/16 12:48:24 INFO mapred.JobClient: map 66% reduce 17%
11/07/16 12:48:36 INFO mapred.JobClient: map 80% reduce 22%
11/07/16 12:48:42 INFO mapred.JobClient: map 80% reduce 26%
11/07/16 12:48:45 INFO mapred.JobClient: map 93% reduce 26%
11/07/16 12:48:53 INFO mapred.JobClient: map 100% reduce 26%
11/07/16 12:48:58 INFO mapred.JobClient: map 100% reduce 33%
11/07/16 12:49:07 INFO mapred.JobClient: map 100% reduce 100%
11/07/16 12:49:14 INFO mapred.JobClient: Job complete: job_201107161244_0001
11/07/16 12:49:15 INFO mapred.JobClient: Counters: 26
11/07/16 12:49:15 INFO mapred.JobClient: Job Counters
11/07/16 12:49:15 INFO mapred.JobClient: Launched reduce tasks=1
11/07/16 12:49:15 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=255488
11/07/16 12:49:15 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
11/07/16 12:49:15 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
11/07/16 12:49:15 INFO mapred.JobClient: Launched map tasks=15
11/07/16 12:49:15 INFO mapred.JobClient: Data-local map tasks=15
11/07/16 12:49:15 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=115656
11/07/16 12:49:15 INFO mapred.JobClient: File Input Format Counters
11/07/16 12:49:15 INFO mapred.JobClient: Bytes Read=25623
11/07/16 12:49:15 INFO mapred.JobClient: File Output Format Counters
11/07/16 12:49:15 INFO mapred.JobClient: Bytes Written=180
11/07/16 12:49:15 INFO mapred.JobClient: FileSystemCounters
11/07/16 12:49:15 INFO mapred.JobClient: FILE_BYTES_READ=82
11/07/16 12:49:15 INFO mapred.JobClient: HDFS_BYTES_READ=27281
11/07/16 12:49:15 INFO mapred.JobClient: FILE_BYTES_WRITTEN=342206
11/07/16 12:49:15 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=180
11/07/16 12:49:15 INFO mapred.JobClient: Map-Reduce Framework
11/07/16 12:49:15 INFO mapred.JobClient: Map output materialized bytes=166
11/07/16 12:49:15 INFO mapred.JobClient: Map input records=716
11/07/16 12:49:15 INFO mapred.JobClient: Reduce shuffle bytes=166
11/07/16 12:49:15 INFO mapred.JobClient: Spilled Records=6
11/07/16 12:49:15 INFO mapred.JobClient: Map output bytes=70
11/07/16 12:49:15 INFO mapred.JobClient: Map input bytes=25623
11/07/16 12:49:15 INFO mapred.JobClient: Combine input records=3
11/07/16 12:49:15 INFO mapred.JobClient: SPLIT_RAW_BYTES=1658
11/07/16 12:49:15 INFO mapred.JobClient: Reduce input records=3
11/07/16 12:49:15 INFO mapred.JobClient: Reduce input groups=3
11/07/16 12:49:15 INFO mapred.JobClient: Combine output records=3
11/07/16 12:49:15 INFO mapred.JobClient: Reduce output records=3
11/07/16 12:49:15 INFO mapred.JobClient: Map output records=3
11/07/16 12:49:16 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
11/07/16 12:49:17 INFO mapred.FileInputFormat: Total input paths to process : 1
11/07/16 12:49:18 INFO mapred.JobClient: Running job: job_201107161244_0002
11/07/16 12:49:19 INFO mapred.JobClient: map 0% reduce 0%
11/07/16 12:49:40 INFO mapred.JobClient: map 100% reduce 0%
11/07/16 12:49:55 INFO mapred.JobClient: map 100% reduce 100%
11/07/16 12:50:00 INFO mapred.JobClient: Job complete: job_201107161244_0002
11/07/16 12:50:00 INFO mapred.JobClient: Counters: 26
11/07/16 12:50:00 INFO mapred.JobClient: Job Counters
11/07/16 12:50:00 INFO mapred.JobClient: Launched reduce tasks=1
11/07/16 12:50:00 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=16946
11/07/16 12:50:00 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
11/07/16 12:50:00 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
11/07/16 12:50:00 INFO mapred.JobClient: Launched map tasks=1
11/07/16 12:50:00 INFO mapred.JobClient: Data-local map tasks=1
11/07/16 12:50:00 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=14357
11/07/16 12:50:00 INFO mapred.JobClient: File Input Format Counters
11/07/16 12:50:00 INFO mapred.JobClient: Bytes Read=180
11/07/16 12:50:00 INFO mapred.JobClient: File Output Format Counters
11/07/16 12:50:00 INFO mapred.JobClient: Bytes Written=52
11/07/16 12:50:00 INFO mapred.JobClient: FileSystemCounters
11/07/16 12:50:00 INFO mapred.JobClient: FILE_BYTES_READ=82
11/07/16 12:50:00 INFO mapred.JobClient: HDFS_BYTES_READ=298
11/07/16 12:50:00 INFO mapred.JobClient: FILE_BYTES_WRITTEN=41947
11/07/16 12:50:00 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=52
11/07/16 12:50:00 INFO mapred.JobClient: Map-Reduce Framework
11/07/16 12:50:00 INFO mapred.JobClient: Map output materialized bytes=82
11/07/16 12:50:00 INFO mapred.JobClient: Map input records=3
11/07/16 12:50:00 INFO mapred.JobClient: Reduce shuffle bytes=82
11/07/16 12:50:00 INFO mapred.JobClient: Spilled Records=6
11/07/16 12:50:00 INFO mapred.JobClient: Map output bytes=70
11/07/16 12:50:00 INFO mapred.JobClient: Map input bytes=94
11/07/16 12:50:00 INFO mapred.JobClient: Combine input records=0
11/07/16 12:50:00 INFO mapred.JobClient: SPLIT_RAW_BYTES=118
11/07/16 12:50:00 INFO mapred.JobClient: Reduce input records=3
11/07/16 12:50:00 INFO mapred.JobClient: Reduce input groups=1
11/07/16 12:50:00 INFO mapred.JobClient: Combine output records=0
11/07/16 12:50:00 INFO mapred.JobClient: Reduce output records=3
11/07/16 12:50:00 INFO mapred.JobClient: Map output records=3
michael@michael-VirtualBox:~/hadoop$

michael@michael-VirtualBox:~/hadoop$ [color=red]cat output/output/*[/color]
[color=blue]cat: output/output/_logs: 是一个目录
1 dfs.replication
1 dfs.server.namenode.
1 dfsadmin[/color]
michael@michael-VirtualBox:~/hadoop$ [/quote]
到此伪分布式的演示成功。


转载请注明来自：Michael's blog @ [url=http://sjsky.iteye.com]http://sjsky.iteye.com[/url]

----------------------------- 分 ------------------------------ 隔 ------------------------------ 线 ------------------------------
4.2-5 初试HDFS Shell
howard2005的专栏
10-25 434
通过本次实战，我们学习了如何使用HDFS Shell进行基本的文件系统操作，包括查看目录、创建目录、上传文件、查看文件内容、删除文件和目录等。这些操作是Hadoop数据处理的基础，为后续的数据分析和处理提供了必要的技能。
大数据离线-Hadoop入门（上）
weixin_42229056的博客
09-10 849
本博客现阶段主要针对大数据的学习会逐一推出相关的学习文档，从简介，原理，入门测试，小的案例练习包括。会每天抽出时间更新，感兴趣的童鞋，欢迎关注订阅，多多交流。 本次主要介绍大数据离线阶段的Hadoop入门，分为上下两篇。 上篇的内容为Hadoop简介，历史，特性，应用四个模块。 下篇的内容为hadoop的集群搭建，集群启动，入门初试三部分。 Hadoop介绍 一般我们讲的Hadoo...
Hadoop环境搭建笔记
zakexu的专栏
06-19 859
（一）单机模式 1.创建hadoop用户组以及用户 （1）创建用户组：sudo addgroup hadoop （2）创建用户：sudo adduser --ingroup hadoop hadoop （3）为用户设置管理员权限：修改/etc/sudoers文件，在root ALL=（ALL：ALL）ALL后面增加 ： hadoop ALL=（ALL：ALL） PS：创建完hadoop用
【hadoop学习】Hadoop配置
11-05 1020
【安装版本】 1、WMware 7.1、RedHat Enterprise Linux 5 2、Hadoop 0.22 【安装过程】 配置的过程实际就是修改相关配置文件的过程，这里总结下配置单节点（single node) 环境下的 单节点配置和伪分布式配置（standalone & Pseudo-Distributed) 1、配置之前需要配置 conf/hadoop_env.s
Hadoop 单机安装
05-29 237
Hadoop 单机安装 HDFS：分布式存储系统 MapReduce：分布式计算系统 YARN：hadoop 的资源调度系统，用于同一管理集群中的资源（内存等） Hive: 分布式数据仓库，其实说白了就是一个数据分析工具，底层用的还是MapReduce Sqoop:用于将传统数据库中数据导入到hbase或者Hdfs中一个导入工具 YARN：首先是主从结构，主节点可以有两个进程（ResourceManager）更好的HA高性能，从节点可以有多个（NodeManager），ResourceManag
cat 输出文件内容
我不会编程，但不是完全不会，我会一点点
02-27 6383
功能：把一个或者多个文件（或者标准输入）连接在一起，并标准输出。（Concatenate FILE(s), or standard input, to standard output.） cat命令常用来显示文件内容，或者将几个文件连接起来显示，或者从标准输入读取内容并显示。它常与重定向符号配合使用。cat是Concatenate的缩写。 Linux 有三个特别文件，分别 1
hadoop单机安装
不会说普通话的程序员
04-27 955
1. 上传tar包 2. 解压 3. 配置jdk环境 4. 进入/usr/local/src/hadoop-2.7.2/etc/hadoop 6. 修改hadoop-env.sh文件里面的jdk环境变量 7. 配置环境变量：vim/etc/profile HADOOP_HOME=/usr/local/src/hadoop-2.7.2/ export PATH=$H...
hadoop初试2
千山万水怎么能隔阻我对你的爱
08-01 176
      前言：前两天尝试分别在位于北京上海的服务器上搭建hadoop环境，最后是以失败告终了，认为主要是因为3台服务器并非内网集群，固今天打算使用一台电脑进行伪分布式的hadoop环境部署，用以学习。   主要参照：https://www.cnblogs.com/biehongli/p/7640469.html #1.首先根据参照文档，将机器上建好的3台虚拟机，进行网络分配，使他们...
Hadoop初试，已解决！
千山万水怎么能隔阻我对你的爱
07-30 1328
前言：Hadoop很重要，有很强的应用，上手学习一下。 目的：具备基本的Hadoop知识，包括所需组件的设置以及成功安装Hadoop集群 书籍：Hadoop权威指南-大数据的存储与分析（清华大学出版社） blog：https://www.cnblogs.com/biehongli/p/7640469.html http://hadoop.apache.org/docs/r3.0.0/ind...
hadoop初试时遇到的一些小麻烦
wgcnqad的博客
06-08 355
1：eclipse打jar 总是找不到 引用的三方包   解决方法 用 fat-jar;(别忘了把main方法摘除来)安装fat-jar使用http://kurucz-grafika.de/fatjar 2:下载fat-jar的时候报错 ，查看别的大牛：解决方法： Help -> Install New Software... -> Work with -> 选择“The Eclip
Hadoop学习-4 Hadoop初试
nodie的专栏
04-30 95
参照官方wordcount示例，统计每个产品的销量数据：产品编号    销量131B    643CB2    61BC1A    41CCC2    59ACC2    92131B     63CB2    323CB2    36BC1A    48ACC2    40将相同的产品编号的销量统计出来程序如下：   1，Mapper： package com.sun.hadoop;...
Hadoop大数据平台开发与案例分析
spark798的博客
06-24 5618
关于举办“Hadoop大数据平台开发与案例分析 ”高级工程师 一、课程介绍 1.需求理解 Hadoop 设计之初的目标就定位于高可靠性、高可拓展性、高容错性和高效性，正是这些设计上与生俱来的优点，才使得Hadoop 一出现就受到众多大公司的青睐，同时也引起了研究界的普遍关注。 对电信运营商而言,用户上网日志包含了大量用户个性化需求、喜好信息,对其进行分析和挖掘...
苏州大学 计算机学科和技术学院 研究生历年真题整理，包含初试和复试题目.zip
03-31
- **大数据处理**：Hadoop、Spark等分布式计算框架，以及MapReduce模型在大数据处理中的应用。 - **云计算技术**：IaaS、PaaS、SaaS的概念，虚拟化技术，如Kubernetes和Docker的使用。 3. **软件工程**： - **...
第10节 大模型分布式推理典型场景实战与架构设计
sinat_20277079的博客
08-11 1011
大模型分布式推理的落地需紧密结合业务场景，不同场景的核心诉求差异显著：有的追求长文本处理能力，有的侧重实时响应速度，有的侧重隐私保护。本节通过6个典型场景，从需求拆解到架构选型、技术优化、性能验证，完整呈现分布式推理系统的设计过程，每个场景均包含可复用的技术方案和代码示例，帮助掌握从理论到实践的落地方法。
【IgH EtherCAT】 一个基于RTAI实时系统的EtherCAT分布式时钟示例程序，实现了高精度的主从站时钟同步
cxyhjl的博客
08-12 431
此代码是一个高级的硬实时内核模块，它不仅利用 RTAI 实现了高精度的周期性控制，更核心的是展示了如何使用 IgH Master 的 DC 功能来主导和控制整个 EtherCAT 网络的时钟同步。它通过将 RTAI 的高精度时间作为基准，周期性地同步网络中的参考从站，并命令所有其他从站跟随，从而在整个分布式系统中建立起一个统一、精确的时间基准。RTAI 的高精度时间是整个系统的基准，它被周期性地“注入”到 EtherCAT 网络中，同步所有从站的时钟。这个时间戳是后续所有 DC 同步计算的基础。
【IgH EtherCAT】 利用标准的 Linux 实时特性（需 PREEMPT_RT 内核补丁）来实现分布式时钟的同步功能
cxyhjl的博客
08-12 1001
它不仅展示了如何利用标准的 Linux 实时特性来构建一个确定性的周期性任务，更重要的是，它详细演示了如何实现 EtherCAT 的分布式时钟同步，将 Linux 系统时间作为整个网络的统一时间基准。Linux 系统的高精度单调时钟是整个系统的基准，它被周期性地“注入”到 EtherCAT 网络中，同步所有从站的时钟。）写入 EtherCAT 帧，发送给参考从站（隐式选择的第一个支持DC的从站，通常是 EK1100），命令参考从站将自己的时钟设置为这个值。：两次连续唤醒之间的时间间隔，反映了周期的稳定性。
DDIA第五章：分布式数据复制中的一致性与冲突处理
言之。
08-11 873
本文系统介绍了分布式数据复制中的核心概念与解决方案。主要内容包括：1）读取一致性保障机制（单调读、一致前缀读）及其实现方法；2）复制延迟问题的优化策略，分为一致性模型优化和技术手段优化两类；3）多主复制架构的优缺点、典型应用场景（如多数据中心、离线操作）及写入冲突处理技术（版本号检测、LWW等）；4）补充了最终一致性、强一致性等重要概念，并给出了多数据中心多主复制的配置实践。文章强调在分布式系统设计中需要根据业务需求权衡一致性与可用性，建议通过抽象一致性模型、防御性编程等方法来构建可靠的分布式系统。这些知识
Java后端面试题（含Dubbo、MQ、分布式、并发、算法）
最新发布
2301_80165178的博客
08-13 1239
本文汇总了Java高级、分布式系统、设计模式、算法数据结构及并发编程五大技术领域的核心面试题。主要内容包括：消息队列选型与消息积压处理方案（Kafka/RabbitMQ等）、Dubbo和Spring Cloud的微服务实现对比、分布式锁设计与分库分表方案；设计模式应用场景及JDK中的典型实现；链表/二叉树等数据结构算法题解；并发编程中Synchronized与ReentrantLock的区别、线程池配置及volatile内存可见性等。涵盖分布式系统设计、高并发处理等关键技术点，适用于中高级Java开发者面试
高并发场景下分布式ID生成方案对比与实践指南
qq_35716689的博客
08-11 763
本文对比分析了多种在高并发场景下常见的分布式ID生成方案，包括Twitter Snowflake、UUID、Redis自增、Leaf等，从原理、性能、可用性和扩展性维度进行评估，并结合实际测试数据给出选型建议和实践示例，为后端开发者提供参考。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值