Hadoop1.0单点安装-Windows

最新推荐文章于 2024-05-12 20:29:05 发布

原创最新推荐文章于 2024-05-12 20:29:05 发布 · 3.9k 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #query #user #ssh #service #path

Hadoop 专栏收录该内容

12 篇文章

订阅专栏

本文详细介绍了在Windows上安装CygWin和配置SSH，然后进行Hadoop1.0的配置和启动过程，包括设置JAVA_HOME、修改配置文件、格式化NameNode、启动Hadoop服务以及验证文件上传。

一：CygWin安装

cygwin1.7.15 下载地址

安装省略、(记得安装ssh)

安装完成后将usr\sbin 目录加入到path环境变量中

二：ssh配置

$ ssh-host-config

*** Query: Should privilege separation be used? (yes/no) no

*** Query: (Say "no" if it is already installed as a service) (yes/no)yes

*** Query: Enter the value of CYGWIN for the daemon: [] ntsec

*** Query: Do you want to use a different name? (yes/no) yes

*** Query: Enter the new user name: admin

*** Query: Reenter: admin

*** Query: Create new privileged user account 'admin'? (yes/no) yes

*** Query: Please enter the password:密码

*** Query: Reenter:重复密码

启动ssh服务

net start sshed

配置无密登录

$ ssh-keygen（win7下以管理员身份运行）

Enter file in which to save the key (/home/Administrator/.ssh/id_rsa):回车

Enter passphrase (empty for no passphrase):回车

Enter same passphrase again:回车

cd /cygdrive/c/cygwin/home/Administrator/.ssh

(对应cygwin安装目录例如:D:\cygwin\home\Administrator\.ssh)

cp id_rsa.pub authorized_keys

$ ssh localhost

         The authenticity of host 'localhost (127.0.0.1)' can't be established.
         ECDSA key fingerprint is 86:07:88:db:34:94:f8:09:6d:f4:7d:19:48:67:fe:e1.
         Are you sure you want to continue connecting (yes/no)? yes

三：hadoop配置启动 (hadoop-1.0.0版本)

1.配置修改hadoop/conf目录下 4个文件

hadoop-env.sh core-site.xml hdfs-site.xml mapred-site.xml

①.hadoop-env.sh
export JAVA_HOME=/cygdrive/d/Java/jdk1.6.0_10

②.conf/core-site.xml:

<configuration>
	<property>
		<name>fs.default.name</name>
		<value>hdfs://localhost:9000</value>
	</property>
</configuration>

③.conf/hdfs-site.xml

<configuration>
	<property>
		<name>dfs.replication</name>
		<value>1</value>
	</property>
</configuration>

④.conf/mapred-site.xml

<configuration>
	<property>
         <name>mapred.job.tracker</name>
         <value>localhost:9001</value>
	 </property>
</configuration>

2. 启动

切换到hadoop 安装目录 cd /cygdrive/d/hadoop/hadoop-1.0.0

格式化NameNode bin/hadoop namenode -format

启动hadoop bin/start-all.sh

在hdsf系统中建立一个名称为test的目录 bin/hadoop fs -mkdir test

上传文件 bin/hadoop fs -put *.txt test (Hadoop根目录下的所有文本文件上传到了test目录)

也可以通过NameNode - http://localhost:50070/ 上面的Browse the filesystem连接验证是否上传成功

JobTracker - http://localhost:50030/

hadoop1.0版本启动时无法启动TaskTracker 解决方案见另一篇：问题记录