hadoop 学习笔记

-

HDFS

 

提供了中央NameNode管理各 DataNode 

 

提供了数据的复制备份机制（3份） 一份拷贝在本机柜另外一台机器 另两份备份在另外一个机柜的两台机器上

 

NameNode会记录重做日志 以及记录各个DataNode中保持的数据索引，所有文件读写请求都经过NameNode几点进行分发

 

健壮性：

心跳检测

数据完整性（校验和）

快照

 

DataNode Pipelining机制 （一个DataNode写完后，会由该DataNode将数据写入下一个DataNode，如此继续）

 

问题：在一个DataNode写满后，会不会自动寻找下一个DataNode？

 

------------

hdfs 设置了 replicate 参数为 1 ，需要两台datanode 机器，加上master 等于需要3台

 

 

--------------

索引文件应该是比较小的，最多包含例如所有中文和所有英文单词（索引项不应去保存内容，高亮的处理可以用其他方式）

 

 

MapReduce

 

【Task Side-Effect Files】

So, just create any side-files in the path returned by FileOutputFormat.getWorkOutputPath() from MapReduce task to take advantage of this feature.

 

javac -classpath ${HADOOP_HOME}/hadoop-core-${HADOOP_VERSION}.jar:${HADOOP_HOME}/hadoop-mapred-${HADOOP_VERSION}.jar:${HADOOP_HOME}/hadoop-hdfs-${HADOOP_VERSION}.jar -d wordcount_classes WordCount.java 

 

 

 

javac -classpath /root/cloud/hadoop/hadoop-common-0.21.0.jar:/root/cloud/hadoop/hadoop-mapred-0.21.0.jar

:/root/cloud/hadoop/hadoop-hdfs-0.21.0.jar -d . WordCount.java

 

jar -cvf wordcount.jar -C . . 

 

 

Bye 1

Goodbye 1

Hadoop 2

Hello 2

World 2 

 

 

参考文章：使用 Linux 和 Hadoop 进行分布式计算

 

http://bbs.hadoopor.com/thread-970-1-1.html hadoop 书籍及项目

http://wenku.baidu.com/view/51734f2c7375a417866f8f42.html mapreduce一些经验

 

 

安装：

准备三台机器，修改hostname 分别为node1,node2,node3 (vi /etc/sysconfig/network : HOSTNAME=node1 ，在命令行: hostname node1)

1.edit the file conf/hadoop-env.sh to define at least JAVA_HOME

2.vi conf/core-site.xml 

<name>fs.default.name</name>

<value>hdfs://node1:9000</value>

3.vi conf/hdfs-site.xml

<property>

         <name>dfs.replication</name>

         <value>1</value> <!-- 这里配置数据要拷贝到的节点数，如果这里配置为1，则必须有两台datanode节点

     </property>

4.vi conf/mapred-site.xml

<property>

         <name>mapred.job.tracker</name>

         <value>node1:9001</value>

     </property>

 

5.vi conf/masters

node1

6.vi conf/slaves

node2

node3

7.配置各台服务器ssh登录不用密码

在node1: 如果ssh localhost需要输入密码，执行以下步骤。

 1)$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

 2)$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

否则，只要将node1的id_dsa.pub 拷贝到其他两台服务器，并在两台服务器上执行步骤2)

 

8.拷贝hadoop文件到各服务器

参考命令：

 scp -r /home/xx/hadoop-0.21.0 node2:/home/xx/hadoop-0.21.0

 

9.在node1

bin/hadoop namenode -format

bin/start-all.sh

看日志输出是否有误

10.

（记得设置各服务器的防火墙，让他们之间可以互相通信）

在每台服务器 ps -aef | grep hadoop 看进程是否正常。

测试：bin/hadoop dfs -put conf input  

bin/hadoop dfs -ls  看是否上传成功

 

打开浏览器，输入http://node1:50070/即可看到整个hadoop情况，

JobTracker情况可以看http://node1:50030/

 

参考：

http://blog.youkuaiyun.com/inkfish/archive/2010/01/07/5150717.aspx Hadoop的安装-3.完全分布模式

https://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop3/?ca=drs-tp3008 用 Hadoop 进行分布式并行编程

 

 

 

 

 

 

FAQ:

1.ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs

下面给出两种解决办法，我使用的是第二种。

 

Workaround 1: Start from scratch

 

I can testify that the following steps solve this error, but the side effects won't make you happy (me neither). The crude workaround I have found is to:

 

1.     stop the cluster

 

2.     delete the data directory on the problematic datanode: the directory is specified by dfs.data.dir in conf/hdfs-site.xml; if you followed this tutorial, the relevant directory is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data

 

3.     reformat the namenode (NOTE: all HDFS data is lost during this process!)

 

4.     restart the cluster

 

When deleting all the HDFS data and starting from scratch does not sound like a good idea (it might be ok during the initial setup/testing), you might give the second approach a try.

 

Workaround 2: Updating namespaceID of problematic datanodes

 

Big thanks to Jared Stehler for the following suggestion. I have not tested it myself yet, but feel free to try it out and send me your feedback. This workaround is "minimally invasive" as you only have to edit one file on the problematic datanodes:

 

1.     stop the datanode

 

2.     edit the value of namespaceID in <dfs.data.dir>/current/VERSION to match the value of the current namenode

 

3.     restart the datanode

 

If you followed the instructions in my tutorials, the full path of the relevant file is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data/current/VERSION (background: dfs.data.dir is by default set to ${hadoop.tmp.dir}/dfs/data, and we set hadoop.tmp.dir to /usr/local/hadoop-datastore/hadoop-hadoop).

 

If you wonder how the contents of VERSION look like, here's one of mine:

 

#contents of <dfs.data.dir>/current/VERSION

 

namespaceID=393514426

 

storageID=DS-1706792599-10.10.10.1-50010-1204306713481

 

cTime=1215607609074

 

storageType=DATA_NODE

 

layoutVersion=-13

 

 

原因:每次namenode format会重新创建一个namenodeId,而tmp/dfs/data下包含了上次format下的id,namenode format清空了namenode下的数据,

但是没有晴空datanode下的数据,导致启动时失败,所要做的就是每次fotmat前,清空tmp一下 的所有目录.