hadoop+spark配置笔记

vi  .bashrc

添加

export SCALA_HOME=/opt/scala/scala-2.11.7
export JAVA_HOME=/usr/lib/jvm/java
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export HADOOP_HOME=/opt/hadoop-2.6.0
export HADOOP_CONFIG_HOME=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/opt/spark-1.6.0-bin-hadoop2.6
export SBT_HOME=/opt/scala/scala-2.11.7/sbt
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:/opt/scala/scala-2.11.7/bin:$JAVA_HOME/jre/bin:$JAVA_HOME/bin:$SPARK_HOME/bin:$SBT_HOME/bin

#autorun
/usr/sbin/sshd

安装ssh     启动/etc/init.d/ssh start

下载jdk  scala hadoop spark spark并安装配置具体见

http://tashan10.com/yong-dockerda-jian-hadoopwei-fen-bu-shi-ji-qun/

http://blog.youkuaiyun.com/stark_summer/article/details/42458081

配置hadoop

core-site.xm

<configuration>
<property>
            <name>hadoop.tmp.dir</name>
            <value>file:/opt/hadoop-2.6.0/tmp</value>
            <description>A base for other temporary directories.</description>
    </property>


    <property>
            <name>fs.default.name</name>
            <value>hdfs://master01:9000</value>
            <final>true</final>


    </property>
</configuration>

hadoop-env.sh 

export JAVA_HOME=/usr/lib/jvm/java

hdfs-site.xml 

<configuration>
<property>
        <name>dfs.replication</name>
        <value>2</value>
        <final>true</final>
    </property>
 <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>master01:50090</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/opt/hadoop-2.6.0/namenode</value>
        <final>true</final>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/opt/hadoop-2.6.0/datanode</value>
        <final>true</final>
    </property>
</configuration>

 mapred-site.xml

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
</configuration>

yarn-site.xml 

<configuration>
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master01</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<!-- Site specific YARN configuration properties -->


</configuration>

配置 yarn-env.sh （修改JAVA_HOME）

# some Java parameters

export JAVA_HOME=/usr/lib/jvm/java

slaves

slave01

slave02

slave03

然后把它弄成镜像发布

在开启

sudo docker commit -m "hadoop+weiyunxing" master01 shanyx/ubuntu:spark10

sudo docker run -it  -v /home/syx/ins:/mnt -p 50071:50070 -p 50076:50075 -p 4041:4040 -p 8081:8080 -p 8089:8088 -p 19889:19888 -p 8043:8042 -h master01 --name master01 shanyx/ubuntu:spark10 /bin/bash

格式化在hadoop

./bin/hdfs namenode -format

.sbin/start-all.sh

jps