Spark_2.0 on hadoop_2.7.2

最新推荐文章于 2024-10-10 11:56:45 发布

领衔男一号

最新推荐文章于 2024-10-10 11:56:45 发布

阅读量622

点赞数

分类专栏： spark 文章标签： hadoop spark

本文链接：https://blog.youkuaiyun.com/qq_35613461/article/details/52169273

版权

spark 专栏收录该内容

2 篇文章

订阅专栏

参考文档：
http://spark.apache.org/docs/latest/running-on-yarn.html

一、在安装hadoop2.7.2

参考：
http://blog.youkuaiyun.com/qq_35613461/article/details/52125795

二、安装Spark2.0

hadoop2.6.x与spark1.6兼容
hadoop2.7.x与spark2.0兼容
1、获取spark2.0:

http://spark.apache.org/downloads.html
http://d3kbcqa49mib13.cloudfront.net/spark-2.0.0-bin-hadoop2.7.tgz

2、获取scala：

http://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz

3、上传安装文件到服务器，用winscp这个工具非常方便
WINSCP
4、解压scala、spark

[hadoop@master ~]$ pwd
/home/hadoop
[hadoop@master ~]$ tar xvf scala-2.11.8.tgz
[hadoop@master ~]$ tar xvzf spark-2.0.0-bin-hadoop2.7.tgz
[hadoop@master ~]$ ln -s scala-2.11.8 scala
[hadoop@master ~]$ ln -s spark-2.0.0-bin-hadoop2.7 spark
[hadoop@master ~]$ ls -l
[hadoop@master ~]$ ll
total 12
lrwxrwxrwx  1 hadoop hadoop   13 Aug  4 00:41 hadoop2 -> hadoop-2.7.2/
drwxr-xr-x 10 hadoop hadoop 4096 Aug  4 02:28 hadoop-2.7.2
lrwxrwxrwx  1 hadoop hadoop   12 Aug  9 18:13 scala -> scala-2.11.8
drwxrwxr-x  6 hadoop hadoop 4096 Mar  4 07:30 scala-2.11.8
lrwxrwxrwx  1 hadoop hadoop   25 Aug  9 18:20 spark -> spark-2.0.0-bin-hadoop2.7
drwxr-xr-x 13 hadoop hadoop 4096 Aug  9 18:56 spark-2.0.0-bin-hadoop2.7

5、修改spark-en.sh 、slaves、log4j.properties

[hadoop@master ~]$  mv spark-env.sh.template spark-env.sh        
[hadoop@master ~]$  mv slaves.template slaves                    
[hadoop@master ~]$  mv log4j.properties.template log4j.properties

[hadoop@master ~]$ vi spark/conf/spark-env.sh 
## 添加如下内容：
export HADOOP_CONF_DIR=/home/hadoop/hadoop2/etc/hadoop
export SCALA_HOME=/home/hadoop/scala
export HADOOP_HOME=/home/hadoop/hadoop2

[hadoop@master conf]$ more slaves 
# A Spark Worker will be started on each of the machines listed below.
slave1
slave2

6、启动Spark2.0
[hadoop@master ~]$ /home/hadoop/spark/sbin/start-all.sh

[hadoop@master ~]$ jps
5334 Master
4657 NameNode
5012 ResourceManager
4818 SecondaryNameNode
6141 Jps
[hadoop@master ~]$

[hadoop@slave1 ~]$ jps
2868 Jps
2820 Worker
2649 NodeManager
2529 DataNode
[hadoop@slave1 ~]$

[hadoop@slave2 ~]$ jps
4588 NodeManager
4469 DataNode
4811 Jps
4763 Worker
[hadoop@slave2 ~]$

7、监控界面
http://192.168.150.30:8080/

8、测试spark

[hadoop@master ~]$ /home/hadoop/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --queue default ../examples/jars/spark-examples_2.11-2.0.0.jar

[hadoop@master ~]$ /home/hadoop/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --queue default /home/hadoop/spark/examples/jars/spark-examples_2.11-2.0.0.jar
[hadoop@master ~]$ /home/hadoop/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --queue default /home/hadoop/spark/examples/jars/spark-examples_2.11-2.0.0.jar
16/08/09 20:23:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/08/09 20:23:40 INFO RMProxy: Connecting to ResourceManager at master/192.168.150.30:8032
16/08/09 20:23:40 INFO Client: Requesting a new application from cluster with 2 NodeManagers
16/08/09 20:23:40 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (2000 MB per container)
16/08/09 20:23:40 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
16/08/09 20:23:40 INFO Client: Setting up container launch context for our AM
16/08/09 20:23:40 INFO Client: Setting up the launch environment for our AM container
16/08/09 20:23:40 INFO Client: Preparing resources for our AM container
16/08/09 20:23:41 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
16/08/09 20:23:43 INFO Client: Uploading resource file:/tmp/spark-d3674319-c85c-4256-b8ab-f0728fc9611e/__spark_libs__8560551026056083403.zip -> hdfs://master:9000/user/hadoop/.sparkStaging/application_1470793690302_0008/__spark_libs__8560551026056083403.zip
16/08/09 20:23:48 INFO Client: Uploading resource file:/home/hadoop/spark/examples/jars/spark-examples_2.11-2.0.0.jar -> hdfs://master:9000/user/hadoop/.sparkStaging/application_1470793690302_0008/spark-examples_2.11-2.0.0.jar
16/08/09 20:23:48 INFO Client: Uploading resource file:/tmp/spark-d3674319-c85c-4256-b8ab-f0728fc9611e/__spark_conf__5039562237723981219.zip -> hdfs://master:9000/user/hadoop/.sparkStaging/application_1470793690302_0008/__spark_conf__.zip
16/08/09 20:23:48 INFO SecurityManager: Changing view acls to: hadoop
16/08/09 20:23:48 INFO SecurityManager: Changing modify acls to: hadoop
16/08/09 20:23:48 INFO SecurityManager: Changing view acls groups to: 
16/08/09 20:23:48 INFO SecurityManager: Changing modify acls groups to: 
16/08/09 20:23:48 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
16/08/09 20:23:48 INFO Client: Submitting application application_1470793690302_0008 to ResourceManager
16/08/09 20:23:48 INFO YarnClientImpl: Submitted application application_1470793690302_0008
16/08/09 20:23:49 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED)
16/08/09 20:23:49 INFO Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1470799428761
         final status: UNDEFINED
         tracking URL: http://master:8088/proxy/application_1470793690302_0008/
         user: hadoop
16/08/09 20:23:50 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED)
16/08/09 20:23:51 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED)
16/08/09 20:23:52 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED)
16/08/09 20:23:53 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED)
16/08/09 20:23:54 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED)
16/08/09 20:23:55 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED)
16/08/09 20:23:56 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED)
16/08/09 20:23:57 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED)
16/08/09 20:23:58 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED)
16/08/09 20:23:59 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED)
16/08/09 20:24:00 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:00 INFO Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 192.168.150.32
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1470799428761
         final status: UNDEFINED
         tracking URL: http://master:8088/proxy/application_1470793690302_0008/
         user: hadoop
16/08/09 20:24:01 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:02 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:03 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:04 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:05 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:06 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:07 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:08 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:09 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:10 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:11 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:12 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:13 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:14 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:15 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING)
16/08/09 20:24:16 INFO Client: Application report for application_1470793690302_0008 (state: FINISHED)
16/08/09 20:24:16 INFO Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 192.168.150.32
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1470799428761
         final status: SUCCEEDED
         tracking URL: http://master:8088/proxy/application_1470793690302_0008/
         user: hadoop
16/08/09 20:24:16 INFO ShutdownHookManager: Shutdown hook called
16/08/09 20:24:16 INFO ShutdownHookManager: Deleting directory /tmp/spark-d3674319-c85c-4256-b8ab-f0728fc9611e
[hadoop@master ~]$