spark安装(win7单机模式)

最新推荐文章于 2022-06-21 19:32:36 发布

iteye_878

最新推荐文章于 2022-06-21 19:32:36 发布

阅读量314

点赞数

分类专栏： Hadoop 文章标签： spark

本文链接：https://blog.youkuaiyun.com/iteye_878/article/details/82638858

版权

Hadoop 专栏收录该内容

6 篇文章

订阅专栏

[i][b]2016/06/16更新[/b][/i]
windows环境下需要额外下载winutils

1.环境
首先要注意不同版本的spark需要不同版本的scala,python。
我这里试验成功的版本如下：

Windows 7
JDK 1.7.0_72
scala 2.10.5
python 2.7.8
spark 1.4.1
winutils

2.下载

2.1 下载spark+hadoop
去[url=http://spark.apache.org/downloads.html]官网[/url]下载
选择如下
Choose a Spark release: 1.4.1
Choose a package type: pre-built for hadoop 2.6 and later
Choose a download type:随意
Download Spark: spark-1.4.1-bin-hadoop2.6.tgz

下载完解压。

2.2 下载winutils,hadoop.dll
另外要在windows上成功运行，需要下载[url=http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe]winutils[/url]
(文末附件也有下载)

将winutils.exe,hadoop.dll复制到spark-1.4.1-bin-hadoop2.6\bin目录下。
设置环境变量HADOOP_HOME = spark-1.4.1-bin-hadoop2.6根目录

3.shell测试
以下shell会输出部分错误信息，但不影响主要功能。

3.1 python shell

D:\opensource\hadoop\spark-1.4.1-bin-hadoop2.6>[color=blue]bin\pyspark[/color]

>>> [color=blue]lines = sc.textFile("README.md")[/color]
>>> [color=blue]lines.count()[/color]
[Stage 0:> (0 + 2) / 2]

98
>>> [color=blue]lines.first()[/color]
u'# Apache Spark'
>>>

3.2 scala shell

D:\opensource\hadoop\spark-1.4.1-bin-hadoop2.6>[color=blue]bin\spark-shell[/color]

scala> [color=blue]val lines = sc.textFile("README.md")[/color]
lines: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile at <console>:21

scala> [color=blue]lines.count()[/color]
res0: Long = 98

scala> [color=blue]lines.first()[/color]
res1: String = # Apache Spark

scala>

3.3 查看SparkUI
以上2种shell任意打开一种，然后浏览http://localhost:4040即可。

4. 例子程序——PI计算

D:\opensource\hadoop\spark-1.4.1-bin-hadoop2.6>[color=blue]bin\run-example org.apache.spark.examples.SparkPi[/color]
16/06/16 15:42:54 WARN NativeCodeLoader: Unable to load native-hadoop library fo
r your platform... using builtin-java classes where applicable
[Stage 0:> (0 + 2) / 2]
[Stage 0:=============================> (1 + 1) / 2]

Pi is roughly 3.1416

5. 可独立运行的java程序
《Learning Spark》一书的源码
[url]https://github.com/databricks/learning-spark[/url]
编译以后运行

D:\opensource\hadoop\spark-1.4.1-bin-hadoop2.6>[color=blue]bin\spark-submit --class com.oreilly.learningsparkexamples.java.WordCount ./java-0.0.2.jar local ./README.md ./wordCount.txt[/color]
16/06/30 16:48:54 WARN SparkConf: null jar passed to SparkContext constructor
16/06/30 16:48:54 WARN NativeCodeLoader: Unable to load native-hadoop library fo
r your platform... using builtin-java classes where applicable

结果在spark根目录下生成了wordCount.txt文件夹，里面有WordCount结果。