spark1.6+hadoop2.6+kafka2.10-0.8.2.1+zookeeper3.3.6安装及sparkStreaming代码编写和调试

最新推荐文章于 2021-08-17 10:38:53 发布

qq_16563637

最新推荐文章于 2021-08-17 10:38:53 发布

阅读量667

点赞数 1

CC 4.0 BY-SA版权

分类专栏： spark 文章标签： sparkStreaming

本文链接：https://blog.youkuaiyun.com/qq_16563637/article/details/82903432

spark 专栏收录该内容

11 篇文章

订阅专栏

本文详细介绍了在CentOS环境下搭建Hadoop、Spark和Kafka集群的过程，包括软件版本选择、Docker安装配置、环境变量设置及测试验证等关键步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

安装环境
安装之前确保设备至少有4GB内存,推荐8GB
centos7.2
docker(这个安装请参考我的另一篇博客https://blog.youkuaiyun.com/qq_16563637/article/details/81699251)

目标安装软件	目标安装版本	实际安装版本
hadoop	2.6	2.6
spark	1.6	1.6
kafka	2.10-0.8.2.1	2.10-0.8.2.1
zookeeper	3.3.3	3.3.6

说明因为kafka要配置HOST_NAME,HOST_NAME必须是宿主机IP地址,否则消费者程序会根据容器IP地址寻找kafka,所以kafka不能使用docker安装
本人亲自测试能够正常使用
-------------------------------------------开始安装spark和hadoop------------------------------------------------------
下载spark(说明这个说明:spark安装时会自动安装hadoop,不用再单独安装hadoop)

docker pull registry.docker-cn.com/sequenceiq/spark:1.6.0

docker运行

docker run -it -p 4040:4040 -p 7077:7077 -p 8088:8088 -p 8081:8081 -p 8080:8080 -p 8042:8042 -p 8030:8030 -p 8031:8031 -p 8040:8040 -p 9000:9000 -p 49707:49707 -p 50010:50010 -p 50070:50070 -p 50075:50075 -p 50020:50020 -p 50090:50090 --name spark --rm sequenceiq/spark:1.6.0 /bin/bash

设置spark
进入容器(docker run运行后会直接进入容器,该步骤可以省略)
docker exec -it 容器名 /bin/bash

docker exec -it 05d499dd260f /bin/bash
cd /usr/local/spark-1.6.0-bin-hadoop2.6
cd conf
cp spark-env.sh.template spark-env.sh
vi spark-env.sh

在最底部添加

export JAVA_HOME=/usr/java/jdk1.7.0_51
export SPARK_MASTER_PORT=7077

保存

cp slaves.template slaves
vi slaves

去掉localhost
添加192.168.1.103
保存

cd ../sbin
./stop-all.sh
./start-master.sh
./start-slave.sh 192.168.1.103:7077 --webui-port 8081

为了正常使用配置环境变量,进入容器中设置

docker exec -it 05d499dd260f /bin/bash
vi /etc/profile
export SPARK_HOME=/usr/local/spark-1.6.0-bin-hadoop2.6
export HADOOP_HOME="/usr/local/hadoop-2.6.0"
export PATH=$PATH:$SPARK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

保存:wq
使配置文件生效

source /etc/profile

测试spark是否安装OK
执行下面命令(该算法是利用蒙特·卡罗算法求PI)

/usr/local/spark-1.6.0-bin-hadoop2.6/bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://192.168.1.103:7077 \
--executor-memory 1G \
--total-executor-cores 2 \
/usr/local/spark-1.6.0-bin-hadoop2.6/lib/spark-examples-1.6.0-hadoop2.6.0.jar \
100

如果没有报错说明spark安装正常
打开spark-shell

spark-shell

出现 scala> 说明安装正常
开始检查hadoop是否正常
ctrl+C退出spark-shell
查看命令行是否能用

hadoop version

如果输出版本信息继续输入

cd /usr/local/hadoop-2.6.0
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'

如果输出mapreduce程序执行说明安装正常
查看spark 管控台
http://192.168.1.103:8080/
查看hadoop 管控台
http://192.168.1.103:50070
结束
------------------------------------------------zookeeper的docker安装------------------------------------------------------
说明:kafka_2.10-0.8.2.1推荐的zookeeper版本为3.3.3,然而zookeeper官方镜像没有3.3.3,选用3.3.6安装

docker pull zookeeper:3.3.6

启动zookeeper容器

docker run -d --name zookeeper -p 2181:2181 -t zookeeper:3.3.6

zookeeper安装完成
------------------------------------------------kafka本地安装--------------------------------------------------------------
先安装jdk(可参考我的另一篇博客https://blog.youkuaiyun.com/qq_16563637/article/details/81738113)
安装成功后下载kafka_2.10-0.8.2.1.tgz
kafka下载地址:https://archive.apache.org/dist/kafka/0.8.2.1/kafka_2.10-0.8.2.1.tgz
将文件上传至服务器解压缩

tar zxf kafka_2.10-0.8.2.1.tgz

修改配置文件

cd /root/kafka_2.10-0.8.2.1/config
vi server.properties

修改下面几项内容

host.name=192.168.1.103
log.dirs=/root/kafka_2.10-0.8.2.1/logs
zookeeper.connect=192.168.1.103:2181

保存

cd ..

启动kafka(前台启动)
建议先前台启动观看日志没有报错,ctrl+c退出,再后台启动

bin/kafka-server-start.sh  config/server.properties

启动kafka(后台启动)

bin/kafka-server-start.sh  config/server.properties > /dev/null 2>&1 &

创建topic(此处partitions数量不能大于broker,replication-factor 为副本数量)

bin/kafka-topics.sh --create --zookeeper 192.168.1.103:2181 --replication-factor 1 --partitions 1 --topic test

列出所有topic

bin/kafka-topics.sh --list --zookeeper 192.168.1.103:2181

向topic中写入数据

bin/kafka-console-producer.sh --broker-list 192.168.1.103:9092 --topic test

消费数据

bin/kafka-console-consumer.sh --zookeeper 192.168.1.103:2181 --topic test --from-beginning

查看指定topic的详情

bin/kafka-topics.sh --describe --zookeeper 192.168.1.103:2181 --topic test

停止kafka

kill -s TERM $(jps -l | grep 'kafka\.Kafka' | awk '{print $1}')

----------------------------------------------spark测试代码---------------------------------------------------------
说明:下面这段代码采用maven项目,请直接在本地启动运行,并且在Program arguments中写入下面内容

192.168.1.103:2181 g1 test 2

package cn.itcast.spark.day5

import org.apache.spark.storage.StorageLevel
import org.apache.spark.{HashPartitioner, SparkConf}
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{Seconds, StreamingContext}

/**
  * Created by root on 2016/5/21.
  */
//(多个zookeeper用,隔开)
//zookeper groupid topics numThreads
//入参:192.168.1.103:2181 g1 test 2
object KafkaWordCount {

  val updateFunc = (iter: Iterator[(String, Seq[Int], Option[Int])]) => {
    //iter.flatMap(it=>Some(it._2.sum + it._3.getOrElse(0)).map(x=>(it._1,x)))
    iter.flatMap { case (x, y, z) => Some(y.sum + z.getOrElse(0)).map(i => (x, i)) }
  }


  def main(args: Array[String]) {
    LoggerLevels.setStreamingLogLevels()
    val Array(zkQuorum, group, topics, numThreads) = args
    val sparkConf = new SparkConf().setAppName("KafkaWordCount").setMaster("local[2]")
    val ssc = new StreamingContext(sparkConf, Seconds(5))
    ssc.checkpoint("c://ck2")
    //"alog-2016-04-16,alog-2016-04-17,alog-2016-04-18"
    //"Array((alog-2016-04-16, 2), (alog-2016-04-17, 2), (alog-2016-04-18, 2))"
    val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
    val data = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap, StorageLevel.MEMORY_AND_DISK_SER)
    val words = data.map(_._2).flatMap(_.split(" "))
    val wordCounts = words.map((_, 1)).updateStateByKey(updateFunc, new HashPartitioner(ssc.sparkContext.defaultParallelism), true)
    wordCounts.print()
    ssc.start()
    ssc.awaitTermination()
  }
}

pom.xml如下

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>cn.itcast.spark</groupId>
    <artifactId>hello-spark</artifactId>
    <version>1.0</version>

    <properties>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <scala.version>2.10.6</scala.version>
        <spark.version>1.6.1</spark.version>
        <hadoop.version>2.6.4</hadoop.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka_2.10</artifactId>
            <version>1.6.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-flume_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.38</version>
        </dependency>

        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>2.8.1</version>
        </dependency>

    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.2.2</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                        <configuration>
                            <args>
                                <arg>-make:transitive</arg>
                                <arg>-dependencyfile</arg>
                                <arg>${project.build.directory}/.scala_dependencies</arg>
                            </args>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>


</project>