Spark Streaming词频统计实例

本实例旨在:通过Spark Streaming流式地处理一个数据服务从TCP套接字中接收到的数据。

一创建maven工程，引入相应依赖jar包

<properties>
        <scala.version>2.11.8</scala.version>
    </properties>

    <repositories>
        <repository>
            <id>repos</id>
            <name>Repository</name>
            <url>http://maven.aliyun.com/nexus/content/groups/public</url>
        </repository>
        <repository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>repos</id>
            <name>Repository</name>
            <url>http://maven.aliyun.com/nexus/content/groups/public</url>
        </pluginRepository>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>

    <dependencies>
        <!--Spark Core核心依赖-->
        <dependency> <!-- Spark ,依赖的Scala版本为Scala 2.12.x版本 (View all targets) 否则会报错
                           java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)[Ljava/lang/Object; -->
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.4.0</version>
            <scope>provided</scope><!--运行时提供,打包不添加,Spark集群已自带-->
        </dependency>
        <!-- Spark SQL核心依赖-->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.4.0</version>
            <scope>provided</scope><!--运行时提供,打包不添加,Spark集群已自带-->
        </dependency>
        <!-- Spark Streaming依赖包-->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>2.4.0</version>
            <scope>provided</scope><!--运行时提供,打包不添加,Spark集群已自带-->
        </dependency>
        <!-- 2.12.x需要与spark的2.12对应-->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <!-- 混合Scala/Java编译-->
            <plugin><!--Scala编译插件 -->
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                        <configuration>
                            <includes>
                                <include>**/*.scala</include>
                            </includes>
                        </configuration>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>1.8</source><!-- 设置Java源-->
                    <target>1.8</target>
                </configuration>
            </plugin>

            <plugin><!-- 将所有的依赖包打入同一个jar包-->
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <appendAssemblyId>false</appendAssemblyId>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef><!--jar包的后缀名-->
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <mainClass>org.jy.data.yh.bigdata.drools.scala.sparkstreaming.SparkStreamingWordsFrep</mainClass>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin><!--Maven打包插件 -->
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>2.4</version>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath><!--添加类路径-->
                            <!--设置程序的入口类-->
                            <mainClass>org.jy.data.yh.bigdata.drools.scala.sparkstreaming.SparkStreamingWordsFrep</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

二：Scala代码如下:

package org.jy.data.yh.bigdata.drools.scala.sparkstreaming

import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}

/**
  * Spark Streaming 数据流统计词频
  */
object SparkStreamingWordsFrep {
  def main(args: Array[String]): Unit = {
    // Spark 配置项
    val sparkConf = new SparkConf()
        .setAppName("SparkStreamingWordsFrep")
        .setMaster("spark://centoshadoop1:7077,centoshadoop2:7077")
    // 创建流式上下文
    val sparkStreamContext = new StreamingContext(sparkConf,Seconds(2))
    // 创建一个DStream,连接指定的hostname:port,比如localhost:9999
    val lines = sparkStreamContext.socketTextStream("centoshadoop1",9999)
    // 将接收到的每条信息分割成次词语
    val words = lines.flatMap(line =>{line.split(" ")})
    // 统计每个batch的词频
    val pairs = words.map(word =>(word,1))
    // 汇总词频
    val wordCounts = pairs.reduceByKey(_+_) // 将key相同的元组的value累积
    // 打印从DStream中生成的RDD的前10个元素到控制台
    wordCounts.print(10000)
    sparkStreamContext.start() // 开始计算
    sparkStreamContext.awaitTermination() //等待计算结束
  }

}

三,linux系统安装nmap-ncat软件

yum install nc

四,打包到spark集群上运行,命令如下

bin/spark-submit \

--class org.jy.data.yh.bigdata.drools.scala.sparkstreaming.SparkStreamingWordsFrep \

--num-executors 4 \

--driver-memory 2G \

--executor-memory 1g \

--executor-cores 1 \

--conf spark.default.parallelism=1000 \

/home/hadoop/tools/SSO-Scala-SparkStreaming-1.0-SNAPSHOT.jar

五,打开两个linux界面,在其中一个界面输入要统计的文本内容

如下图：

[hadoop@centoshadoop1 ~]$ nc -lk 9999
Spark streaming is an extension of the core Spark API

输出效果如下：