Spark RDD文件词频统计

原创已于 2022-06-12 23:12:11 修改 · 3.9k 阅读

17 ·

CC 4.0 BY-SA版权

文章标签：

#spark #scala #big data

于 2020-04-12 21:23:11 首次发布

Spark(RDD与DataFrame数据结构模型实战) 同时被 3 个专栏收录

4 篇文章

订阅专栏

Spark统计文本词频(Scala版)

1 篇文章

订阅专栏

Spark RDD(弹性分布式数据集)

1 篇文章

订阅专栏

本文介绍如何使用Apache Spark Streaming处理文本文件，通过flatMap、map和reduceByKey操作统计单词频率。在本地模式下运行Spark应用程序，并展示如何配置Maven工程、构建Spark任务及输出结果。

一:准备sparkStreamingWordFrep.txt文本文件,内容如下:

this is a processing of the sparkStreaming data learn use I can process spark it big streming
data learn use I can process spark it big streming
to want I can data learn use I can process spark it big streming

二,搭建maven管理的工程

<properties>
        <scala.version>2.11.8</scala.version>
    </properties>

    <repositories>
        <repository>
            <id>repos</id>
            <name>Repository</name>
            <url>http://maven.aliyun.com/nexus/content/groups/public</url>
        </repository>
        <repository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>repos</id>
            <name>Repository</name>
            <url>http://maven.aliyun.com/nexus/content/groups/public</url>
        </pluginRepository>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-Tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>

    <dependencies>
        <dependency> <!-- Spark ,依赖的Scala版本为Scala 2.12.x版本 (View all targets) 否则会报错
                           java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)[Ljava/lang/Object; -->
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.4.0</version>
        </dependency>
        <dependency> <!-- Spark -->
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.4.0</version>
        </dependency>
        <!-- 2.12.x需要与spark的2.12对应-->
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>src/main/scala</sourceDirectory>
        <testSourceDirectory>src/test/scala</testSourceDirectory>
        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <version>2.15.2</version>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                        <configuration>
                            <includes>
                                <include>**/*.scala</include>
                            </includes>
                        </configuration>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>

            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <appendAssemblyId>false</appendAssemblyId>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <mainClass>org.jy.data.yh.bigdata.drools.scala.sparkstreaming.SparkStreamingWordFreq</mainClass>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>2.4</version>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <mainClass>org.jy.data.yh.bigdata.drools.scala.sparkstreaming.SparkStreamingWordFreq</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

三：完整代码如下

package org.jy.data.yh.bigdata.drools.scala.sparkstreaming
import org.apache.spark.{SparkConf, SparkContext}

/**
 *  Spark 文本单词的计数: 从文本中读取数据,利用空格进行数据切分后,按照<词，1>的key,value映射后，
  * 最终以key为reduce对象，整合每个单词出现的次数。
  * 
  * 利用flagMap以空格为切分将输入的文本映射为一个向量,之后用map将每个元素映射为(元素,词频)
  * 的二元组,最后以每个词元素为key 进行reduce合并操作,从而统计出每个单词出现的词频
  * (注：这一步是分散在集群当中的所有Worker中执行的,即每个Worker可能只计算了一小部分)
 *
 */
object SparkStreamingWordFreq {
    def main(args :Array[String]):Unit = {
      // 创建SparkConf对象
      val sparkConf = new SparkConf();
      sparkConf.setAppName("SparkStreamingWordFreq") // 设置应用名称,该名称在Spark Web Ui中显示
      sparkConf.setMaster("local[*]") // 设置本地模式
      // 创建SparkContext对象
      val sparkContext = new SparkContext(sparkConf);
      // 数据源为文本文件
      val txtFile = "D://jar/sparkStreamingWordFrep.txt"
      // 读取文本文件的内容
      val txtData = sparkContext.textFile(txtFile);
      // 缓存文本RDD
      txtData.cache()
      // 计数
      txtData.count()
      // 以空格分割进行词频统计
      val wcData = txtData.flatMap{line => line.split(" ")}
          .map(word => (word,1))
          .reduceByKey(_+_)
          // 汇总RDD信息并打印
      wcData.collect() // 返回一个数组
        .foreach(e=>{println(e)})
      sparkContext.stop()

    }
}

四,在IntelliJ IDEA中运行该程序,输出统计结果如下：

(learn,3)
(this,1)
(is,1)
(can,4)
(big,3)
(data,3)
(,1)
(want,1)
(it,3)
(spark,3)
(process,3)
(a,1)
(streming,3)
(processing,1)
(sparkStreaming,1)
(I,4)
(to,1)
(of,1)
(use,3)
(the,1)