Spark Core学习.WordCount代码

最新推荐文章于 2023-06-06 14:17:06 发布
原创 最新推荐文章于 2023-06-06 14:17:06 发布 · 295 阅读 · 0 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#spark

Spark Core学习.WordCount代码

Spark Core的WordCount Java代码

方式一

import java.util.Arrays;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.api.java.function.VoidFunction;

import scala.Tuple2;

public class WordCount {

	public static void main(String[] args) {
		
		SparkConf sc = new SparkConf().setAppName("wordcount").setMaster("local[1]");
		JavaSparkContext context = new JavaSparkContext(sc);
		
		JavaRDD<String> input = context.textFile("wc");
		JavaRDD<String> words = input.flatMap(new FlatMapFunction<String, String>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Iterable<String> call(String line) throws Exception {
				return Arrays.asList(line.split(" "));
			}
		});
		JavaPairRDD<String, Integer> pairs = words.mapToPair(new PairFunction<String, String, Integer>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Tuple2<String, Integer> call(String word) throws Exception {
				return new Tuple2<String,Integer>(word, 1);
			}
		});
		JavaPairRDD<String, Integer> results = pairs.reduceByKey(new Function2<Integer, Integer, Integer>() {
			
			private static final long serialVersionUID = 1L;

			@Override
			public Integer call(Integer v1, Integer v2) throws Exception {
				return v1 + v2;
			}
		});
		results.foreach(new VoidFunction<Tuple2<String,Integer>>() {
			
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Tuple2<String, Integer> tuple) throws Exception {
				System.out.println(tuple);
			}
		});
		
		context.close();
	}
}

方式二

import java.util.Arrays;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.api.java.function.VoidFunction;

import scala.Tuple2;

public class WordCount2 {
	public static void main(String[] args) {
		SparkConf sc = new SparkConf().setAppName("wordcount").setMaster("local[1]");
		JavaSparkContext context = new JavaSparkContext(sc);
		
		JavaRDD<String> rdd1 = context.textFile("wc");
		JavaRDD<String> rdd2 = rdd1.flatMap(new FlatMapFunction<String, String>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Iterable<String> call(String line) throws Exception {
				return Arrays.asList(line.split(" "));
			}
		});
		JavaPairRDD<String, Integer> rdd3 = rdd2.mapToPair(new PairFunction<String, String, Integer>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Tuple2<String, Integer> call(String word) throws Exception {
				return new Tuple2<String,Integer>(word, 1);
			}
		});
		JavaPairRDD<String, Iterable<Integer>> rdd4=rdd3.groupByKey();
		JavaPairRDD<String, Integer> rdd5=rdd4.mapToPair(new PairFunction<Tuple2<String,Iterable<Integer>>, String, Integer>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Tuple2<String, Integer> call(
					Tuple2<String, Iterable<Integer>> t) throws Exception {
				int sum=0;
				for(Integer i:t._2()){
					sum=sum+i;
				}
				return new Tuple2<String, Integer>(t._1(), sum);
			}
		});
		rdd5.foreach(new VoidFunction<Tuple2<String,Integer>>() {
			
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Tuple2<String, Integer> t) throws Exception {
				System.out.println(t);
			}
		});
		
		
		context.close();

	}
}

方式三

import java.util.Arrays;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.api.java.function.VoidFunction;

import scala.Tuple2;

public class WordCount3 {
	public static void main(String[] args) {
		SparkConf sc = new SparkConf().setAppName("wordcount").setMaster("local[1]");
		JavaSparkContext context = new JavaSparkContext(sc);
		
		JavaRDD<String> rdd1 = context.textFile("wc");
		JavaRDD<String> rdd2 = rdd1.flatMap(new FlatMapFunction<String, String>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Iterable<String> call(String line) throws Exception {
				return Arrays.asList(line.split(" "));
			}
		});
		JavaPairRDD<String, Integer> rdd3 = rdd2.mapToPair(new PairFunction<String, String, Integer>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Tuple2<String, Integer> call(String word) throws Exception {
				return new Tuple2<String,Integer>(word, 1);
			}
		});
		
		JavaPairRDD<String, Integer> rdd4=rdd3.aggregateByKey(0, new Function2<Integer,Integer,Integer>(){

			private static final long serialVersionUID = 1L;

			@Override
			public Integer call(Integer v1, Integer v2)
					throws Exception {
				return v1+v2;
			}
		}, new Function2<Integer,Integer,Integer>(){

			private static final long serialVersionUID = 1L;

			@Override
			public Integer call(Integer v1, Integer v2)
					throws Exception {
				return v1+v2;
			}
		});
		
		rdd4.foreach(new VoidFunction<Tuple2<String,Integer>>() {
			
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Tuple2<String, Integer> t) throws Exception {
				System.out.println(t);
			}
		});
		
		context.close();
	}
}

方式四

import java.util.ArrayList;
import java.util.List;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFlatMapFunction;
import org.apache.spark.api.java.function.VoidFunction;

import scala.Tuple2;

public class WordCount4 {
	public static void main(String[] args) {
		SparkConf sc = new SparkConf().setAppName("wordcount").setMaster("local[1]");
		JavaSparkContext context = new JavaSparkContext(sc);
		
		JavaRDD<String> rdd1 = context.textFile("wc");
		
		JavaPairRDD<String, Integer> rdd2=rdd1.flatMapToPair(new PairFlatMapFunction<String, String, Integer>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Iterable<Tuple2<String, Integer>> call(String t)
					throws Exception {
				List<Tuple2<String,Integer>> list=new ArrayList<Tuple2<String,Integer>>();
				String[] arrs=t.split(" ");
				for(String s:arrs){
					list.add(new Tuple2<String, Integer>(s, 1));
				}
				return list;
			}
		});
		
		JavaPairRDD<String, Integer> rdd3=rdd2.reduceByKey(new Function2<Integer, Integer, Integer>() {
			
			private static final long serialVersionUID = 1L;

			@Override
			public Integer call(Integer v1, Integer v2) throws Exception {
				return v1+v2;
			}
		});
		
		rdd3.foreach(new VoidFunction<Tuple2<String,Integer>>() {
			
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Tuple2<String, Integer> t) throws Exception {
				System.out.println(t);
			}
		});
		
		context.close();
	}
}

方式五

import java.util.Arrays;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.api.java.function.VoidFunction;

import scala.Tuple2;

public class WordCount5 {
	public static void main(String[] args) {
		SparkConf sc = new SparkConf().setAppName("wordcount").setMaster("local[1]");
		JavaSparkContext context = new JavaSparkContext(sc);
		
		JavaRDD<String> rdd1 = context.textFile("wc");
		JavaRDD<String> rdd2 = rdd1.flatMap(new FlatMapFunction<String, String>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Iterable<String> call(String line) throws Exception {
				return Arrays.asList(line.split(" "));
			}
		});
		JavaPairRDD<String, Integer> rdd3 = rdd2.mapToPair(new PairFunction<String, String, Integer>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Tuple2<String, Integer> call(String word) throws Exception {
				return new Tuple2<String,Integer>(word, 1);
			}
		});
		JavaPairRDD<String, Integer> rdd4=rdd3.foldByKey(0, new  Function2<Integer, Integer, Integer>(){

			private static final long serialVersionUID = 1L;

			@Override
			public Integer call(Integer v1, Integer v2) throws Exception {
				return v1+v2;
			}
		});
		rdd4.foreach(new VoidFunction<Tuple2<String,Integer>>() {
			
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Tuple2<String, Integer> t) throws Exception {
				System.out.println(t);
			}
		});
		context.close();
	}
}

## 方式六


import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFlatMapFunction;
import org.apache.spark.api.java.function.VoidFunction;

import scala.Tuple2;

public class WordCount6 {
public static void main(String[] args) {
		
		SparkConf sc = new SparkConf().setAppName("wordcount").setMaster("local[1]");
		JavaSparkContext context = new JavaSparkContext(sc);
		
		JavaRDD<String> input = context.textFile("wc");
		JavaRDD<String> words = input.flatMap(new FlatMapFunction<String, String>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Iterable<String> call(String line) throws Exception {
				return Arrays.asList(line.split(" "));
			}
		});
		JavaPairRDD<String, Integer> pairs=words.mapPartitionsToPair(new PairFlatMapFunction<Iterator<String>, String, Integer>() {

			private static final long serialVersionUID = 1L;

			@Override
			public Iterable<Tuple2<String, Integer>> call(Iterator<String> t)
					throws Exception {
				List<Tuple2<String,Integer>> list=new ArrayList<Tuple2<String,Integer>>();
				for(String s:ListUtil.copyIterator(t)){
					list.add(new Tuple2<String, Integer>(s, 1));
				}
				return list;
			}
		});
		JavaPairRDD<String, Integer> results = pairs.reduceByKey(new Function2<Integer, Integer, Integer>() {
			
			private static final long serialVersionUID = 1L;

			@Override
			public Integer call(Integer v1, Integer v2) throws Exception {
				return v1 + v2;
			}
		});
		results.foreach(new VoidFunction<Tuple2<String,Integer>>() {
			
			private static final long serialVersionUID = 1L;

			@Override
			public void call(Tuple2<String, Integer> tuple) throws Exception {
				System.out.println(tuple);
			}
		});
		
		context.close();
	}
}

总结

WordCount.java：运用flatMap、mapToPair、reduceByKey算子
WordCount2.java：运用flatMap、mapToPair、groupByKey算子
WordCount3.java：运用flatMap、mapToPair、aggregateByKey算子
WordCount4.java：运用flatMapToPair、reduceByKey算子
WordCount5.java：运用flatMap、mapToPair、foldByKey算子
WordCount6.java：运用flatMap、mapPartitionsToPair、reduceByKey算子

Spark之WordCount案例
QYHuiiQ
10-08 463
【代码】Spark之WordCount案例。
Spark深入解析（六）：SparkCore之Spark代码编写WordCount
wzc8961661的博客
04-24 3374
学习目标WordCount思路WordCount代码实现 WordCount思路 准备数据 将数据放在以下目录中 1.txt Hello World Hello Scala 2.txt Hello Spark 图解分析 说明： 1、本地读取两个文件 2、两个文件内的数据 3、将文件内的数据进行扁平化 4、将相同单词进行分组 5、聚合计算每个单词的个数 WordCoun...
Spark：用Scala和Java实现WordCount
01-30
为了在IDEA中编写scala，今天安装配置学习了IDEA集成开发环境。IDEA确实很优秀，学会之后，用起来很顺手。关于如何搭建scala和IDEA开发环境，请看文末的参考资料。用Scala和Java实现WordCount，其中Java实现的JavaWordCount是spark自带的例子（$SPARK_HOME/examples/src/main/java/org/apache/spark/examples/JavaWordCount.java）OS:RedHatEnterpriseLinuxServer release6.
sparkcore实现word count
LW_ICE
06-10 320
spark core几种实现word count的方式（基于spark3.1.2和scala2.12.13）： import org.apache.spark.{SparkConf, SparkContext} object WordCount { def main(args: Array[String]): Unit = { val conf = new SparkConf().setMaster("local[4]").setAppName("WordCount") val sc
SparkCoreWordCount
KayipLin的博客
08-29 150
sparkCore wordCount小应用
qq_38994739的博客
09-19 484
需求：按照value值降序 wordcount.map(x=>(x._2,x._1)).sortByKey(false ).map(x=>(x._2,x._1)).collect()需求：Top Key wordcount.map(x=>(x._2,x._1)).sortByKey(false ).map(x=>(x._2,x._1)).take(3)需求：Group Top K
Spark之wordcount程序（Java Scala）
weixin_34353714的博客
09-22 637
1.Java/** 使用java开发本地测试的wordcount程序 @author Administrator* */public class WordCountLocal { public static void main(String[] args) { // 编写Spark应用程序 // 本地执行，是可以执行在eclipse...
WordCount_Spark!_spark_wordcount_java_
10-01
总结，Spark上的WordCount程序是理解和学习Spark基础操作的好起点。通过这个简单的例子，我们可以深入理解Spark的工作原理，以及如何在Java环境中编写并执行Spark任务。在实际应用中，这些基本操作可以扩展到更复杂...
从spark.default.parallelism参数来看Spark并行度、并行计算任务概念
石榴姐yyds
04-18 7358
并行度= partition= task总数。但是同一时刻能处理的task数量由并行计算任务决定（CPU cores决定）。（Parallelism）指的是分布式数据集被划分为多少份，从而用于分布式计算。换句话说，并行度的 出发点是数据（），它明确了数据划分的粒度。并行度越高，数据的粒度越细，数据分片越 多，数据越分散。由此可见，像分区数量、分片数量、Partitions 这些概念都是并行度的 同义词。。
hadoop&spark使用教程.docx
最新发布
07-12
### Hadoop & Spark 使用教程详解 #### Hadoop 使用教程 ...综上所述，通过遵循上述步骤，您可以系统地学习Hadoop和Spark的相关知识，并逐渐成为一名熟练的用户。在实践中不断探索新功能，提高解决问题的能力。
Java实现Spark词配对Wordcount计数代码实现
05-24
使用Spark实现对一个文档中的每一行的单词进行词配对计数，要求去标点符号，将大写符号统一转化成为小写单词。
spark学习笔记（入门）——sparkcore实现wordcount的三种方式
qq_55906442的博客
07-19 391
spark：sparkcore实现wordcount的三种方式
spark wordcount
oDianZi1234567的博客
06-05 251
1.scala-sdk
spark的wordcount代码
zhangguangxu98的博客
10-27 1004
1、读取HDFS文件 scala> val lines=sc.textFile("hdfs://master-ubuntu:9000/wordcount/wordcount.txt") 2、压扁文件内容，以空格分割 scala> val rdd1=lines.flatMap(line=>line.split(" ")) 3、映射为（word,1)格式 scala> val rdd2=rdd1.map(word=>(word,1)) 4、对相同key的...
【Spark】第一个代码（WordCount）
HR的博客
12-07 493
Spark--->第一个代码（WordCount） import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} object Spark_WordCount { def main(args: Array[String]): Unit = { //todo 建立和spark框架的连接 val sparkConf = new SparkConf().setMaster("local
spark编写WordCount
weixin_65195823的博客
05-12 712
1.在hdfs的/user/spark/in/目录下建一个in.txt文件。2.在spark-shell中编写WordCount代码和运行。3>文件创建成功之后，将本地的in.txt文件上传至hdfs中。(2).在根目录创建in.txt文件并将其传到hdfs中。(1).我们需要现在hdfs中创建文件所需目录。4> 在hdfs中查看是否已经存在。2>通过pwd查看本地文件所在路径。在hdfs创建in.txt文件。2>也可通过命令创建多级目录。3>通过命令查看是否创建成功。1>首先创建txt文件。
[Spark Core] Spark Shell 实现 Word Count
weixin_34297704的博客
10-08 146
　　0. 说明 　　在 Spark Shell 实现 Word Count 　　RDD (Resilient Distributed dataset), 弹性分布式数据集。 　　示意图 　　1.实现 　　1.1 分步实现 # step 1 加载文档 val rdd1 = sc.textFile("file:///home/centos/...
spark入门WordCount代码解读
04-09 1998
WordCount代码解读 object WordCount extends App { //setMaster：指定spark的运行模式，这里指定为本地运行 //local[1]：指定线程数 val conf = new SparkConf().setMaster("local[1]").setAppName("WC") //创建SparkContext...
spark入门SparkCore 之WordCount (一)
weixin_43205308的博客
06-06 435
配置scala 可参考 WordCount2使用reduceByKey实现 配置因为打印会有很多日志文件，所以 配置log4j去除日志 配置前 配置后
Spark Core案例分析：WordCount与统计技巧
"Spark Core的学习笔记，主要涵盖了WordCount案例的详细执行流程，以及如何统计最受欢迎的老师topN的不同方法，包括自定义分区器的应用。此外，还涉及到根据IP计算归属地的问题。" Spark Core是Apache Spark的核心...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值