Spark 获取文本内最大的前3个数字实例

最新推荐文章于 2024-10-01 00:36:07 发布

weixin_34033624

最新推荐文章于 2024-10-01 00:36:07 发布

阅读量282

点赞数

CC 4.0 BY-SA版权

文章标签：大数据 java python

原文链接：https://my.oschina.net/hehongbo/blog/1486434

本文介绍使用Apache Spark进行数据排序并获取前三个最大值的方法。通过实例演示如何读取文件，利用JavaPairRDD进行键值对转换，再进行排序及输出结果。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

2019独角兽企业重金招聘Python工程师标准>>>

package cn.hhb.spark.core;

import org.apache.spark.Accumulator;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.api.java.function.VoidFunction;
import scala.Tuple2;

import java.util.Arrays;
import java.util.List;

/**
 * Created by dell on 2017/7/13.
 */
public class Top3 {
    public static void main(String[] args) {
        SparkConf conf = new SparkConf()
                .setAppName("AccumulatorVariable")
                .setMaster("local")
                .set("spark.testing.memory", "2147480000");

        JavaSparkContext sc = new JavaSparkContext(conf);

        JavaRDD<String> lines = sc.textFile("C://number.log");
        JavaPairRDD<Integer, String> pairs = lines.mapToPair(new PairFunction<String, Integer, String>() {
            @Override
            public Tuple2<Integer, String> call(String s) throws Exception {
                return new Tuple2<Integer, String>(Integer.valueOf(s), s);
            }
        });

        JavaPairRDD<Integer, String> sortPairs = pairs.sortByKey(false);
        JavaRDD<Integer> sortNumbers = sortPairs.map(new Function<Tuple2<Integer, String>, Integer>() {
            @Override
            public Integer call(Tuple2<Integer, String> v1) throws Exception {
                return v1._1;
            }
        });

        sortNumbers.foreach(new VoidFunction<Integer>() {
            @Override
            public void call(Integer v1) throws Exception {
                System.out.println(v1);
            }
        });

        List<Integer> sortedNumberList = sortNumbers.take(3);
        for (Integer number : sortedNumberList){
            System.out.println(number);
        }

        sc.close();
    }
}

转载于:https://my.oschina.net/hehongbo/blog/1486434