rdd.collect()后得到的List排序问题：Exception in thread “main“ java.lang.UnsupportedOperationException

最新推荐文章于 2024-09-11 22:33:50 发布

原创最新推荐文章于 2024-09-11 22:33:50 发布 · 584 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #java

该博客介绍了如何利用Apache Spark处理Apache日志文件，按小时统计访问量。通过Java API实现数据读取、切分、转换为键值对、求和并排序，最终成功输出按访问小时的访问次数。代码运行过程中遇到UnsupportedOperationException问题，通过调整代码解决了此问题，并展示了部分处理后的结果。

跟着视频的写了一个数据统计的demo，实现按小时统计访问量。代码如下

package com.billy.test;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;
import scala.Tuple2;

import java.util.*;

public class SparkTest {

    public static void main(String[] args) {
        SparkConf sparkConf = new SparkConf().setMaster("local[*]").setAppName("RDD");
        JavaSparkContext sc = new JavaSparkContext(sparkConf);
        JavaRDD<String> lines = sc.textFile("files/apache.log");

        //数据分片
        JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {

            @Override
            public Iterator<String> call(String s) throws Exception {
                return Arrays.asList(s.split(":")[1]).iterator();
            }
        });
        //数据转化为 对
        JavaPairRDD<String,Integer> wordsMap = words.mapToPair(new PairFunction<String, String, Integer>() {
            @Override
            public Tuple2<String, Integer> call(String s) throws Exception {
                return new Tuple2(s,1);
            }
        });
        //求和
        JavaPairRDD<String,Integer> wordsReduce = wordsMap.reduceByKey(new Function2<Integer, Integer, Integer>() {
            @Override
            public Integer call(Integer arg0, Integer arg1) throws Exception {
                return arg0 + arg1;
            }
        });
        //JavaPairRDD<String, Integer> sortByKey = wordsReduce.sortByKey(true);
        List<Tuple2<String, Integer>> collect = wordsReduce.collect();
        System.out.println("collect0: " + collect);

        //collect = new ArrayList(collect);
        System.out.println("collect1: " + collect);

        //System.out.println(sorted);
        Comparator<? super Tuple2<String, Integer>> myComparator = new Comparator<>() {
            @Override
            public int compare(Tuple2<String, Integer> t1, Tuple2<String, Integer> t2) {
                return t1._2.compareTo(t2._2);
            }
        };
        collect.sort(myComparator);
        System.out.println("collect2: " + collect);
    }
}

结果一直报Exception in thread “main” java.lang.UnsupportedOperationException错误，后来参考文章
https://blog.youkuaiyun.com/Tracycater/article/details/77592472?locationNum=2&fps=1
发现由数组转为List有一些坑，然后看rdd.collect()方法的源码发现果然collect返回的是一个array
在这里插入图片描述
因此也需要将collect得到的List转化一下，在上面的代码中，打开
collect = new ArrayList(collect);的注释，然后就不报错了。
打印结果：

collect0: [(06,366), (20,486), (19,493), (15,496), (00,361), (02,365), (04,355), (22,346), (17,484), (13,475), (11,459), (08,345), (14,498), (09,364), (21,453), (18,478), (16,473), (03,354), (07,357), (12,462), (05,371), (01,360), (10,443), (23,356)]
collect1: [(06,366), (20,486), (19,493), (15,496), (00,361), (02,365), (04,355), (22,346), (17,484), (13,475), (11,459), (08,345), (14,498), (09,364), (21,453), (18,478), (16,473), (03,354), (07,357), (12,462), (05,371), (01,360), (10,443), (23,356)]
collect2: [(08,345), (22,346), (03,354), (04,355), (23,356), (07,357), (01,360), (00,361), (09,364), (02,365), (06,366), (05,371), (10,443), (21,453), (11,459), (12,462), (16,473), (13,475), (18,478), (17,484), (20,486), (19,493), (15,496), (14,498)]

apache.log摘要如下：

83.149.9.216 - - 17/05/2015:10:05:03 +0000 GET /presentations/logstash-monitorama-2013/images/kibana-search.png
83.149.9.216 - - 17/05/2015:10:05:43 +0000 GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png
83.149.9.216 - - 17/05/2015:10:05:47 +0000 GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js
83.149.9.216 - - 17/05/2015:10:05:12 +0000 GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js
83.149.9.216 - - 17/05/2015:10:05:07 +0000 GET /presentations/logstash-monitorama-2013/plugin/notes/notes.js
83.149.9.216 - - 17/05/2015:10:05:34 +0000 GET /presentations/logstash-monitorama-2013/images/sad-medic.png
83.149.9.216 - - 17/05/2015:10:05:57 +0000 GET /presentations/logstash-monitorama-2013/css/fonts/Roboto-Bold.ttf
83.149.9.216 - - 17/05/2015:10:05:50 +0000 GET /presentations/logstash-monitorama-2013/css/fonts/Roboto-Regular.ttf
83.149.9.216 - - 17/05/2015:10:05:24 +0000 GET /presentations/logstash-monitorama-2013/images/frontend-response-codes.png
83.149.9.216 - - 17/05/2015:10:05:50 +0000 GET /presentations/logstash-monitorama-2013/images/kibana-dashboard.png
83.149.9.216 - - 17/05/2015:10:05:46 +0000 GET /presentations/logstash-monitorama-2013/images/Dreamhost_logo.svg
83.149.9.216 - - 17/05/2015:10:05:11 +0000 GET /presentations/logstash-monitorama-2013/images/kibana-dashboard2.png
83.149.9.216 - - 17/05/2015:10:05:19 +0000 GET /presentations/logstash-monitorama-2013/images/apache-icon.gif
83.149.9.216 - - 17/05/2015:10:05:33 +0000 GET /presentations/logstash-monitorama-2013/images/nagios-sms5.png
83.149.9.216 - - 17/05/2015:10:05:00 +0000 GET /presentations/logstash-monitorama-2013/images/redis.png
83.149.9.216 - - 17/05/2015:10:05:25 +0000 GET /presentations/logstash-monitorama-2013/images/elasticsearch.png
83.149.9.216 - - 17/05/2015:10:05:59 +0000 GET /presentations/logstash-monitorama-2013/images/logstashbook.png
83.149.9.216 - - 17/05/2015:10:05:30 +0000 GET /presentations/logstash-monitorama-2013/images/github-contributions.png
83.149.9.216 - - 17/05/2015:10:05:53 +0000 GET /presentations/logstash-monitorama-2013/css/print/paper.css
83.149.9.216 - - 17/05/2015:10:05:24 +0000 GET /presentations/logstash-monitorama-2013/images/1983_delorean_dmc-12-pic-38289.jpeg
83.149.9.216 - - 17/05/2015:10:05:54 +0000 GET /presentations/logstash-monitorama-2013/images/simple-inputs-filters-outputs.jpg
83.149.9.216 - - 17/05/2015:10:05:33 +0000 GET /presentations/logstash-monitorama-2013/images/tiered-outputs-to-inputs.jpg
83.149.9.216 - - 17/05/2015:10:05:56 +0000 GET /favicon.ico
24.236.252.67 - - 17/05/2015:10:05:40 +0000 GET /favicon.ico
93.114.45.13 - - 17/05/2015:10:05:14 +0000 GET /articles/dynamic-dns-with-dhcp/
93.114.45.13 - - 17/05/2015:10:05:04 +0000 GET /reset.css
93.114.45.13 - - 17/05/2015:10:05:45 +0000 GET /style2.css
93.114.45.13 - - 17/05/2015:10:05:14 +0000 GET /favicon.ico
93.114.45.13 - - 17/05/2015:10:05:17 +0000 GET /images/jordan-80.png
93.114.45.13 - - 17/05/2015:10:05:21 +0000 GET /images/web/2009/banner.png