spark读取hdfs异常记录

测试代码如下:

   val linesRDD = sc.textFile("hdfs://192.168.89.60:9000/tmp/wc2")


   val wordsRDD = linesRDD.flatMap { _.split(" ") }


   val pairRDD = wordsRDD.map { (_,1) }


   val resultRDD = pairRDD.reduceByKey(_+_)


   resultRDD.sortBy(_._2).foreach(println)
   


执行到action算子后报错如下


异常:
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "h8.xxhadoop.com/192.168.89.60"; destination host is: "h8.xxhadoop.com":9000;


Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.




排查问题:


[root@h8 ~]# hadoop fs -ls hdfs://192.168.89.60:9000/tmp/wc2
ls: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "h8.xxhadoop.com/192.168.89.60"; destination host is: "h8.x'xhadoop.com":9000; 


[root@h8 ~]# hdfs getconf -confKey fs.default.name
17/11/23 16:07:29 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS hdfs://h12.xxhadoop.com:8020
[root@h8 ~]# hadoop fs -ls hdfs://h12.xxhadoop.com:8020/tmp/wc2
-rw-r--r--   3 root supergroup        118 2017-10-20 10:04 hdfs://h12.xxhadoop.com:8020/tmp/wc2




解决:


找到是访问端口问题将
val linesRDD = sc.textFile("hdfs://192.168.89.60:9000/tmp/wc2")
改为
val linesRDD = sc.textFile("hdfs://h12.kkhadoop.com:8020/tmp/wc2")