spark java IDEA 读文件json

首先json 样式：

源介绍文档说的是， 要求每行是正确的json 类型

{"name":"jack","age":12}
{"name":"lili","age":22}
{"name":"cc","age":11}
{"name":"vv","age":13}
{"name":"lee","age":14

报错： java.lang.ArrayIndexOutOfBoundsException: 10582

原因 有人说是 jdk 1.8 造成的

解决办法：

pom .xml 添加

<dependency>
    <groupId>com.thoughtworks.paranamer</groupId>
    <artifactId>paranamer</artifactId>
    <version>2.8</version>
</dependency>

 

 Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the
referenced columns only include the internal corrupt record column