Spark- Spark常见问题及使用要点

 

合并字段

result.coalesce(1).write().option("mergeschema","true").format("json").save("result/");

Spark报错了一

Caused by: java.io.IOException: Unable to acquire 4194304 bytes of memory

解决方式

        sqlContext.setConf("spark.sql.tungsten.enabled", "false");

查询的字段和bean的内部成员一样的命名才能不报错

sqlContext.sql("select userId, userName,gender, age  from person").write().mode("append")
                .jdbc(url,tableName,properties);