性能

性能问题

for 循环大的list或者map 嵌套小的循环 => 循环小的 嵌套大的

去重 用hashset 不要 list contains

gc 调整

• https://spark.apache.org/docs/latest/tuning.html#memory-management-overview
• spark.memory.fraction 降低
• --conf "spark.executor.extraJavaOptions==-XX:+UseG1GC -XX:NewRatio=2 -XX:SurvivorRatio=4"

语法

• s"${}" => StringBuilder

Spark Adaptive Execution

• https://blog.youkuaiyun.com/LINBE_blazers/article/details/94956568