spark on yarn报错ERROR GPLNativeCodeLoader: Could not load native gpl library

最新推荐文章于 2024-09-10 21:22:07 发布

原创最新推荐文章于 2024-09-10 21:22:07 发布 · 4.2k 阅读

2 ·

CC 4.0 BY-SA版权

Spark 专栏收录该内容

23 篇文章

订阅专栏

本文记录了一个Spark SQL启动过程中遇到的警告及错误，并给出了详细的解决步骤。主要问题是由于缺少必要的Hadoop本地库支持导致的错误，通过修改spark-env.sh文件中的环境变量解决了该问题。

启动spark-sql
spark-sql --master yarn-client --executor-memory 1G

报错：
16/03/30 14:45:32 WARN BlockReaderLocal: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.

使用命令查看databases
spark-sql> show databases;

16/03/30 14:46:01 ERROR GPLNativeCodeLoader: Could not load native gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
        at java.lang.Runtime.loadLibrary0(Runtime.java:849)
        at java.lang.System.loadLibrary(System.java:1088)
        at com.hadoop.compression.lzo.GPLNativeCodeLoader.(GPLNativeCodeLoader.java:32)
        at com.hadoop.compression.lzo.LzoCodec.(LzoCodec.java:71)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:270)
        at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1834)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1799)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.(CompressionCodecFactory.java:175)
        at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

16/03/30 14:46:01 ERROR LzoCodec: Cannot load native-lzo without native-hadoop

解决方法：
spark-env.sh中增加
export SPARK_SUBMIT_LIBRARY_PATH=/usr/lib/hadoop/lib/native/:$SPARK_SUBMIT_LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/hadoop/lib/native/
export SPARK_LIBRARY_PATH=/usr/lib/hadoop/lib/native/:$SPARK_LIBRARY_PATH
export SPARK_CLASSPATH=/usr/local/spark/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar:$CLASSPATH

后来尝试了下，保留以下两个即可：
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/hadoop/lib/native/
export SPARK_CLASSPATH=/usr/local/spark/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar:$CLASSPATH

参考：
http://find.searchhub.org/document/a128707a98fe4ec6
http://lotso.blog.51cto.com/3681673/1441737
http://guoyunsky.iteye.com/blog/1237327