Unable to load native-hadoop library for your platform... using builtin-java cl

在配置Hadoop时遇到'Unable to load native-hadoop library for your platform...'的错误,通过开启调试日志和检查发现java.library.path环境变量未正确设置。解决方案是将JAVA_LIBRARY_PATH添加到指定路径,从而成功加载native库。

Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

在配置的时候,频繁出现这个问题,在网上找了相关资料,都是说libray包没找到。

检查所有配置,一切都正常。都配置正常。

修改过:etc/profile

问题还是没解决。

 

之后尝试如下步骤:

1, 开启调试日志 export HADOOP_ROOT_LOGGER=DEBUG,console

2,/data/server/hadoop-2.6.0/bin 

   ./hadoop checknative -a

查看日志,报如下错误:

[hadoop@DEV-NEW-HADOOP-01 bin]$ ./hadoop checknative

18/09/28 09:55:34 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...

18/09/28 09:55:34 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path

18/09/28 09:55:34 DEBUG util.NativeCodeLoader: java.library.path=:/data/server/hadoop-2.6.0/lib/native/Linux-amd64-64/

18/09/28 09:55:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

18/09/28 09:55:35 DEBUG util.Shell: setsid exited with exit code 0

Native library checking:

hadoop:  false 

zlib:    false 

snappy:  false 

lz4:     false 

bzip2:   false 

openssl: false 

18/09/28 09:55:35 INFO util.ExitUtil: Exiting with status 1

 

查看环境变量,并没有配置 java.library.path=:/data/server/hadoop-2.6.0/lib/native/Linux-amd64-64/。

之后,尝试使用:export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/data/server/hadoop-2.6.0/lib/native/

然后在检查:./hadoop checknative

恢复正常,问题解决

[hadoop@DEV-NEW-HADOOP-02 bin]$ ./hadoop checknative

18/09/28 09:57:14 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version

18/09/28 09:57:14 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library

Native library checking:

hadoop:  true /data/server/hadoop-2.6.0/lib/native/libhadoop.so

zlib:    true /lib64/libz.so.1

snappy:  true /data/server/hadoop-2.6.0/lib/native/Linux-amd64-64/libsnappy.so.1

lz4:     true revision:99

bzip2:   false 

openssl: true /usr/lib64/libcrypto.so

 

WARNING: Using incubator modules: jdk.incubator.vector Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 25/08/07 11:16:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ❌ 驱动加载失败: An error occurred while calling z:java.lang.Class.forName. : java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:184) at py4j.ClientServerConnection.run(ClientServerConnection.java:108) at java.base/java.lang.Thread.run(Thread.java:842) ⚠️ 请检查: 1. jar包位置 2. spark-defaults.conf配置 3. 文件权限 进程已结束,退出代码为 0 指定jar包位置后 config("spark.jars", "/export/server/spark/jars/mysql-connector-java-8.0.28.jar"). 25/08/07 11:20:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). ✅ MySQL驱动已加载: com.mysql.cj.jdbc.Driver Traceback (most recent call last): File "/tmp/pycharm_project_49/00_example/test.py", line 26, in <module> .load() ^^^^^^ File "/root/.virtualenvs/pySpark/lib/python3.12/site-packages/pyspark/sql/readwriter.py", line 318, in load return self._df(self._jreader.load()) ^^^^^^^^^^^^^^^^^^^^ File "/root/.virtualenvs/pySpark/lib/python3.12/site-packages/py4j/java_gateway.py", line 1362, in __call__ return_value = get_return_value( ^^^^^^^^^^^^^^^^^ File "/root/.virtualenvs/pySpark/lib/python3.12/site-packages/pyspark/errors/exceptions/captured.py", line 282, in deco return f(*a, **kw) ^^^^^^^^^^^ File "/root/.virtualenvs/pySpark/lib/python3.12/site-packages/py4j/protocol.py", line 327, in get_return_value raise Py4JJavaError( py4j.protocol.Py4JJavaError: An error occurred while calling o39.load. : java.sql.SQLException: No suitable driver at java.sql/java.sql.DriverManager.getDriver(DriverManager.java:299) at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$2(JDBCOptions.scala:118) at scala.Option.getOrElse(Option.scala:201)
最新发布
08-08
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值