spark连接es的问题java.lang.ClassNotFoundException:org.elasticsearch.client.transport.NoNodeAvailableExcep

最新推荐文章于 2025-06-11 14:15:59 发布

原创最新推荐文章于 2025-06-11 14:15:59 发布 · 8.2k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #elasticsearch

spark 专栏收录该内容

11 篇文章

订阅专栏

跑spark任务：

daily.out.20171012的日志中有如下错误：

17/10/17 15:51:06 WARNspark.ThrowableSerializationWrapper: Task exception could not be deserialized

java.lang.ClassNotFoundException:org.elasticsearch.client.transport.NoNodeAvailableException

atjava.net.URLClassLoader$1.run(URLClassLoader.java:366)

atjava.net.URLClassLoader$1.run(URLClassLoader.java:355)

atjava.security.AccessController.doPrivileged(Native Method)

atjava.net.URLClassLoader.findClass(URLClassLoader.java:354)

atjava.lang.ClassLoader.loadClass(ClassLoader.java:425)

atsun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

atjava.lang.ClassLoader.loadClass(ClassLoader.java:358)

atjava.lang.Class.forName0(Native Method)

atjava.lang.Class.forName(Class.java:270)

atorg.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:67)

atjava.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)

atjava.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)

atjava.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)

atjava.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)

atjava.io.ObjectInputStream.readObject(ObjectInputStream.java:370)

atorg.apache.spark.ThrowableSerializationWrapper.readObject(TaskEndReason.scala:167)

atsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

atjava.lang.reflect.Method.invoke(Method.java:606)

atjava.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)

atjava.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)

atjava.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)

atjava.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)

atjava.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)

atjava.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)

atjava.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)

atjava.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)

atjava.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)

atjava.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)

atjava.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)

atjava.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)

atjava.io.ObjectInputStream.readObject(ObjectInputStream.java:370)

atorg.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:72)

atorg.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:98)

atorg.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$2.apply$mcV$sp(TaskResultGetter.scala:108)

atorg.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$2.apply(TaskResultGetter.scala:105)

atorg.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699)

atorg.apache.spark.scheduler.TaskResultGetter$$anon$3.run(TaskResultGetter.scala:105)

atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

atjava.lang.Thread.run(Thread.java:745)

17/10/17 15:51:07 INFOstorage.BlockManager: BlockManager stopped

17/10/17 15:51:07 INFOstorage.BlockManagerMaster: BlockManagerMaster stopped

17/10/17 15:51:07 INFOscheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:OutputCommitCoordinator stopped!

17/10/17 15:51:07 INFOremote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.

17/10/17 15:51:07 INFO remote.RemoteActorRefProvider$RemotingTerminator:Remote daemon shut down; proceeding with flushing remote transports.

17/10/17 15:51:07 INFO spark.SparkContext:Successfully stopped SparkContext

Exception in thread "main"org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 instage 20.0 failed 4 times, most recent failure: Lost task 8.3 in stage 20.0(TID 65, zjhydsz11.shbank.com): UnknownReason

Driver stacktrace:

atorg.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1294)

atorg.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1282)

atorg.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1281)

atscala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

atscala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

atorg.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1281)

atorg.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)

atscala.Option.foreach(Option.scala:236)

进spark界面进一步查看stderr中，可以进一步发现如下问题：

17/10/17 16:40:05 INFOelasticsearch.plugins: [Ritchie Gilmore] loaded [], sites []

17/10/17 16:40:05 INFOelasticsearch.plugins: [Jarella] loaded [], sites []

17/10/17 16:40:06 ERROR client.transport:[Jarella] failed to get node info for{#transport#-1}{10.240.44.40}{10.240.44.40:9300}, disconnecting...

RemoteTransportException[[Failed todeserialize response of type[org.elasticsearch.action.admin.cluster.node.liveness.LivenessResponse]]];nested: TransportSerializationException[Failed to deserialize response of type[org.elasticsearch.action.admin.cluster.node.liveness.LivenessResponse]];nested: ExceptionInInitializerError; nested: IllegalArgumentException[An SPIclass of type org.apache.lucene.codecs.PostingsFormat with name 'Lucene50' doesnot exist. You need to add thecorresponding JAR file supporting this SPI to your classpath. The current classpath supports the followingnames: [es090, completion090, XBloomFilter]];

Caused by:TransportSerializationException[Failed to deserialize response of type[org.elasticsearch.action.admin.cluster.node.liveness.LivenessResponse]];nested: ExceptionInInitializerError; nested: IllegalArgumentException[An SPIclass of type org.apache.lucene.codecs.PostingsFormat with name 'Lucene50' doesnot exist. You need to add thecorresponding JAR file supporting this SPI to your classpath. The current classpath supports the followingnames: [es090, completion090, XBloomFilter]];

atorg.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:179)

atorg.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:138)

attranswarp.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

attranswarp.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

attranswarp.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)

attranswarp.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)

attranswarp.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)

attranswarp.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)

attranswarp.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)

attranswarp.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)

attranswarp.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

attranswarp.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)

attranswarp.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)

attranswarp.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)

attranswarp.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)

attranswarp.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)

attranswarp.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)

attranswarp.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)

attranswarp.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)

attranswarp.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

attranswarp.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)

atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

atjava.lang.Thread.run(Thread.java:745)

Caused by:java.lang.ExceptionInInitializerError

atorg.elasticsearch.Version.fromId(Version.java:500)

atorg.elasticsearch.Version.readVersion(Version.java:276)

atorg.elasticsearch.cluster.node.DiscoveryNode.readFrom(DiscoveryNode.java:326)

atorg.elasticsearch.cluster.node.DiscoveryNode.readNode(DiscoveryNode.java:309)

atorg.elasticsearch.action.admin.cluster.node.liveness.LivenessResponse.readFrom(LivenessResponse.java:52)

atorg.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:177)

...23 more

Caused by:java.lang.IllegalArgumentException: An SPI class of typeorg.apache.lucene.codecs.PostingsFormat with name 'Lucene50' does notexist. You need to add the correspondingJAR file supporting this SPI to your classpath. The current classpath supports the following names: [es090,completion090, XBloomFilter]

atorg.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:109)

atorg.apache.lucene.codecs.PostingsFormat.forName(PostingsFormat.java:112)

atorg.elasticsearch.common.lucene.Lucene.<clinit>(Lucene.java:103)

...29 more

17/10/17 16:40:06 INFO executor.Executor:Finished task 4.0 in stage 20.0 (TID 46). 1560 bytes result sent to driver

17/10/17 16:40:06 INFO executor.Executor:Finished task 1.0 in stage 20.0 (TID 43). 1560 bytes result sent to driver

17/10/17 16:40:06 INFO executor.CoarseGrainedExecutorBackend:Got assigned task 50

17/10/17 16:40:06 INFO executor.Executor:Running task 3.1 in stage 20.0 (TID 50)

17/10/17 16:40:06 INFOexecutor.CoarseGrainedExecutorBackend: Got assigned task 51

17/10/17 16:40:06 INFO executor.Executor:Running task 8.0 in stage 20.0 (TID 51)

17/10/17 16:40:06 INFOstorage.ShuffleBlockFetcherIterator: Getting 7 non-empty blocks out of 16blocks

17/10/17 16:40:06 INFOstorage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 16blocks

17/10/17 16:40:06 INFOstorage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms

17/10/17 16:40:06 INFOstorage.ShuffleBlockFetcherIterator: Started 1 remote fetches in 1 ms

17/10/17 16:40:06 INFOelasticsearch.plugins: [Ultragirl] loaded [], sites []

17/10/17 16:40:06 INFOelasticsearch.plugins: [Karma] loaded [], sites []

17/10/17 16:40:06 ERROR client.transport:[Karma] failed to get node info for{#transport#-1}{10.240.44.40}{10.240.44.40:9300}, disconnecting...