spark开启history-server 提交任务报错InvalidAuxServiceException The auxService:spark_shuffle does not exist

在开启history-server 提交spark任务

问题描述 提交任务后在页面现实一直运行无法结束，查询异常报错

24/11/29 16:37:22 ERROR YarnAllocator: Failed to launch executor 46 on container container_e04_1732847420607_0022_01_000052
org.apache.spark.SparkException: Exception while starting container container_e04_1732847420607_0022_01_000052 on host hadoop78
	at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:127)
	at org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:65)
	at org.apache.spark.deploy.yarn.YarnAllocator.$anonfun$runAllocatedContainers$6(YarnAllocator.scala:785)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:spark_shuffle does not exist
	at jdk.internal.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateExceptionImpl(SerializedExceptionPBImpl.java:171)
	at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:182)
	at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
	at org.apache.hadoop.yarn.client.api.impl.NMClientImpl.startContainer(NMClientImpl.java:217)
	at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:123)111

在history-server 中的conf配置是

然后再对应的yarn nodemanager 节点的yarn-site配置参数

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle,spark_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
  <property>
    <name>spark.shuffle.service.port</name>
    <value>7337</value>
  </property>
  <property>
    <name>spark.authenticate</name>
    <value>false</value>
  </property>

然后将spark目录下的找到目录yarn 将jar包移动到Hadoop目录的hadoop-3.4.0/share/hadoop/yarn/lib 目录下即可

注意问题

有部分文章说移动

spark-network-shuffle_2.12-3.4.3.jar
spark-network-common_2.12-3.4.3.jar

这时候报错 java.lang.NoSuchMethodError: ‘org.apache.spark.network.shuffledb.DB org.apache.spark.network.util.DBProvider.initDB(org.apache.spark.network.shuffledb.DBBackend, java.io.File, org.apache.spark.network.shuffledb.StoreVersion, com.fasterxml.jackson.databind.ObjectMapper)’

org.apache.spark.network.yarn.YarnShuffleService: Use LEVELDB as the implementation of spark.shuffle.service.db.backend
2024-12-02 13:22:18,155 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
java.lang.NoSuchMethodError: 'org.apache.spark.network.shuffledb.DB org.apache.spark.network.util.DBProvider.initDB(org.apache.spark.network.shuffledb.DBBackend, java.io.File, org.apache.spark.network.shuffledb.StoreVersion, com.fasterxml.jackson.databind.ObjectMapper)'
       at org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<init>(ExternalShuffleBlockResolver.java:131)
       at org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.<init>(ExternalShuffleBlockResolver.java:100)
       at org.apache.spark.network.shuffle.ExternalBlockHandler.<init>(ExternalBlockHandler.java:90)
       at org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:278)
       at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
       at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.initAuxService(AuxServices.java:475)
       at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:758)
       at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
       at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
       at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:336)
       at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
       at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:110)
       at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:501)
       at org.apache.hadoop.service.AbstractService.init(AbstractService.java:165)
       at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:969)
       at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1058)
2024-12-02 13:22:18,158 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system...
2024-12-02 13:22:18,159 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped.
2024-12-02 13:22:18,159 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.
2024-12-02 13:22:18,160 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG: