实时程序操作Hadoop HDFS一段时间后自动退出_not attempting to re-login since the last re-login-优快云博客

本文链接：https://blog.youkuaiyun.com/joeyhacker/article/details/143564675

长时间驻留程序操作带Kerberos认证的Hadoop集群组件时，出现下面异常

24/11/06 10:05:34 ERROR security.UserGroupInformation: TGT is expired. Aborting renew thread for oracle@XXXX.COM.

24/11/06 10:05:55 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: org.apache.hadoop.security.KerberosAuthException: Login failure for user: oracle@XXXX.COM javax.security.auth.login.LoginException: Unable to obtain password from user

; Host Details : local host is: "BigData-Test-host1/192.168.0.101"; destination host is: "BigData-Test-host1":8032; , while invoking ApplicationClientProtocolPBClientImpl.getApplicationReport over null. Trying to failover immediately.

24/11/06 10:05:58 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1730858755350

24/11/06 10:06:08 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1730858755350

24/11/06 10:06:10 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1730858755350

24/11/06 10:06:14 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1730858755350

24/11/06 10:06:18 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1730858755350

24/11/06 10:06:20 WARN ipc.Client: Couldn't setup connection for oracle@XXXX.COM to BigData-Test-host1/192.168.0.101:8032

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)

at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:408)

at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:614)

at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:410)

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799)

at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)

at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)

at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)

at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)

at org.apache.hadoop.ipc.Client.call(Client.java:1391)

at org.apache.hadoop.ipc.Client.call(Client.java:1355)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)

at com.sun.proxy.$Proxy14.getApplicationReport(Unknown Source)

at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:236)

at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)

at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)

at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)

at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)

at com.sun.proxy.$Proxy15.getApplicationReport(Unknown Source)

at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:516)

at org.apache.spark.deploy.yarn.Client.getApplicationReport(Client.scala:328)

at org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:1048)

at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:115)

Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)

at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)

at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)

at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)

at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)

at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)

... 30 more

24/11/06 10:06:20 ERROR yarn.Client: Failed to contact YARN for application application_1730479537272_0028.

java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for oracle@XXXX.COM to BigData-Test-host1/192.168.0.101:8032; Host Details : local host is: "BigData-Test-host1/192.168.0.101"; destination host is: "BigData-Test-host1":8032;

at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808)

at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1503)

at org.apache.hadoop.ipc.Client.call(Client.java:1445)