【hadoop】关于ERROR security.UserGroupInformation

最新推荐文章于 2019-11-14 11:11:47 发布

原创最新推荐文章于 2019-11-14 11:11:47 发布 · 1.2w 阅读

0 ·

CC 4.0 BY-SA版权

Hadoop 专栏收录该内容

20 篇文章

订阅专栏

本文分析了在使用CDH版本Hadoop时遇到的一个错误，该错误与用户权限及Protobuf解析有关。为了成功运行MapReduce程序，文章建议将用户从hadoop切换为hdfs。

14/01/23 20:49:47 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop1/192.168.1.148"; destination host is: "hadoop1":9000; 
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "hadoop1/192.168.1.148"; destination host is: "hadoop1":9000; 
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
	at org.apache.hadoop.ipc.Client.call(Client.java:1242)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
	at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
	at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:629)
	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1545)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:820)
	at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1681)
	at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1627)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:211)
	at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248)
	at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1079)
	at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1096)
	at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:177)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:995)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:948)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:948)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
	at com.zdworks.bigdata.backup.ParseLog_Redis.runstat(ParseLog_Redis.java:383)
	at com.zdworks.bigdata.backup.ParseLog_Redis.main(ParseLog_Redis.java:394)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
	at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:73)
	at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
	at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:213)
	at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746)
	at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
	at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282)
	at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
	at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
	at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752)
	at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:985)
	at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:949)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:847)

从apache hadoop换用CDH之后，权限用户发生了变化(hadoop转为hdfs)。所以要运行MR程序时，要切换到hdfs用户才行。