flume启动报错

最新推荐文章于 2024-10-09 07:00:00 发布

lkinga

最新推荐文章于 2024-10-09 07:00:00 发布

阅读量487

点赞数

CC 4.0 BY-SA版权

文章标签： flume

本文链接：https://blog.youkuaiyun.com/qq_41837900/article/details/98074986

本文详细解析了Apache Flume中Sink Runner在处理HDFS事件时遇到的HDFS IO错误，具体为InvalidProtocolBufferException异常，揭示了问题根源在于协议消息结束组标签与预期标签不匹配，并提供了检查CM节点管理界面端口号的解决方案。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

01 八月 2019 16:05:28,201 WARN  [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:443)  - HDFS IO error
java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: "cm1/222.22.91.81"; destination host is: "cm1":9000;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)

        at org.apache.hadoop.ipc.Client.call(Client.java:1472)
        at org.apache.hadoop.ipc.Client.call(Client.java:1399)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
        at com.sun.proxy.$Proxy12.create(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNameno
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.ja
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1593)
        at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:397)
        at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:393)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:393)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:337)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
        at org.apache.flume.sink.hdfs.HDFSDataStream.doOpen(HDFSDataStream.java:81)
        at org.apache.flume.sink.hdfs.HDFSDataStream.open(HDFSDataStream.java:108)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:262)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:252)
        at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:701)
        at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
        at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:698)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
        at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94)
        at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124)
        at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202)
        at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
        at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
        at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
        at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
        at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167)
        at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072)
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)

解决方式，查看CHD管理界面发现端口不是9000 是8022
在这里插入图片描述