Logstash-Output-Webhdfs

最新推荐文章于 2024-01-11 21:16:33 发布

蓝心语

最新推荐文章于 2024-01-11 21:16:33 发布

阅读量4.4k

点赞数 1

CC 4.0 BY-SA版权

分类专栏： elk 文章标签： ELK webhdfs logstash

本文链接：https://blog.youkuaiyun.com/FengYe_YuLu/article/details/75308148

elk 专栏收录该内容

1 篇文章

订阅专栏

本文记录了一次使用Logstash结合WebHDFS输出插件遇到的问题及解决过程。问题表现为无法正常写入HDFS，经过排查发现是由于未正确配置用户及hosts文件导致。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

无奈,有时候为了解决问题而找答案,解决后却忘了作笔记..这次狠狠给自己个教训..
[WARN ][logstash.outputs.webhdfs ] Failed to flush outgoing items {:outgoing_count=>500, :exception=>”WebHDFS::ServerError”, :backtrace=>[“/opt/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:351:in request'", "/opt/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:270:inoperate_requests’”, “/opt/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:73:in create'", "/opt/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.2/lib/logstash/outputs/webhdfs.rb:211:inwrite_data’”, “/opt/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.2/lib/logstash/outputs/webhdfs.rb:195:in flush'", "org/jruby/RubyHash.java:1342:ineach’”, “/opt/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.2/lib/logstash/outputs/webhdfs.rb:183:in flush'", "/opt/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:inbuffer_flush’”, “org/jruby/RubyHash.java:1342:in each'", "/opt/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:inbuffer_flush’”, “/opt/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:193:in buffer_flush'", "/opt/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:159:inbuffer_receive’”, “/opt/logstash-5.4.1/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.2/lib/logstash/outputs/webhdfs.rb:166:in receive'", "/opt/logstash-5.4.1/logstash-core/lib/logstash/outputs/base.rb:92:inmulti_receive’”, “org/jruby/RubyArray.java:1613:in each'", "/opt/logstash-5.4.1/logstash-core/lib/logstash/outputs/base.rb:92:inmulti_receive’”, “/opt/logstash-5.4.1/logstash-core/lib/logstash/output_delegator_strategies/legacy.rb:22:in multi_receive'", "/opt/logstash-5.4.1/logstash-core/lib/logstash/output_delegator.rb:47:inmulti_receive’”, “/opt/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:407:in output_batch'", "org/jruby/RubyHash.java:1342:ineach’”, “/opt/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:406:in output_batch'", "/opt/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:352:inworker_loop’”, “/opt/logstash-5.4.1/logstash-core/lib/logstash/pipeline.rb:317:in `start_workers’”]}

想半天,hdfs不可能有问题的,线上一直跑着呢,原来webhdfs的配置中user是不必须填的,倒腾半天,没结果查下网上,终于找到了答案,解决了,随便自己也作下笔记..

webhdfs
user:必须填写启动hdfs的用户名
必须在logstash所在的服务器上的/etc/hosts配置hdfs集群节点路由列表
分享借鉴地址:http://stackmirror.cn/page/s1b6a4mbbvas