flume sink hive的那些坑啊

最新推荐文章于 2024-08-20 14:06:26 发布
原创 最新推荐文章于 2024-08-20 14:06:26 发布 · 4.9k 阅读 · 0 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#flume hive

本文详细记录了在使用Flume整合Hive过程中遇到的各种错误及其解决方案，包括NoClassDefFoundError、NullPointerException、HiveWriter$ConnectException等，并介绍了如何通过调整配置、复制JAR包、创建分桶表及使用ORC存储格式来解决这些问题。

=======flume整合hive=========================
遇到找不到18/03/19 20:08:09 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter
    at org.apache.flume.sink.hive.HiveSink.createSerializer(HiveSink.java:219)

拷贝jar包
cp    /apps/hive-1.2.2/hcatalog/share/hcatalog/hive-hcatalog-streaming-1.2.2.jar /apps/flume-1.8.0/lib

Caused by: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
        at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:251)
        at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:460)
        at org.apache.flume.sink.hive.HiveSink.makeEndPoint(HiveSink.java:379)
        at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:290)
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
        ... 3 more
解决：
ag1.sinks.sink1.useLocalTimeStamp=true


java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/common/HCatUtil
拷贝jar包
cp    /apps/hive-1.2.2/hcatalog/share/hcatalog/*.jar /apps/flume-1.8.0/lib


////
8/12/27 06:01:16 WARN hive.HiveSink: sink1 : Failed connecting to EndPoint {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[18-12-27] }
org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[18-12-27] }
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
        at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
        at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[18-12-27] }
        at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:383)
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:86)
        ... 6 more
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1149)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1193)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1082)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.runDDL(HiveEndPoint.java:404)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.createPartitionIfNotExists(HiveEndPoint.java:369)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:276)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:110)
        at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:379)
        at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:376)
        at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more
18/12/27 06:01:16 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[18-12-27] }
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:267)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[18-12-27] }
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
        at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
        at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
        ... 3 more
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[18-12-27] }
        at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:383)
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:86)
        ... 6 more
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1149)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1193)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1082)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1072)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.runDDL(HiveEndPoint.java:404)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.createPartitionIfNotExists(HiveEndPoint.java:369)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:276)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:110)
        at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:379)
        at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:376)
        at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

//删除partition后
18/12/27 09:56:49 WARN hive.HiveSink: sink1 : Failed connecting to EndPoint {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[] }
org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[] }
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
        at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
        at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hive.hcatalog.streaming.StreamingException: Cannot stream to table that has not been bucketed : {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[] }
        at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.<init>(AbstractRecordWriter.java:71)
        at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:115)
        at org.apache.flume.sink.hive.HiveDelimitedTextSerializer.createRecordWriter(HiveDelimitedTextSerializer.java:66)
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:89)

//分桶之后
create external table ue_through (imsi string,dlwan int,ulwan int) partitioned by (time string) CLUSTERED BY (imsi) INTO 4 BUCKETS row format delimited fields terminated by '\t' location '/omc/pm/ue'

18/12/27 10:12:06 INFO hive.HiveSink: sink1: Creating Writer to Hive end point : {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[] }
18/12/27 10:12:06 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.ClassCastException: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat cannot be cast to org.apache.hadoop.hive.ql.io.AcidOutputFormat
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:267)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat cannot be cast to org.apache.hadoop.hive.ql.io.AcidOutputFormat
        at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.<init>(AbstractRecordWriter.java:75)
        at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:115)
        at org.apache.flume.sink.hive.HiveDelimitedTextSerializer.createRecordWriter(HiveDelimitedTextSerializer.java:66)
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:89)     
        
网上找的--------看了下源码实现AcidOutputFormat的类只有OrcOutputFormat，那么答案显而易见，Hive表需要stored as orc。
create external table ue_through (imsi string,dlwan int,ulwan int) partitioned by (time string) CLUSTERED BY (imsi) INTO 4 BUCKETS  row format delimited fields terminated by '\t' STORED AS ORC location '/omc/pm/ue' ;


//ORC之后
18/12/27 10:32:01 WARN hive.HiveSink: sink1 : Failed connecting to EndPoint {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[] }
org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[] }
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
        at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
        at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.sink.hive.HiveWriter$TxnBatchException: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://hdp01:9083', database='myhive', table='ue_through', partitionVals=[] }
        at org.apache.flume.sink.hive.HiveWriter.nextTxnBatch(HiveWriter.java:400)
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:90)
        ... 6 more
//原来是 这个注解写到原来行的后面导致的
##flume事务控制所需要的缓存容量600条event 
ag1.channels.channel1.transactionCapacity = 600 

olive18
关注
flume收集日志到hive遇到问题总结
suojie的博客
01-18 3433
异常如下： 19/01/18 02:12:51 WARN hive.HiveSink: k1 : Failed connecting to EndPoint {metaStoreUri='thrift://wangfutai:9083', database='hive', table='flume2', partitionVals=[] } org.apache.flume.sink.hive...
flume读取kafka写入hive异常解决-Unable to deliver event. Exception follows.
hello_java_lcl的博客
05-18 2664
用flume访问kafka，写入hive数据库时，启动命令：bin/flume-ng agent -n agent -c conf -f conf/flume.properties -Dflume.root.logger=DEBUG,console 配置文件是安装flume官方文档配置的，很全，地址：http://flume.apache.org/FlumeUserGuide.html#kafka-sink 启动flume之后，报错： ERROR flume.SinkRunner: Unable t
flume简单测试hdfssink && hivesink
帆了个帆的专栏
08-28 3919
博客地址:http://www.fanlegefan.com 文章地址:http://www.fanlegefan.com/archives/flume-hdfssink-hivesinkquich startvi example.conf# example.conf: A single-node Flume configuration# Name the components on this a
flume sink hive 启动命令中常见异常
hello_java_lcl的博客
05-20 1380
常见问题1 Failed to start agent because dependencies were not found in classpath. Error follows. java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter 解决办法一: 将hive中相关的jar复制到$FLUME_HOME/lib下 cp /home/xiaobin/soft/apache-hive-0.14
Flume exec source日志 sink到Hive表中
代码创造世界
10-31 909
日志文件格式： 2018-10-31-15-46-26|PDAT20181031T330494|24496|TDZA20181031D344959|epicc 以“|”分隔分别表示为time,policyno,sumpremium,proposalno,qudao ** FLume配置文件: ** a1.sources=r1 a1.channels=c1 a1.sinks=k1 a1.sourc...
Flume sink hive Flume对接Hive（Sink）遇到的坑，以及最终放弃hive选用hdfs。欢迎讨论指点
MannerXu的博客
03-18 843
项目中打算使用Flume把数据直接传到Hive表而不是HDFS上，使用Hive作为Sink，Flume版本为1.9.0。 前期启动遇到各种报错： NoClassDefFoundError: org/apache/hadoop/hive/ql/session/SessionState NoClassDefFoundError: org/apache/hadoop/hive/cli/CliSessionState NoClassDefFoundError: org/apache/hadoop/hive/
Flume和Hive集成之hive sink
weixin_42455459的博客
12-19 1268
1.Hive表必须设置bucket并且 stored as orc。 对hive表的要求 a：表必须是事物表 b：表必须是分区表，分桶表 c: 表stored as orc clustered分桶、transactional事务、orc存储格式。 [root@master bin]# ps -ef|grep mysql root 23856 19418 0 00:41 pts/2 00:00:00 /bin/sh ./mysqld_safe mysql 239...
flume-hive-sink-1.8.0.jar
07-01
flume1.8.0 采集日志入库到hive，后台日志报错，是心跳bug，此包为修复jar包
Flume - HiveSink 踩坑记录
大数据研究小白
06-21 404
flume数据传输到hive的一些配置（hivesink）
flume sink 简介及官方用例
v15220的博客
05-19 1053
https://flume.apache.org/FlumeUserGuide.html#source-and-sink-batch-sizes-and-channel-transaction-capacities
日常问题——flume连接hive时报错Caused by: java.lang.NoSuchMethodError
栗筝i的博客
06-29 3901
问题描述： 今天新安装的flume，使用flume来做kafka与hive对接时出现了以下两个的错误： Caused by: org.apache.hive.hcatalog.streaming.ConnectionError: HiveEndPoint{metaStoreUri='thrift://localhost:9083', database='db', table='student', partitionVals=} (HiveWriter.java:99) at org.apache.flum
运行flume文件到hdfs出现java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument或拒绝连接
weixin_74865657的博客
04-01 850
2 ] 只需要删除flume的lib下面的guava包，然后拷贝kafka下面的guava包给flume即可。去查看flume配置文件中的端口是否与hadoop配置文件core-site.xml的端口是否保持一致。该命令将返回当前安全模式的状态，可能的状态包括ON和OFF。是由于flume文件与hadoop配置的文件端口不一致导致的。flume的lib下面的为guava-11.0.2.jar。该命令将使集群退出安全模式，允许正常的读写操作。kafka下面的为guava-20.0.jar。
大数据-从flume采集数据通过kafaka下沉到hive表的一个详细案列
jiayunfei521的博客
10-24 1031
HDFS 实施前的准备 安装nginx 安装flume 安装kafka 打jar包 双击package即可打成jar包 在此处可以看到打成jar包的地址 将jar包运行在虚拟机上 我准备了四台虚拟机 我将jar包 ...
flume运行报错：NoSuchMethodError: com.google.common.base.Preconditions.checkArgument
最新发布
m0_64758401的博客
08-20 470
这是由于hadoop和flume中的jar冲突了，flume的lib中 /opt/flume/apache-flume-1.9.0-bin/lib （guava-11.0.2.jar）而 hadoop 的 lib 中 /opt/hadoop/hadoop-3.2.1/share/hadoop/common/lib/ （guava-27.0-jre.jar）。
Flume报错：java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/
WM_55555的博客
09-20 1172
公司项目遇到这个错，找网上资料很多都是说是jar包版本冲突导致，应该也是这部分原因，但最后的解决办法是将flume工程与hive放在同一个服务器下后就ok了。顺便问一下同僚们有没有遇到过下放这个报错，如何解决的？
Flume写入hive错误记录
蜗牛小将
03-24 3756
Flume写入hive错误记录
flume_kafka_hdfs_hive数据的处理
weixin_30323961的博客
03-14 169
使用flume收集数据，将数据传递给kafka和hdfs，kafka上的数据可以使用storm构建实时计算，而hdfs上的数据，经过MR处理之后可以导入hive中进行处理。 环境：hadoop1.2.1，hive 0.13.1，maven 3.2.5，flume 1.4，kafka 0.7.2，eclipse luna，jdk 1.7_75；mysql-connector-j...
使用flume时出现NoSuchMethodError: com.google.common.base.Preconditions.checkArgument
qq_35472225的博客
06-08 1324
使用flume时出现NoSuchMethodError: com.google.common.base.Preconditions.checkArgument flume是一个十分好用的数据采集框架，但对于新手来说，在使用过程中，很有可能会出现NoSuchMethodError: com.google.common.base.Preconditions.checkArgument这一错误，梳理了一下，大概有三种解决办法 一、环境变量未配置 在安装了flume的所有机器上，首先都需要配置hadoop环境变量
评论 7
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
表情包 代码片
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值