hello_java_lcl的博客

Name Default Description channel – type – The component type name, needs to behive hive.metastore – Hive metastore URI (eg thrift://a.b.com:9083 ) hive.database – Hive database name hive.table – Hive table n..

flume读取log日志文件，写hive表注意：hive创建桶表；flume读取文件，按行读取，默认行最大值为2048个长度，过长会截取。如果不是桶表：异常报错如下Unexpected end-of-input in VALUE_STRING由于log日志每条记录是以json形式存储，然后Agent.sink.hive.serializer = JSON解析json，写入hive表。最开始找到的报错信息是json解析异常：[DEBUG - org....

#!/bin/bash#参数：start|stop|restart#功能：flume 启动停止重启#使用方法:#./execflume.sh start flume_cmbc.conf(配置文件，自己修改) Cobub（代理名称，自己修改）#./execflume.sh stop#./execflume.sh restart flume_cmbc.conf(配置文件，自己修改) Cobub（代理名称，自己修改）path=$(cd `dirname $0`; pwd)echo $p

配置文件 localAgent.sources = skafka localAgent.sinks = shive localAgent.channels = k2h #k2h shive localAgent.sinks.shive.channel = k2h #skafka k2h localAgent.sources.skafka.channels = k2h #describe the source localAg...

flume配置文件 #source: sdir sink: skafka channel: d2k agent1.sources = sdir agent1.sinks = skafka agent1.channels = d2k #d2k skafka agent1.sinks.skafka.channel = d2k #sdir d2k agent1.sources.sdir.channels = d2k ...

用flume访问kafka，写入hive数据库时，启动命令：bin/flume-ng agent -n agent -c conf -f conf/flume.properties -Dflume.root.logger=DEBUG,console配置文件是安装flume官方文档配置的，很全，地址：http://flume.apache.org/FlumeUserGuide.html#kafka-sink启动flume之后，报错：ERROR flume.SinkRunner: Unable t

常见问题1Failed to start agent because dependencies were not found in classpath. Error follows.java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/streaming/RecordWriter解决办法一:将hive中相关的jar复制到$FLUME_HOME/lib下cp /home/xiaobin/soft/apache-hive-0.14

flume内存溢出，在conf文件夹中，有flume-env.sh文件，里面有# export JAVA_OPTS="$JAVA_OPTS -Dorg.apache.flume.log.rawdata=true -Dorg.apache.flume.log.printconfig=true "这个配置项，但是好像不起作用，需要到bin目录下，修改flume-ng启动命令：vim打开这个文件，执行搜索命令 :/JAVA_OPTS找到这里，进行修改;内存溢出解决。...

flume一直是用bin/flume-ng agent -n agent -c conf -f conf/flume.properties -Dflume.root.logger=DEBUG,console 这种方式启动的，但是总是报错，而且日志打印在控制台，还没怎么着呢就刷过去了，实在不方便了，查到一种后台启动flume的方法，并且日志也能在指定目录输出，很不错首先修改conf/log4j.properties配置文件，log的目录然后启动命令改为：bin/flume-ng a