flume实时抽取mysql到hdfs

最新推荐文章于 2023-03-30 09:42:13 发布
原创 最新推荐文章于 2023-03-30 09:42:13 发布 · 1.6k 阅读 · 5 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#大数据 #hadoop #flume #数据库

本文分享了使用Flume进行异库同步的经验，通过配置Flume agent实现从MySQL到HDFS的准实时数据同步，包括详细的配置参数说明及常见问题解决。

最近有个项目可能需要异库同步，于是就上网搜了下方法，发现通过flume到hdfs可以做到准实时，照网上的方法试了发现不行，自己又去github看了下源码，这才行
附上源码地址
我用的是Flume 1.6.0-cdh5.16.1
以下是我的flume agent配置，配置信息以上链接中也有详细说明

# The configuration file needs to define the sources, 
# the channels and the sinks.
# Sources, channels and sinks are defined per agent, 
# in this case called 'agent'

test.sources = sqls
test.channels = c1
test.sinks = hdfssink

# For each one of the sources, the type is defined
test.sources.sqls.type =  org.keedio.flume.source.SQLSource

test.sources.sqls.hibernate.connection.url = jdbc:mysql://localhost:3306/testflume
test.sources.sqls.hibernate.connection.user = root
test.sources.sqls.hibernate.connection.password = ~!@Mysql@2018
test.sources.sqls.hibernate.connection.autocommit = true
test.sources.sqls.hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvider
test.sources.sqls.hibernate.connection.driver_class = com.mysql.jdbc.Driver
test.sources.sqls.hibernate.c3p0.min_size=1
test.sources.sqls.hibernate.c3p0.max_size=10
test.sources.sqls.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect

test.sources.sqls.max.rows = 1000000
test.sources.sqls.table = test
test.sources.sqls.columns.to.select = *
test.sources.sqls.column.name = id	
test.sources.sqls.incremental.value = 0
test.sources.sqls.delimiter.entry = ,
test.sources.sqls.enclose.by.quotes = false
test.sources.sqls.run.query.delay=5000


test.sources.sqls.status.file.path = /data/flume
test.sources.sqls.status.file.name = sqls.status


# The channel can be defined as follows.
test.sources.sqls.channels = c1

# Each sink's type must be defined
test.sinks.hdfssink.channel = c1
test.sinks.hdfssink.type = hdfs
test.sinks.hdfssink.hdfs.path = hdfs://nameservice1:8020/flume/mysqls
test.sinks.hdfssink.hdfs.fileType = DataStream
test.sinks.hdfssink.hdfs.writeFormat = Text
test.sinks.hdfssink.hdfs.rollSize = 268435456
test.sinks.hdfssink.hdfs.rollInterval = 7200
test.sinks.hdfssink.hdfs.rollCount = 0
# Each channel's type is defined.
test.channels.c1.type = memory

以下是对一些属性的说明

属性说明
.hibernate.connection.url数据库连接地址
hibernate.connection.user用户名
hibernate.connection.password密码
hibernate.connection.autocommit是否自动提交
table想要获取的表名
columns.to.select想要获取的列，默认是*
column.name索引字段
delimiter.entry分隔符
query.delay查询间隔
enclose.by.quotes是否加引号
status.file.path状态文件路径
status.file.name状态文件名称
custom.query自定义查询语句，使用需谨慎，详情请查看源码地址

先去上面给的github地址中下载源码，使用IDEA打开
在这里插入图片描述

然后将项目打包，将所有用到的jar包放到flume的lib目录下面，我的目录是/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/lib/，切记要所有jar包，不然会有各种错误！
在这里插入图片描述
对了有个问题要说明下，cdh安装的flume只要开启了就是默认agent一直开启的，所有我们把它关掉在这里插入图片描述
不使用配置中的properties，我们自己创一个，我在/data下面创了个agent的配置信息文件在这里插入图片描述
除此之外我们还需要创建个状态文件目录，存放标识上次抽取数据到什么位置了的状态文件，
我的是在/data/flume/下面，然后chmod 777 /data/flume
sink我用的是hdfs sink，所以在hdfs上建一个目录存放接收的文件，我建的是/flume/mysqls，一样hdfs dfs -chmod 777 /flume/mysqls
然后看看source端，在testflume库下面有张test表
在这里插入图片描述
我们开启flume agent

[root@mf04 ~]# flume-ng agent -c /opt/cloudera/parcels/CDH/lib/flume-ng/lib/ -f /data/test.properties -n test
Info: Including Hadoop libraries found via (/bin/hadoop) for HDFS access
Info: Including HBASE libraries found via (/bin/hbase) for HBASE access
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
Info: Including Hive libraries found via () for Hive access
+ exec /usr/local/java/jdk1.8.0_191/bin/java -Xmx20m -cp '/opt/cloudera/parcels/CDH/lib/flume-ng/lib:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/lib/*:/etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../conf:/usr/local/java/jdk1.8.0_191/lib/tools.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/..:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/activation-1.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/api-util-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/asm-3.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/avro.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/aws-java-sdk-bundle-1.11.134.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-beanutils-1.9.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-codec-1.9.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-collections-3.2.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-configuration-1.6.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-daemon-1.0.13.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-digester-1.8.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-el-1.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-httpclient-3.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-io-2.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-lang-2.6.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-logging-1.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-math-2.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-math3-3.1.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/commons-net-3.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/core-3.1.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/curator-client-2.7.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/curator-framework-2.7.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/curator-recipes-2.7.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/disruptor-3.3.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/gson-2.2.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/guava-12.0.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hamcrest-core-1.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-annotations-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-annotations-1.2.0-cdh5.16.1-tests.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-client-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-common-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-common-1.2.0-cdh5.16.1-tests.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-examples-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-external-blockcache-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-hadoop2-compat-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-hadoop2-compat-1.2.0-cdh5.16.1-tests.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-hadoop-compat-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-hadoop-compat-1.2.0-cdh5.16.1-tests.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-it-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-it-1.2.0-cdh5.16.1-tests.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-prefix-tree-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-procedure-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-protocol-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-resource-bundle-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-rest-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-rsgroup-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-rsgroup-1.2.0-cdh5.16.1-tests.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-server-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-server-1.2.0-cdh5.16.1-tests.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-shell-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-spark-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hbase-thrift-1.2.0-cdh5.16.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/hsqldb-1.8.0.10.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/htrace-core-3.2.0-incubating.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/htrace-core4-4.0.1-incubating.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/htrace-core.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jackson-annotations-2.2.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jackson-core-2.2.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jackson-core-asl-1.8.10.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jackson-databind-2.2.3-cloudera.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jackson-jaxrs-1.8.10.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jackson-mapper-asl-1.8.10-cloudera.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jackson-xc-1.8.10.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jamon-runtime-2.4.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jaxb-api-2.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jcodings-1.0.8.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jersey-client-1.9.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jersey-core-1.9.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jersey-json-1.9.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jersey-server-1.9.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jets3t-0.9.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jettison-1.3.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jetty-6.1.26.cloudera.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jetty-sslengine-6.1.26.cloudera.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jetty-util-6.1.26.cloudera.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/joni-2.1.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jruby-cloudera-1.0.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jsch-0.1.42.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/jsp-api-2.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/junit-4.12.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/leveldbjni-all-1.8.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/libthrift-0.9.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/log4j-1.2.17.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/metrics-core-2.2.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/netty-all-4.0.50.Final.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/paranamer-2.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/protobuf-java-2.5.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/servlet-api-2.5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/slf4j-log4j12.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/spymemcached-2.11.6.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/xmlenc-0.52.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/xz-1.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/zookeeper.jar:/etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/etc/hadoop/conf/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/bin/../lib/hadoop/*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/bin/../lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/bin/../lib/zookeeper/*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/bin/../lib/zookeeper/lib/*::/conf:/lib/*:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/antlr-2.7.7.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/antlr-runtime-3.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/apache-mime4j-core-0.7.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/apache-mime4j-dom-0.7.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/api-asn1-api-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/api-util-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/argparse4j-0.4.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/asm-3.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/asm-4.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/asm-commons-4.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/asm-debug-all-4.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/aspectjrt-1.6.5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/avro.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/bcmail-jdk15-1.45.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/bcprov-jdk15-1.45.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/boilerpipe-1.1.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-codec-1.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-collections-3.2.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-configuration-1.6.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-el-1.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-fileupload-1.3.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-io-2.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-lang-2.6.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-logging-1.1.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-math3-3.1.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/commons-net-3.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/config-1.0.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/curator-client-2.7.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/curator-framework-2.7.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/curator-recipes-2.7.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/dom4j-1.6.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/fastutil-6.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/fontbox-1.8.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/gson-2.2.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/guava-11.0.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/hadoop-auth.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/hadoop-common.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/hadoop-hdfs.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/hppc-0.5.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/htrace-core-3.2.0-incubating.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/htrace-core4-4.0.1-incubating.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/httpclient-4.2.5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/httpcore-4.2.5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/httpmime-4.2.5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/isoparser-1.0-RC-1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jackson-annotations-2.3.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jackson-core-2.3.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jackson-core-asl-1.8.10.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jackson-databind-2.3.1-cloudera.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jackson-mapper-asl-1.8.10-cloudera.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/javax.servlet-3.0.0.v201112011016.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jcl-over-slf4j-1.7.5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jdom-1.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jempbox-1.8.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jersey-core-1.9.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jersey-server-1.9.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jetty-continuation-8.1.10.v20130312.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jetty-deploy-8.1.10.v20130312.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jetty-http-8.1.10.v20130312.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jetty-io-8.1.10.v20130312.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jetty-jmx-8.1.10.v20130312.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jetty-security-8.1.10.v20130312.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jetty-server-8.1.10.v20130312.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jetty-servlet-8.1.10.v20130312.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jetty-util-8.1.10.v20130312.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jetty-webapp-8.1.10.v20130312.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jetty-xml-8.1.10.v20130312.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jhighlight-1.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/joda-time-1.6.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jsch-0.1.42.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/jsr305-1.3.9.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/juniversalchardet-1.0.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-hadoop-compatibility.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-avro.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-core.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-hadoop-core.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-hadoop-parquet-avro.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-hadoop-rcfile.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-hadoop-sequencefile.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-json.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-maxmind.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-metrics-servlets.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-saxon.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-solr-cell.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-solr-core.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-tika-core.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-tika-decompress.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-twitter.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/kite-morphlines-useragent.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/leveldbjni-all-1.8.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/log4j-1.2.17.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-analyzers-common.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-analyzers-kuromoji.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-analyzers-phonetic.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-codecs.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-core.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-expressions.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-grouping.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-highlighter.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-join.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-memory.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-misc.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-queries.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-queryparser.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-spatial.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/lucene-suggest.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/maxmind-db-1.0.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/metadata-extractor-2.6.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/metrics-core-3.0.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/metrics-healthchecks-3.0.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/metrics-json-3.0.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/metrics-jvm-3.0.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/metrics-servlets-3.0.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/netcdf-4.2-min.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/netty-3.10.5.Final.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/netty-all-4.0.23.Final.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/noggit-0.5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/org.restlet-2.1.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/org.restlet.ext.servlet-2.1.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/paranamer-2.3.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/parquet-avro.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/parquet-column.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/parquet-common.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/parquet-encoding.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/parquet-format.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/parquet-hadoop.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/parquet-jackson.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/pdfbox-1.8.4.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/poi-3.10-beta2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/poi-ooxml-3.10-beta2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/poi-ooxml-schemas-3.10.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/poi-scratchpad-3.10-beta2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/protobuf-java-2.5.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/rome-0.9.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/Saxon-HE-9.5.1-5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/snakeyaml-1.10.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/snappy-java-1.0.4.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/solr-cell.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/solr-core.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/solr-solrj.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/spatial4j-0.4.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/tagsoup-1.2.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/tika-core-1.5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/tika-parsers-1.5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/tika-xmp-1.5.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/ua-parser-1.3.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/vorbis-java-core-0.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/vorbis-java-core-0.1-tests.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/vorbis-java-tika-0.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/wstx-asl-3.2.7.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/xercesImpl-2.9.1.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/xml-apis-1.3.04.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/xmlbeans-2.6.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/xmlenc-0.52.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/xmpcore-5.1.2.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/xz-1.0.jar:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/flume-ng/../search/lib/zookeeper.jar' -Djava.library.path=:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.16.1-1.cdh5.16.1.p0.3/lib/hbase/bin/../lib/native/Linux-amd64-64 org.apache.flume.node.Application -f /data/test.properties -n test
19/01/15 11:41:42 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
19/01/15 11:41:42 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/data/test.properties
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:hdfssink
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:hdfssink
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:hdfssink
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:hdfssink
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:hdfssink
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:hdfssink
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:hdfssink
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Added sinks: hdfssink Agent: test
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:c1
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:hdfssink
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Processing:sqls
19/01/15 11:41:42 WARN conf.FlumeConfiguration: Agent configuration for 'test' has no configfilters.
19/01/15 11:41:42 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [test]
19/01/15 11:41:42 INFO node.AbstractConfigurationProvider: Creating channels
19/01/15 11:41:42 INFO channel.DefaultChannelFactory: Creating instance of channel c1 type memory
19/01/15 11:41:42 INFO node.AbstractConfigurationProvider: Created channel c1
19/01/15 11:41:42 INFO source.DefaultSourceFactory: Creating instance of source sqls, type org.keedio.flume.source.SQLSource
19/01/15 11:41:42 INFO source.SQLSource: Reading and processing configuration values for source sqls
19/01/15 11:41:42 INFO common.Version: HCANN000001: Hibernate Commons Annotations {4.0.5.Final}
19/01/15 11:41:42 INFO hibernate.Version: HHH000412: Hibernate Core {4.3.10.Final}
19/01/15 11:41:42 INFO cfg.Environment: HHH000206: hibernate.properties not found
19/01/15 11:41:42 INFO cfg.Environment: HHH000021: Bytecode provider name : javassist
19/01/15 11:41:42 INFO source.HibernateHelper: Opening hibernate session
19/01/15 11:41:42 INFO internal.ConnectionProviderInitiator: HHH000130: Instantiating explicit connection provider: org.hibernate.connection.C3P0ConnectionProvider
19/01/15 11:41:42 INFO internal.C3P0ConnectionProvider: HHH010002: C3P0 using driver: com.mysql.jdbc.Driver at URL: jdbc:mysql://192.168.199.237:3306/testflume
19/01/15 11:41:42 INFO internal.C3P0ConnectionProvider: HHH000046: Connection properties: {user=root, password=****, autocommit=true}
19/01/15 11:41:42 INFO internal.C3P0ConnectionProvider: HHH000006: Autocommit mode: true
19/01/15 11:41:42 INFO log.MLog: MLog clients using log4j logging.
19/01/15 11:41:42 INFO c3p0.C3P0Registry: Initializing c3p0-0.9.2.1 [built 20-March-2013 10:47:27 +0000; debug? true; trace: 10]
19/01/15 11:41:42 INFO impl.AbstractPoolBackedDataSource: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@7cfc77e7 [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@1fd7de4b [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1hgf4cha0m33vwywt512i|5e7d509f, idleConnectionTestPeriod -> 0, initialPoolSize -> 1, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 10, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 1, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@d530b185 [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 1hgf4cha0m33vwywt512i|b07ce20, jdbcUrl -> jdbc:mysql://192.168.199.237:3306/testflume, properties -> {user=******, password=******, autocommit=true} ], preferredTestQuery -> null, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 1hgf4cha0m33vwywt512i|74150459, numHelperThreads -> 3 ]
Tue Jan 15 11:41:42 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Tue Jan 15 11:41:42 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
19/01/15 11:41:42 INFO dialect.Dialect: HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect
19/01/15 11:41:42 INFO internal.LobCreatorBuilder: HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException
19/01/15 11:41:43 INFO internal.TransactionFactoryInitiator: HHH000399: Using default transaction strategy (direct JDBC transactions)
19/01/15 11:41:43 INFO ast.ASTQueryTranslatorFactory: HHH000397: Using ASTQueryTranslatorFactory
19/01/15 11:41:43 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfssink, type: hdfs
19/01/15 11:41:43 INFO node.AbstractConfigurationProvider: Channel c1 connected to [sqls, hdfssink]
19/01/15 11:41:43 INFO node.Application: Starting new configuration:{ sourceRunners:{sqls=PollableSourceRunner: { source:org.keedio.flume.source.SQLSource{name:sqls,state:IDLE} counterGroup:{ name:null counters:{} } }} sinkRunners:{hdfssink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@f2cf2a9 counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
19/01/15 11:41:43 INFO node.Application: Starting Channel c1
19/01/15 11:41:43 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
19/01/15 11:41:43 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started
19/01/15 11:41:43 INFO node.Application: Starting Sink hdfssink
19/01/15 11:41:43 INFO node.Application: Starting Source sqls
19/01/15 11:41:43 INFO source.SQLSource: Starting sql source sqls ...
19/01/15 11:41:43 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: SOURCESQL.sqls: Successfully registered new MBean.
19/01/15 11:41:43 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: SOURCESQL.sqls started
19/01/15 11:41:43 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: hdfssink: Successfully registered new MBean.
19/01/15 11:41:43 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfssink started
19/01/15 11:41:43 INFO hdfs.HDFSDataStream: Serializer = TEXT, UseRawLocalFileSystem = false
19/01/15 11:41:43 INFO hdfs.BucketWriter: Creating hdfs://nameservice1:8020/flume/mysqls/FlumeData.1547523703138.tmp
Tue Jan 15 11:41:45 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Tue Jan 15 11:41:45 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

没有报错说明没问题，然后查看hdfs下的文件

[root@mf01 ~]# hdfs dfs -tail -f /flume/mysqls/FlumeData.1547523796517.tmp
1;haha;13
2;hah;13
3;haa;13
4;ha;13
5;aha;13
6;haa;13
7;qqq;12
8;qdg;12
9;zxcsdf;12
10;zxcsdf;12
11;zxcsdf;12
12;zxcsdf;12
13;zxcsdf;12
14;zxcsdf;12
15;zxcsdf;12
16;zxcsdf;12
17;zxcsdf;12
18;zxydf;12
19;zxydf;12
20;zxydf;12
21;zxydf;12
22;zxydf;12
23;qqqq;12

没有问题，接着我们向数据库插入一条数据
在这里插入图片描述

[root@mf01 ~]# hdfs dfs -tail -f /flume/mysqls/FlumeData.1547523796517.tmp
1;haha;13
2;hah;13
3;haa;13
4;ha;13
5;aha;13
6;haa;13
7;qqq;12
8;qdg;12
9;zxcsdf;12
10;zxcsdf;12
11;zxcsdf;12
12;zxcsdf;12
13;zxcsdf;12
14;zxcsdf;12
15;zxcsdf;12
16;zxcsdf;12
17;zxcsdf;12
18;zxydf;12
19;zxydf;12
20;zxydf;12
21;zxydf;12
22;zxydf;12
23;qqqq;12
24;aaa;12
24;aaa;12
24;aaa;12

然后就开始出问题了，会重复读最后的操作，我也不知道是什么原因，查看状态文件也正常

[root@mf04 ~]# vim /data/flume/sqls.status 

{"SourceName":"sqls","URL":"jdbc:mysql:\/\/192.168.199.237:3306\/testflume","LastIndex":"24","ColumnsToSelect":"*","Table":"test"}
~                                                                                                                                                                                                                                            
~

哪位大神能帮忙分析什么原因吗？

Flume实时采集mysql数据到kafka中并输出
Alex的博客
11-30 6313
Flume实时采集mysql数据到kafka中并输出。Apache Flume是一个分布式的、可靠的、可用的系统，用于有效地收集、聚合和将大量日志数据从许多不同的源移动到一个集中的数据存储。在大数据生态圈中，flume经常用于完成数据采集的工作。其实时性很高，延迟大约1-2s，可以做到准实时。又因为mysql是程序员常用的数据库，所以以flume实时采集mysql数据库为例子。要了解flume如何采集数据，首先要初探其架构：Flume 运行的核心是 Agent。Flume以agent为最小的独立运行单位。
mysql 数据实时到 hdfs和kafka
zahuali的博客
11-28 459
1.在mysql创建表导入数据 use test; create table wlslog (id int not null, time_stamp varchar(40), category varchar(40), type varchar(40), servername varchar(40), code varchar(40), msg ...
利用Flume将MySQL表数据准实时抽取到HDFS、MySQL、Kafka
03-06
利用Flume将MySQL表数据准实时抽取到HDFS、MySQL、Kafka用到的jar包
Flume采集MySQL写入HDFS
qq_65967263的博客
08-22 1392
mysql--->HDFS
利用Flume将MySQL表数据准实时抽取到HDFS
zhanggx512的博客
05-31 672
https://blog.youkuaiyun.com/wzy0623/article/details/73650053
五.hadoop 从mysql中读取数据写到hdfs
weixin_33811539的博客
07-19 304
目录： 目录见文章1 本文是基于windows下来操作，linux下，mysql-connector-java-5.1.46.jar包的放置有讲究。 mr程序 import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; import java.sql.PreparedStat...
flume mysql hdfs_利用Flume将MySQL表数据准实时抽取到HDFS
weixin_39670857的博客
01-19 99
[导读]在以前搭建HAWQ数据仓库实验环境时，我使用Sqoop抽取从MySQL数据库增量抽取数据到HDFS，然后用HAWQ的外部表进行访问。这种方式只需要很少量的配置即可完成数据抽取任务，但缺点同样明显，那就是实时性。一、为什么要用到Flume在以前搭建HAWQ数据仓库实验环境时，我使用Sqoop抽取从MySQL数据库增量抽取数据到HDFS，然后用HAWQ的外部表进行访问。这种方式只需要很少量的配...
flume抽取mysql数据至hdfs
wangfengjunaaa的专栏
09-25 1982
场景分析：一般情况下关系型数据库（mysql、oracle、sqlserver）数据抽取至hdfs、hive、hbase使用sqoop工具。 但sqoop抽取底层依靠mapreduce处理的实时性得不到保证。如果能将数据抽取和SparkStreaming+sparkSql结合将大大提高了处理效率。因而想到了flume抽取关系型数据库数据至kafka中，有SparkStreaming读取。本文介绍如
flume抽取mysql数据_利用Flume将MySQL表数据准实时抽取到HDFS
weixin_42523720的博客
01-26 996
一、为什么要用到Flume在以前搭建HAWQ数据仓库实验环境时，我使用Sqoop抽取从MySQL数据库增量抽取数据到HDFS，然后用HAWQ的外部表进行访问。这种方式只需要很少量的配置即可完成数据抽取任务，但缺点同样明显，那就是实时性。Sqoop使用MapReduce读写数据，而MapReduce是为了批处理场景设计的，目标是大吞吐量，并不太关心低延时问题。就像实验中所做的，每天定时增量抽取数据一...
ZDH大数据采集平台-采集mysql数据到hdfs
热门推荐
啊大海全是水的博客
03-07 210万+
目录 项目源码 创建mysql数据源 创建本地hdfs数据源 配置采集任务 配置调度 点击执行记录查看具体日志 高级使用 项目源码 数据采集平台管理端 https://github.com/zhaoyachao/zdh_web 数据采集平台服务 https://github.com/zhaoyachao/zdh_server web端在线查看 http://zycblog.cn:8081/login 用户名：zyc 密码：123456 界面只是为了参考功能,底层的数据.
flume 之监视日志数据抽出到hdfs上示例
maketubu7的博客
06-30 872
今天学习了一个新的轻量级框架，就是一个的实时日志收集系统，由cloudera公司开发的框架下面贴上代码##对log的完成数据，进行数据抽取，并对文件夹下所有文件进行实时监视 ###define agent a3.sources = r3 a3.channels = c3 a3.sinks = k3 #define sources a3.sources.r3.type = spooldir a3...
转载：Flume准实时采集MySQL数据
黄道婆的专栏
11-06 1596
原始链接：https://blog.youkuaiyun.com/lianghecai52171314/article/details/104919999 Flume准实时采集MySQL数据 梁云亮2020-03-17 13:59:52311收藏2原力计划 分类专栏：# Flume文章标签：实时采集MySQL数据实时采集 版权 需求 利用Flume将MySQL表数据准实时抽取到HDFS -软件版本 Flume：1.9.0 MySQL：5.7 db_test下面有tb_dept...
实时采集MySQL数据之轻量工具Maxwell实操
qq_35292638的博客
03-30 1704
Maxwell 官网地址Maxwell GitHub源码地址Maxwell 是由美国 Zendesk 开源,采用 Java 语言开发的 MySQL 实时抓取工具，通过实时读取MySQL binlog二进制日志并作为生产者生产 JSON 格式消息发送给 Kafka、Kinesis、RabbitMQ、Redis或其他流媒体平台的应用程序。最新版本为1.39.4Maxwell的操作开销很低，只需要mysql和数据同步目的地，常用场景包括ETL、缓存构建/过期、指标收集、搜索索引和服务间通信。
如何读取MySQL中的数据存放到HDFS_大数据培训
zjjcchina的博客
12-08 946
大数据培训读取MySQL中的数据存放到HDFS1 查看官方模板[atguigu@hadoop102 ~]$ python /opt/module/datax/bin/datax.py -r mysqlreader -w hdfswriter{ “job”: { “content”: [ { “reader”: { “name”: “mysqlreader”, “
flume可以实时监控mysql嘛_flume使用（三）：实时log4j日志通过flume输出到MySql数据库...
weixin_39717152的博客
02-23 478
本文在【flume使用(二)：采集远程日志数据到MySql数据库】一文基础之上进行测试操作。本文使用到的：flume版本、jdk版本、mysql、数据库表、javaBean、自定义的mysqlSink、以及多agent配置均同【flume使用(二)：采集远程日志数据到MySql数据库】一、需求说明采集远程系统上log4j实时生成的日志到本机上进行数据库持久化。二、设计数据流：(1)log4j(配置...
Flume-ng与Mysql整合开发
阿努比斯的藏宝箱
04-23 453
Refer To : http://www.iteblog.com/archives/1109　 我们知道，Flume可以和许多的系统进行整合，包括了Hadoop、Spark、Kafka、Hbase等等；当然，强悍的Flume也是可以和Mysql进行整合，将分析好的日志存储到Mysql（当然，你也可以存放到pg、oracle等等关系型数据库）。 　　不过我这里想多说一些：Flume是分
Flumeng和Mysql进行整合 Flumeng批量处理
liyonghui123的专栏
05-13 276
    package com.iteblog.flume; import com.google.common.base.Preconditions; import com.google.common.base.Throwables; import com.google.common.collect.Lists; import org.apache.flume.*...
Flume在数据抽取中的应用：MySQL到HDFS、MySQL和Kafka
本篇知识讲解主要围绕如何使用 Flume 将 MySQL 表中的数据实时抽取到 HDFS（Hadoop Distributed File System）、MySQL 和 Kafka 中。实现这一功能的前提是需要了解 Flume 的工作原理和架构，以及如何配置 Flume 以...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值