数据采集项目实战-下

最新推荐文章于 2025-02-21 08:00:00 发布

张培根同学

最新推荐文章于 2025-02-21 08:00:00 发布

阅读量678

点赞数

分类专栏：学习记录文章标签： kafka java 大数据

本文链接：https://blog.youkuaiyun.com/TTDZS/article/details/128246506

版权

本文档详细介绍了如何配置和使用Kafka、Flume进行数据采集。主要内容包括Kafka的配置和启动，Flume的配置与日志采集，拦截器的设置，MySQL的配置以及Maxwell用于增量数据同步，最后提到了DataX用于全量表数据同步到HDFS的操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Kafka

配置Kafka

配置server.properties

/opt/module/kafka/config目录下的该文件,主要修改以下三行

broker.id=2
log.dirs=/opt/module/kafka/datas
zookeeper.connect=bigdata102:2181,bigdata103:2181,bigdata104:2181/kafka

修改后

#broker的全局唯一编号，不能重复，只能是数字。id对应主机名方便区分
broker.id=2
#处理网络请求的线程数量
num.network.threads=3
#用来处理磁盘IO的线程数量
num.io.threads=8
#发送套接字的缓冲区大小
socket.send.buffer.bytes=102400
#接收套接字的缓冲区大小
socket.receive.buffer.bytes=102400
#请求套接字的缓冲区大小
socket.request.max.bytes=104857600
#kafka运行日志(数据)存放的路径，路径不需要提前创建，kafka自动帮你创建，可以配置多个磁盘路径，路径与路径之间可以用"，"分隔
log.dirs=/opt/module/kafka/datas
#topic在当前broker上的分区个数
num.partitions=1
#用来恢复和清理data下数据的线程数量
num.recovery.threads.per.data.dir=1
# 每个topic创建时的副本数，默认时1个副本
offsets.topic.replication.factor=1
#segment文件保留的最长时间，超时将被删除
log.retention.hours=168
#每个segment文件的大小，默认最大1G
log.segment.bytes=1073741824
# 检查过期数据的时间，默认5分钟检查一次是否数据过期
log.retention.check.interval.ms=300000
#配置连接Zookeeper集群地址（在zk根目录下创建/kafka，方便管理）
zookeeper.connect=bigdata102:2181,bigdata103:2181,bigdata104:2181/kafka

分别修改bigdata103、bigdata104上的broker.id为3，4

追加环境变量到my_env.sh

#KAFKA_HOME
export KAFKA_HOME=/opt/module/kafka
export PATH=$PATH:$KAFKA_HOME/bin

切换root用户分发my_env.sh

source /etc/profile

启动Kafka

先启动zookeeper集群

zk.sh start

再启动kafka集群

kf.sh start

以下是kf.sh脚本内容

#! /bin/bash

case $1 in
"start"){
    for i in bigdata102 bigdata103 bigdata104
    do
        echo " --------启动 $i Kafka-------"
        ssh $i "/opt/module/kafka/bin/kafka-server-start.sh -daemon /opt/module/kafka/config/server.properties"
    done
};;
"stop"){
    for i in bigdata102 bigdata103 bigdata104
    do
        echo " --------停止 $i Kafka-------"
        ssh $i "/opt/module/kafka/bin/kafka-server-stop.sh "
    done
};;
esac

Flume

配置Flume

将lib文件夹下的guava-11.0.2.jar删除以兼容Hadoop 3.1.3

rm /opt/module/flume/lib/guava-11.0.2.jar

修改conf目录下的log4j.properties配置文件，配置日志文件路径

vim log4j.properties
## 修改日志目录
flume.log.dir=/opt/module/flume/logs

xsync分发flume目录

配置日志采集Flume

Flume配置文件

kafka上游：

bigdata102的Flume的job目录下创建file_to_kafka.conf

因为在线教育项目的数据，业务数据和用户行为日志数据都是一个jar包生成的，所以kafka上游只配置一个flume就可以

内容：

#定义组件
a1.sources = r1
a1.channels = c1

#配置source
a1.sources.r1.type = TAILDIR
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /opt/module/applog/log/app.*
a1.sources.r1.positionFile = /opt/module/flume/taildir_position.json
a1.sources.r1.interceptors =  i1
a1.sources.r1.interceptors.i1.type = com.atguigu.edu.flume.interceptor.ETLInterceptor$Builder

#配置channel
a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.c1.kafka.bootstrap.servers = bigdata102:9092
a1.channels.c1.kafka.topic = topic_log
a1.channels.c1.parseAsFlumeEvent = false

#组装 
a1.sources.r1.channels = c1

日志生成

日志生成脚本lg.sh

#!/bin/bash
  xcall -w 'bigdata102' 'cd /opt/module/data_mocker;java -jar edu2021-mock-2022-06-18.jar >/dev/null 2>&1 &'

将jar包和脚本传到目录/opt/module/data_mocker

mkdir /opt/module/data_mocker

日志采集脚本f1.sh

log—>kafka

#!/bin/bash

case $1 in
"start"){
   
        for i in bigdata102
        do
                echo " --------启动 $i 采集flume-------"
                ssh $i "nohup /opt/module/flume/bin/flume-ng agent -n a1 -c /opt/module/flume/conf/ -f /opt/module/flume/job/file_to_kafka.conf >/dev/null 2>&1 &"
        done
};;
"stop"){
   
        for i in bigdata102
        do
                echo " --------停止 $i 采集flume-------"
                ssh $i "ps -ef | grep file_to_kafka | grep -v grep |awk  '{print \$2}' | xargs -n1 kill -9 "
        done

};;
esac

kafka下游

bigdata104的Flume的job目录下创建kafka_to_hdfs_db.conf

a1.sources = r1
a1.channels = c1
a1.sinks = k1

a1.sources.r1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.r1.batchSize = 5000
a1.sources.r1.batchDurationMillis = 2000
a1.sources.r1.kafka.bootstrap.servers = bigdata102:9092
a1.sources.r1.kafka.topics = topic_db
a1.sources.r1.kafka.consumer.group.id = flume
a1.sources.r1.setTopicHeader = true
a1.sources.r1.topicHeader = topic
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = com.atguigu.edu.flume.interceptor.TimestampAndTableNameInterceptor$Builder

a1.channels.c1.type = file
a1.channels.c1.checkpointDir = /opt/module/flume/checkpoint/behavior2
a1.channels.c1.dataDirs = /opt/module/flume/data/behavior2/
a1.channels.c1.maxFileSize = 2146435071
a1.channels.c1.capacity = 1000000
a1.channels.c1.keep-alive = 6

## sink1
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = /origin_data/edu/db/%{tableName}_inc/%Y-%m-%d
a1.sinks.k1.hdfs.filePrefix = db
a1.sinks.k1.hdfs.round = false


a1.sinks.k1.hdfs.rollInterval = 10
a1.sinks.k1.hdfs.rollSize = 134217728
a1.sinks.k1.hdfs.rollCount = 0


a1.sinks.k1.hdfs.fileType = CompressedStream
a1.sinks.k1.hdfs.codeC = gzip

## 拼装
a1.sources.r1.channels = c1
a1.sinks.k1.channel= c1

业务数据采集脚本f2.sh

kafka—>hdfs

#!/bin/bash

case $1 in
"start")
        echo " --------启动 bigdata104 日志数据flume-------"
        ssh bigdata104 "nohup /opt/module/flume/bin/flume-ng agent -n a1 -c /opt/module/flume/conf -f /opt/module/flume/job/kafka_to_hdfs_db.conf >/dev/null 2>&1 &"
;;
"stop")

        echo " --------停止 bigdata104 日志数据flume-------"
        ssh bigdata104 "ps -ef | grep kafka_to_hdfs_db | grep -v grep |awk '{print \$2}' | xargs -n1 kill"
;;
esac

拦截器

maven项目pom.xml添加依赖

 <dependencies>
        <dependency>
            <groupId>org.apache.flume</groupId>
            <artifactId>flume-ng-core</artifactId>
            <version>1.9.0</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.62</version>
        </dependency>
    </dependencies>

创建com.atguigu.edu.flume.interceptor包

编写以下内容

ETLInterceptor

package com.atguigu.edu.flume.interceptor;

import com.atguigu.edu.flume.utils.JSONUtil;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;


import java.nio.charset.StandardCharsets;
import java.util.Iterator;
import java.util.List;

public class ETLInterceptor implements Interceptor {
   

    @Override
    public void initialize() {
   

    }

    @Override
    public Event intercept(Event event) {
   

        //1、获取body当中的数据并转成字符串
        byte[] body = event.getBody();
        String log = new String(body, StandardCharsets.UTF_8);
        //2、判断字符串是否是一个合法的json，是：返回当前event；不是&