大数据技术Flink电商实时数仓-数据采集--第四章 日志数据采集

4.1 模拟日志生成器的作用

这里提供了一个模拟生成数据的 jar 包，可以将日志发送给某一个指定的端口，需要大数据程序员了解如何从指定端口接收数据并数据进行处理的流程。

(1) 拷贝/资料/数据生成脚本/行为数据的内容到 hadoop102 的/opt/module/rt_applog目录

[hadoop@hadoop102 rt_applog]$ pwd
/opt/module/rt_applog
[hadoop@hadoop102 rt_applog]$ ll
总用量 15284
-rw-rw-r--. 1 hadoop hadoop      981 3月   2 21:51 application.yml
-rw-rw-r--. 1 hadoop hadoop 15642393 3月   1 07:31 gmall2020-mock-log.jar

(2) 根据实际需要修改 application.yml

# 外部配置打开
# logging.config=./logback.xml
#业务日期
mock.date: "2021-04-11"

  #模拟数据发送模式
mock.type: "http"
  #http模式下，发送的地址
mock.url: "http://hadoop102:8081/applog"

#mock:
#  kafka-server: "hdp1:9092,hdp2:9092,hdp3:9092"
#  kafka-topic: "ODS_BASE_LOG"

#启动次数
mock.startup.count: 1000
  #设备最大值
mock.max.mid: 20
  #会员最大值
mock.max.uid: 50
  #商品最大值
mock.max.sku-id: 10
  #页面平均访问时间
mock.page.during-time-ms: 20000
  #错误概率 百分比
mock.error.rate: 3
  #每条日志发送延迟 ms
mock.log.sleep: 100
  #商品详情来源  用户查询，商品推广，智能推荐, 促销活动
mock.detail.source-type-rate: "40:25:15:20"

#领取购物券概率
mock.if_get_coupon_rate: 75

#购物券最大id
mock.max.coupon-id: 3

  #搜索关键词  
mock.search.keyword: "图书,小米,iphone11,电视,口红,ps5,苹果手机,小米盒子"

（3）上传日志采集服务到 hadoop102 的/opt/module/rt_gmall目录，采集到用户行为数据，并发送的kafka中，主题为： ods_base_log

[hadoop@hadoop102 rt_gmall]$ pwd
/opt/module/rt_gmall
[hadoop@hadoop102 rt_gmall]$ ll
总用量 28540
drwxrwxr-x. 2 hadoop hadoop     4096 4月  14 06:37 gmall0820
-rw-rw-r--. 1 hadoop hadoop 29217691 3月   1 08:05 gmall0820-logger-0.0.1-SNAPSHOT.jar

（4）测试

运行 hadoop102 上的 rt_gmall 下的日志处理 jar 包；

[hadoop@hadoop102 rt_gmall]$ java -jar gmall0820-logger-0.0.1-SNAPSHOT.jar 

运行 rt_applog 下的 jar 包；

[hadoop@hadoop102 rt_applog]$ java -jar gmall2020-mock-log.jar 

启动 kafka 消费者进行测试。

bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --topic ods_base_log

运行成功后，采集服务的控制台会看到模拟数据

[hadoop@hadoop102 rt_gmall]$ java -jar gmall0820-logger-0.0.1-SNAPSHOT.jar 

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.4.2)

{"common":{"ar":"310000","ba":"Xiaomi","ch":"360","is_new":"1","md":"Xiaomi Mix2 ","mid":"mid_5","os":"Android 11.0","uid":"1","vc":"v2.1.134"},"start":{"entry":"icon","loading_time":12799,"open_ad_id":4,"open_ad_ms":2649,"open_ad_skip_ms":2647},"ts":1618098983000}
{"common":{"ar":"440000","ba":"Xiaomi","ch":"oppo","is_new":"1","md":"Xiaomi 10 Pro ","mid":"mid_19","os":"Android 10.0","uid":"44","vc":"v2.1.134"},"start":{"entry":"icon","loading_time":6709,"open_ad_id":14,"open_ad_ms":4284,"open_ad_skip_ms":3015},"ts":1618098994000}
{"common":{"ar":"440000","ba":"iPhone","ch":"Appstore","is_new":"0","md":"iPhone X","mid":"mid_3","os":"iOS 13.3.1","uid":"45","vc":"v2.1.132"},"start":{"entry":"notice","loading_time":10174,"open_ad_id":14,"open_ad_ms":3409,"open_ad_skip_ms":2584},"ts":1618099004000}
{"common":{"ar":"310000","ba":"Xiaomi","ch":"oppo","is_new":"0","md":"Xiaomi Mix2 ","mid":"mid_2","os":"Android 10.0","uid":"29","vc":"v2.1.134"},"start":{"entry":"icon","loading_time":7854,"open_ad_id":2,"open_ad_ms":4526,"open_ad_skip_ms":0},"ts":1618099014000}
{"common":{"ar":"440000","ba":"iPhone","ch":"Appstore","is_new":"1","md":"iPhone X","mid":"mid_16","os":"iOS 13.3.1","uid":"18","vc":"v2.1.134"},"err":{"error_code":3617,"msg":" Exception in thread \\  java.net.SocketTimeoutException\\n \\tat com.atgugu.gmall2020.mock.log.bean.AppError.main(AppError.java:xxxxxx)"},"start":{"entry":"icon","loading_time":8196,"open_ad_id":4,"open_ad_ms":9346,"open_ad_skip_ms":3660},"ts":1618099024000}

kafka 控制台也能看到消费的消息。

模拟数据生成：https://download.youkuaiyun.com/download/jsp_will/18776231

数据采集：https://download.youkuaiyun.com/download/jsp_will/18776124