2017.05.19:MySQL问题

本文解析了MySQL中常见的“MySQL server has gone away”错误的原因,并提供了排查思路。同时,针对Python MySQLdb执行execute()时出现的InterfaceError进行了分析。

1. “MySQL has Gone Away”

mysql出现ERROR: (2006, 'MySQL server has gone away') 的问题意思就是指client和MySQLserver之间的链接断开了。造成这样的原因一般是sql操作的时间过长,或者是传送的数据太大(例如使用insert... values的语句过长

 

原因一. MySQL 服务宕了

判断是否属于这个原因的方法很简单,进入mysql控制台,查看mysql的运行时长

mysql> show global status like 'uptime';

 

原因二. mysql连接超时

即某个mysql长连接很久没有新的请求发起,达到了server端的timeout,被server强行关闭。此后再通过这个connection发起查询的时候,就会报错server has gone away

 

原因三. mysql请求链接进程被主动kill

这种情况和原因二相似,只是一个是人为一个是MYSQL自己的动作

mysql> show global status like 'com_kill';

 

原因四. YourSQL statement was too large.

当查询的结果集超过max_allowed_packet 也会出现这样的报错。定位方法是打出相关报错的语句。用select* into outfile 的方式导出到文件,查看文件大小是否超过max_allowed_packet ,如果超过则需要调整参数,或者优化语句。



2.Python mysqldb 执行execute()时出现InterfaceError:(0,’’)

对任何已经close的conn进行 db相关 操作,包括ping()都会爆出这个错误。(这说明 长时间idle导致的conn失效与conn.close()之后的状态是不一样的)精确catch 这个错误的Exception 是MySQLdb.Error

 

conn.close()写早了,一个py文件里只要有一个close()就够

import java.text.SimpleDateFormat import java.util.Date import org.apache.kafka.common.serialization.StringDeserializer import org.apache.kafka.common.TopicPartition import org.apache.spark.SparkConf import org.apache.spark.streaming.{Seconds, StreamingContext} import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies} object KafkaStream { def main(args: Array[String]): Unit = { val kafkaParams = Map[String, Object]("bootstrap.servers" -> "127.0.0.1:9092", "group.id" -> "kafkaStream", "enable.auto.commit" -> "false", "key.deserializer" -> classOf[StringDeserializer], "value.deserializer" -> classOf[StringDeserializer], "group.id" -> "g1" ) val partition = new TopicPartition("test", 0) val list = List(partition) val offsets = Map(partition -> 0l) val conf = new SparkConf().setMaster("local[*]").setAppName("kafkaStream") /********** Begin **********/ //1.初始化StreamingContext,设置时间间隔为1S //2.使用 KafkaUtils 对象创建流,使用 Assign 订阅主题(Topic),上面已经为你定义好了 Topic列表:list,kafka参数:kafkaParams,偏移量:offsets /** * * 数据格式如下: * 100.143.124.29,1509116285000,'GET www/1 HTTP/1.0',https://www.baidu.com/s?wd=反叛的鲁鲁修,404 * 数据从左往右分别代表:用户IP、访问时间戳、起始URL及相关信息(访问方式,起始URL,http版本)、目标URL、状态码 * * * 原始数据的切割符为逗号,(英文逗号) * * 需求: * 1.将时间戳转换成规定时间(格式为:yyyy-MM-dd HH:mm:ss ) * 2.提取数据中的起始URL(切割符为空格) * 3.拼接结果数据,格式如下: * Ip:124.132.29.10,visitTime:2019-04-22 11:08:33,startUrl:www/2,targetUrl:https://search.yahoo.com/search?p=反叛的鲁鲁修,statusCode:200 * 4.判断rdd是否为空,如果为空,调用 ssc.stop(false, false)与sys.exit(0) 两个方法,反之将结果数据存储到mysql数据库中,调用DBUtils.add(line)即可, line:String */ //3.获取kafka流中的数据,进行清洗、转换(按照上面的需求) //4.判断rdd是否为空,如果为空,调用 ssc.stop(false, false)与sys.exit(0) 两个方法,反之将结果数据存储到mysql数据库中,调用DBUtils.add(line)即可, line:String //5.启动SparkStreaming //6.等待计算结束 /********** End **********/ } } 平台将对你编写的代码进行评测: 预期输出: 1 Ip:100.143.124.29,visitTime:2017-10-27 14:58:05,startUrl:www/1,targetUrl:https://www.baidu.com/s?wd=反叛的鲁鲁修,statusCode:404 2 Ip:30.132.167.100,visitTime:2018-12-02 11:29:39,startUrl:www/4,targetUrl:-,statusCode:302 3 Ip:30.156.187.132,visitTime:2016-05-17 17:18:56,startUrl:www/2,targetUrl:-,statusCode:200 4 Ip:29.100.10.30,visitTime:2016-10-12 01:25:47,startUrl:www/3,targetUrl:http://cn.bing.com/search?q=游戏人生,statusCode:302 5 Ip:132.187.167.143,visitTime:2017-01-08 23:21:09,startUrl:pianhua/130,targetUrl:-,statusCode:200 6 Ip:143.187.100.10,visitTime:2016-09-21 19:27:39,startUrl:www/1,targetUrl:-,statusCode:302 7 Ip:10.100.124.30,visitTime:2018-09-16 02:49:35,startUrl:www/4,targetUrl:http://cn.bing.com/search?q=来自新世界,statusCode:200 8 Ip:29.10.143.187,visitTime:2017-09-29 15:49:09,startUrl:www/1,targetUrl:-,statusCode:404 9 Ip:29.187.132.100,visitTime:2018-11-27 05:43:17,startUrl:www/1,targetUrl:-,statusCode:200 10 Ip:187.167.124.132,visitTime:2016-01-28 13:34:33,startUrl:www/6,targetUrl:-,statusCode:200
最新发布
11-28
2025-05-19 10:33:54 [JobThread.run-130] <br>----------- datax-web job execute start -----------<br>----------- Param: 2025-05-19 10:33:54 [BuildCommand.buildDataXParam-100] ------------------Command parameters:-p"-DlastTime='2025-05-19%08:00:00' -DcurrentTime='2025-05-19%10:33:54'" 2025-05-19 10:33:54 [ExecutorJobHandler.execute-57] ------------------DataX process id: 2669844 2025-05-19 10:33:54 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:54 [AnalysisStatistics.analysisStatisticsLog-53] DataX (DATAX-OPENSOURCE-3.0), From Alibaba ! 2025-05-19 10:33:54 [AnalysisStatistics.analysisStatisticsLog-53] Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved. 2025-05-19 10:33:54 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:54 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:54 [ProcessCallbackThread.callbackLog-186] <br>----------- datax-web job callback finish. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.146 [main] INFO MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.148 [main] INFO MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.161 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.167 [main] INFO Engine - the machine info => 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] osInfo: Linux amd64 4.19.90-17.ky10.x86_64 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] jvmInfo: Huawei Technologies Co., Ltd 1.8 25.242-b08 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] cpu num: 32 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] totalPhysicalMemory: -0.00G 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] freePhysicalMemory: -0.00G 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] maxFileDescriptorCount: -1 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] currentOpenFileDescriptorCount: -1 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] GC Names [PS MarkSweep, PS Scavenge] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] MEMORY_NAME | allocation_size | init_size 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] PS Eden Space | 256.00MB | 256.00MB 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] Code Cache | 240.00MB | 2.44MB 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] Compressed Class Space | 1,024.00MB | 0.00MB 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] PS Survivor Space | 42.50MB | 42.50MB 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] PS Old Gen | 683.00MB | 683.00MB 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] Metaspace | -0.00MB | 0.00MB 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.181 [main] INFO Engine - 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] { 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "content":[ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] { 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "reader":{ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "parameter":{ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "password":"**********", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "connection":[ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] { 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "querySql":[ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "select CONCAT(comm_code,id) as id, `uuid`, `record_time`, `open_mode`, `comm_code`, `open_object`, `name`, `phone_num`, `person_type`, `open_device`, `device_place`, `person_uuid`, `picture_url`, `open_video`, `created_by`, `created_time`, `updated_by`, `updated_time`, `del_flag`, `mczt`, `pass`, `in_out_flag`, `temp`, `temp_normal`, `extra`, `health_info`, `created_time` as GY_RKGXSJ, case when `del_flag`=2 then 1 else 0 end as GY_SFYSC from `door_open_record` where `record_time` >= '2025-05-19%08:00:00' and `record_time` < '2025-05-19%10:33:54'" 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] ], 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "jdbcUrl":[ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "jdbc:mysql://172.20.20.57:8066/wuye?serverTimezone=Asia/Shanghai&useLegacyDatetimeCode=false&useSSL=false&nullNamePatternMatchesAll=true&useUnicode=true&characterEncoding=UTF-8" 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] ] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] } 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] ], 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "splitPk":"", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "username":"root" 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] }, 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "name":"mysqlreader" 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] }, 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "writer":{ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "parameter":{ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "password":"**********", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "column":[ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`id`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`uuid`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`record_time`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`open_mode`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`comm_code`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`open_object`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`name`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`phone_num`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`person_type`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`open_device`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`device_place`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`person_uuid`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`picture_url`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`open_video`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`created_by`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`created_time`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`updated_by`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`updated_time`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`del_flag`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`mczt`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`pass`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`in_out_flag`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`temp`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`temp_normal`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`extra`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`health_info`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`GY_RKGXSJ`", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "`GY_SFYSC`" 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] ], 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "connection":[ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] { 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "jdbcUrl":"jdbc:mysql://172.20.20.67:3306/kitty?serverTimezone=Asia/Shanghai&useLegacyDatetimeCode=false&useSSL=false&nullNamePatternMatchesAll=true&useUnicode=true&characterEncoding=UTF-8", 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "table":[ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "door_open_record" 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] ] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] } 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] ], 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "username":"root" 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] }, 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "name":"mysqlwriter" 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] } 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] } 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] ], 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "setting":{ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "errorLimit":{ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "record":100, 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "percentage":0.02 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] }, 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "speed":{ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "byte":1048576, 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] "channel":3 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] } 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] } 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] } 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.200 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.200 [main] INFO JobContainer - DataX jobContainer starts job. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.201 [main] INFO JobContainer - Set jobId = 0 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.533 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://172.20.20.57:8066/wuye?serverTimezone=Asia/Shanghai&useLegacyDatetimeCode=false&useSSL=false&nullNamePatternMatchesAll=true&useUnicode=true&characterEncoding=UTF-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.791 [job-0] INFO OriginalConfPretreatmentUtil - table:[door_open_record] all columns:[ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] id,uuid,record_time,open_mode,comm_code,open_object,name,phone_num,person_type,open_device,device_place,person_uuid,picture_url,open_video,created_by,created_time,updated_by,updated_time,del_flag,mczt,pass,in_out_flag,GY_RKGXSJ,GY_SFYSC,temp,temp_normal,extra,health_info 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] ]. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.798 [job-0] INFO OriginalConfPretreatmentUtil - Write data [ 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] INSERT INTO %s (`id`,`uuid`,`record_time`,`open_mode`,`comm_code`,`open_object`,`name`,`phone_num`,`person_type`,`open_device`,`device_place`,`person_uuid`,`picture_url`,`open_video`,`created_by`,`created_time`,`updated_by`,`updated_time`,`del_flag`,`mczt`,`pass`,`in_out_flag`,`temp`,`temp_normal`,`extra`,`health_info`,`GY_RKGXSJ`,`GY_SFYSC`) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] ], which jdbcUrl like:[jdbc:mysql://172.20.20.67:3306/kitty?serverTimezone=Asia/Shanghai&useLegacyDatetimeCode=false&useSSL=false&nullNamePatternMatchesAll=true&useUnicode=true&characterEncoding=UTF-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&rewriteBatchedStatements=true&tinyInt1isBit=false] 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.798 [job-0] INFO JobContainer - jobContainer starts to do prepare ... 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.799 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work . 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.799 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] do prepare work . 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.799 [job-0] INFO JobContainer - jobContainer starts to do split ... 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.800 [job-0] INFO JobContainer - Job set Max-Byte-Speed to 1048576 bytes. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.801 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.802 [job-0] INFO JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.827 [job-0] INFO JobContainer - jobContainer starts to do schedule ... 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.830 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.832 [job-0] INFO JobContainer - Running by standalone Mode. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.836 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.840 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to 2000000. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.840 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated. 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.847 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:55.850 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select CONCAT(comm_code,id) as id, `uuid`, `record_time`, `open_mode`, `comm_code`, `open_object`, `name`, `phone_num`, `person_type`, `open_device`, `device_place`, `person_uuid`, `picture_url`, `open_video`, `created_by`, `created_time`, `updated_by`, `updated_time`, `del_flag`, `mczt`, `pass`, `in_out_flag`, `temp`, `temp_normal`, `extra`, `health_info`, `created_time` as GY_RKGXSJ, case when `del_flag`=2 then 1 else 0 end as GY_SFYSC from `door_open_record` where `record_time` >= '2025-05-19%08:00:00' and `record_time` < '2025-05-19%10:33:54' 2025-05-19 10:33:55 [AnalysisStatistics.analysisStatisticsLog-53] ] jdbcUrl:[jdbc:mysql://172.20.20.57:8066/wuye?serverTimezone=Asia/Shanghai&useLegacyDatetimeCode=false&useSSL=false&nullNamePatternMatchesAll=true&useUnicode=true&characterEncoding=UTF-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2025-05-19 10:33:56 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:56.927 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select CONCAT(comm_code,id) as id, `uuid`, `record_time`, `open_mode`, `comm_code`, `open_object`, `name`, `phone_num`, `person_type`, `open_device`, `device_place`, `person_uuid`, `picture_url`, `open_video`, `created_by`, `created_time`, `updated_by`, `updated_time`, `del_flag`, `mczt`, `pass`, `in_out_flag`, `temp`, `temp_normal`, `extra`, `health_info`, `created_time` as GY_RKGXSJ, case when `del_flag`=2 then 1 else 0 end as GY_SFYSC from `door_open_record` where `record_time` >= '2025-05-19%08:00:00' and `record_time` < '2025-05-19%10:33:54' 2025-05-19 10:33:56 [AnalysisStatistics.analysisStatisticsLog-53] ] jdbcUrl:[jdbc:mysql://172.20.20.57:8066/wuye?serverTimezone=Asia/Shanghai&useLegacyDatetimeCode=false&useSSL=false&nullNamePatternMatchesAll=true&useUnicode=true&characterEncoding=UTF-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.242 [0-0-0-writer] WARN CommonRdbmsWriter$Task - 回滚此次写入, 采用每次写入一行方式提交. 因为:Data truncation: Out of range value for column 'id' at row 1 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.247 [0-0-0-writer] ERROR StdoutPluginCollector - 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] com.mysql.jdbc.MysqlDataTruncation: Data truncation: Out of range value for column 'id' at row 1 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3976) ~[mysql-connector-java-5.1.47.jar:5.1.47] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914) ~[mysql-connector-java-5.1.47.jar:5.1.47] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530) ~[mysql-connector-java-5.1.47.jar:5.1.47] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683) ~[mysql-connector-java-5.1.47.jar:5.1.47] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495) ~[mysql-connector-java-5.1.47.jar:5.1.47] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903) ~[mysql-connector-java-5.1.47.jar:5.1.47] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1242) ~[mysql-connector-java-5.1.47.jar:5.1.47] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doOneInsert(CommonRdbmsWriter.java:382) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doBatchInsert(CommonRdbmsWriter.java:362) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWriteWithConnection(CommonRdbmsWriter.java:297) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWrite(CommonRdbmsWriter.java:319) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter$Task.startWrite(MysqlWriter.java:78) [mysqlwriter-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) [datax-core-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242] 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.259 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据: 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] {"exception":"Data truncation: Out of range value for column 'id' at row 1","record":[{"byteSize":22,"index":0,"rawData":"0591202111180387851990","type":5},{"byteSize":32,"index":1,"rawData":"4d0986595d40485aa9839e0cccec4044","type":5},{"byteSize":8,"index":2,"rawData":1747612856000,"type":7},{"byteSize":4,"index":3,"rawData":1004,"type":3},{"byteSize":16,"index":4,"rawData":"0591202111180387","type":5},{"byteSize":29,"index":5,"rawData":"05912021111803871658544258319","type":5},{"byteSize":2,"index":6,"rawData":"周云","type":5},{"byteSize":0,"index":7,"rawData":"","type":5},{"byteSize":1,"index":8,"rawData":1,"type":3},{"byteSize":28,"index":9,"rawData":"NYN0300010000020000S1F001436","type":5},{"byteSize":5,"index":10,"rawData":"西门M29","type":5},{"byteSize":32,"index":11,"rawData":"c68130c53b1441cc9e9205a0df155888","type":5},{"byteSize":83,"index":12,"rawData":"https://iot.yinxismart.top:8888/group1/M00/02/0C/rBQUOGgqdLiARP_wAADchi_tzQg504.jpg","type":5},{"byteSize":0,"index":13,"type":5},{"byteSize":0,"index":14,"type":5},{"byteSize":0,"index":15,"type":7},{"byteSize":0,"index":16,"type":5},{"byteSize":0,"index":17,"type":7},{"byteSize":1,"index":18,"rawData":1,"type":3},{"byteSize":1,"index":19,"rawData":1,"type":3},{"byteSize":1,"index":20,"rawData":1,"type":3},{"byteSize":1,"index":21,"rawData":0,"type":3},{"byteSize":0,"index":22,"type":5},{"byteSize":1,"index":23,"rawData":0,"type":3},{"byteSize":0,"index":24,"type":5},{"byteSize":0,"index":25,"type":5},{"byteSize":0,"index":26,"type":7},{"byteSize":1,"index":27,"rawData":0,"type":3}],"type":"writer"} 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.260 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据: 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] {"exception":"Data truncation: Out of range value for column 'id' at row 1","record":[{"byteSize":22,"index":0,"rawData":"0591202111180387851991","type":5},{"byteSize":32,"index":1,"rawData":"d52cf199b8b54901bad84a21f3fa25b1","type":5},{"byteSize":8,"index":2,"rawData":1747612880000,"type":7},{"byteSize":4,"index":3,"rawData":1004,"type":3},{"byteSize":16,"index":4,"rawData":"0591202111180387","type":5},{"byteSize":29,"index":5,"rawData":"05912021111803871742111902520","type":5},{"byteSize":3,"index":6,"rawData":"庄华琴","type":5},{"byteSize":0,"index":7,"rawData":"","type":5},{"byteSize":1,"index":8,"rawData":2,"type":3},{"byteSize":16,"index":9,"rawData":"0020220987YXJDSQ","type":5},{"byteSize":10,"index":10,"rawData":"中联天玺南出入口门禁","type":5},{"byteSize":32,"index":11,"rawData":"729da5882c6a41d9bfda0949931a1be1","type":5},{"byteSize":83,"index":12,"rawData":"https://iot.yinxismart.top:8888/group1/M00/02/0C/rBQUN2gqdNeASmFSAAJOMjYE8R8862.jpg","type":5},{"byteSize":0,"index":13,"type":5},{"byteSize":0,"index":14,"type":5},{"byteSize":0,"index":15,"type":7},{"byteSize":0,"index":16,"type":5},{"byteSize":0,"index":17,"type":7},{"byteSize":1,"index":18,"rawData":1,"type":3},{"byteSize":1,"index":19,"rawData":1,"type":3},{"byteSize":1,"index":20,"rawData":1,"type":3},{"byteSize":1,"index":21,"rawData":0,"type":3},{"byteSize":0,"index":22,"type":5},{"byteSize":1,"index":23,"rawData":0,"type":3},{"byteSize":0,"index":24,"type":5},{"byteSize":0,"index":25,"type":5},{"byteSize":0,"index":26,"type":7},{"byteSize":1,"index":27,"rawData":0,"type":3}],"type":"writer"} 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.262 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据: 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] {"exception":"Data truncation: Out of range value for column 'id' at row 1","record":[{"byteSize":22,"index":0,"rawData":"0591202111180387851992","type":5},{"byteSize":32,"index":1,"rawData":"4bd01f7d35f4437da3e276c6c31a5c5c","type":5},{"byteSize":8,"index":2,"rawData":1747612889000,"type":7},{"byteSize":4,"index":3,"rawData":1004,"type":3},{"byteSize":16,"index":4,"rawData":"0591202111180387","type":5},{"byteSize":29,"index":5,"rawData":"05912021111803871741681870967","type":5},{"byteSize":2,"index":6,"rawData":"严巧","type":5},{"byteSize":0,"index":7,"rawData":"","type":5},{"byteSize":1,"index":8,"rawData":1,"type":3},{"byteSize":28,"index":9,"rawData":"NYN0300010000020000S1F001436","type":5},{"byteSize":5,"index":10,"rawData":"西门M29","type":5},{"byteSize":32,"index":11,"rawData":"7f932dde89b64205a0d17d58871f9bad","type":5},{"byteSize":83,"index":12,"rawData":"https://iot.yinxismart.top:8888/group1/M00/02/0B/rBQUNmgqdNmAQ_S7AAC22NFh6HM442.jpg","type":5},{"byteSize":0,"index":13,"type":5},{"byteSize":0,"index":14,"type":5},{"byteSize":0,"index":15,"type":7},{"byteSize":0,"index":16,"type":5},{"byteSize":0,"index":17,"type":7},{"byteSize":1,"index":18,"rawData":1,"type":3},{"byteSize":1,"index":19,"rawData":1,"type":3},{"byteSize":1,"index":20,"rawData":1,"type":3},{"byteSize":1,"index":21,"rawData":0,"type":3},{"byteSize":0,"index":22,"type":5},{"byteSize":1,"index":23,"rawData":0,"type":3},{"byteSize":0,"index":24,"type":5},{"byteSize":0,"index":25,"type":5},{"byteSize":0,"index":26,"type":7},{"byteSize":1,"index":27,"rawData":0,"type":3}],"type":"writer"} 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.263 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据: 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] {"exception":"Data truncation: Out of range value for column 'id' at row 1","record":[{"byteSize":22,"index":0,"rawData":"0591202111180387851993","type":5},{"byteSize":32,"index":1,"rawData":"a766ed3cea29444181be6cedd1f816f5","type":5},{"byteSize":8,"index":2,"rawData":1747612931000,"type":7},{"byteSize":4,"index":3,"rawData":1004,"type":3},{"byteSize":16,"index":4,"rawData":"0591202111180387","type":5},{"byteSize":29,"index":5,"rawData":"05912021111803871659255470895","type":5},{"byteSize":3,"index":6,"rawData":"华明娟","type":5},{"byteSize":0,"index":7,"rawData":"","type":5},{"byteSize":1,"index":8,"rawData":1,"type":3},{"byteSize":28,"index":9,"rawData":"NYN0300010000020000S1F001436","type":5},{"byteSize":5,"index":10,"rawData":"西门M29","type":5},{"byteSize":32,"index":11,"rawData":"02fb5d620ac3460f8957de4b5b44336a","type":5},{"byteSize":83,"index":12,"rawData":"https://iot.yinxismart.top:8888/group1/M00/02/0D/rBQUNWgqdQOAeyFkAACw3Hvmovc046.jpg","type":5},{"byteSize":0,"index":13,"type":5},{"byteSize":0,"index":14,"type":5},{"byteSize":0,"index":15,"type":7},{"byteSize":0,"index":16,"type":5},{"byteSize":0,"index":17,"type":7},{"byteSize":1,"index":18,"rawData":1,"type":3},{"byteSize":1,"index":19,"rawData":1,"type":3},{"byteSize":1,"index":20,"rawData":1,"type":3},{"byteSize":1,"index":21,"rawData":0,"type":3},{"byteSize":0,"index":22,"type":5},{"byteSize":1,"index":23,"rawData":0,"type":3},{"byteSize":0,"index":24,"type":5},{"byteSize":0,"index":25,"type":5},{"byteSize":0,"index":26,"type":7},{"byteSize":1,"index":27,"rawData":0,"type":3}],"type":"writer"} 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.264 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据: 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] {"exception":"Data truncation: Out of range value for column 'id' at row 1","record":[{"byteSize":22,"index":0,"rawData":"0591202111180387851994","type":5},{"byteSize":32,"index":1,"rawData":"9a923175d25b41acbf8515f0d2b1d2ff","type":5},{"byteSize":8,"index":2,"rawData":1747613034000,"type":7},{"byteSize":4,"index":3,"rawData":1004,"type":3},{"byteSize":16,"index":4,"rawData":"0591202111180387","type":5},{"byteSize":29,"index":5,"rawData":"05912021111803871745223203681","type":5},{"byteSize":3,"index":6,"rawData":"林清唐","type":5},{"byteSize":0,"index":7,"rawData":"","type":5},{"byteSize":1,"index":8,"rawData":0,"type":3},{"byteSize":28,"index":9,"rawData":"NYN0300010000020000S1F001436","type":5},{"byteSize":5,"index":10,"rawData":"西门M29","type":5},{"byteSize":32,"index":11,"rawData":"d4d1b42ec6d540b9b459029dec96334a","type":5},{"byteSize":83,"index":12,"rawData":"https://iot.yinxismart.top:8888/group1/M00/02/0D/rBQUNWgqdWqAEVqIAAC_n9Jyueo433.jpg","type":5},{"byteSize":0,"index":13,"type":5},{"byteSize":0,"index":14,"type":5},{"byteSize":0,"index":15,"type":7},{"byteSize":0,"index":16,"type":5},{"byteSize":0,"index":17,"type":7},{"byteSize":1,"index":18,"rawData":1,"type":3},{"byteSize":1,"index":19,"rawData":1,"type":3},{"byteSize":1,"index":20,"rawData":1,"type":3},{"byteSize":1,"index":21,"rawData":0,"type":3},{"byteSize":0,"index":22,"type":5},{"byteSize":1,"index":23,"rawData":0,"type":3},{"byteSize":0,"index":24,"type":5},{"byteSize":0,"index":25,"type":5},{"byteSize":0,"index":26,"type":7},{"byteSize":1,"index":27,"rawData":0,"type":3}],"type":"writer"} 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.265 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据: 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] {"exception":"Data truncation: Out of range value for column 'id' at row 1","record":[{"byteSize":22,"index":0,"rawData":"0591202111180387851995","type":5},{"byteSize":32,"index":1,"rawData":"847f99174d4d4d26aaae669dd638e123","type":5},{"byteSize":8,"index":2,"rawData":1747613052000,"type":7},{"byteSize":4,"index":3,"rawData":1004,"type":3},{"byteSize":16,"index":4,"rawData":"0591202111180387","type":5},{"byteSize":29,"index":5,"rawData":"05912021111803871710205089306","type":5},{"byteSize":3,"index":6,"rawData":"陈惠钦","type":5},{"byteSize":0,"index":7,"rawData":"","type":5},{"byteSize":1,"index":8,"rawData":1,"type":3},{"byteSize":16,"index":9,"rawData":"0020220996YXJDSQ","type":5},{"byteSize":7,"index":10,"rawData":"北门消防门入口","type":5},{"byteSize":32,"index":11,"rawData":"2e6fdb84fdd44a94939c019a8e0d2b85","type":5},{"byteSize":83,"index":12,"rawData":"https://iot.yinxismart.top:8888/group1/M00/02/0C/rBQUOGgqdYWAEJkoAAHEBO1Px4w499.jpg","type":5},{"byteSize":0,"index":13,"type":5},{"byteSize":0,"index":14,"type":5},{"byteSize":0,"index":15,"type":7},{"byteSize":0,"index":16,"type":5},{"byteSize":0,"index":17,"type":7},{"byteSize":1,"index":18,"rawData":1,"type":3},{"byteSize":1,"index":19,"rawData":1,"type":3},{"byteSize":1,"index":20,"rawData":1,"type":3},{"byteSize":1,"index":21,"rawData":0,"type":3},{"byteSize":0,"index":22,"type":5},{"byteSize":1,"index":23,"rawData":0,"type":3},{"byteSize":0,"index":24,"type":5},{"byteSize":0,"index":25,"type":5},{"byteSize":0,"index":26,"type":7},{"byteSize":1,"index":27,"rawData":0,"type":3}],"type":"writer"} 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.266 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据: 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] {"exception":"Data truncation: Out of range value for column 'id' at row 1","record":[{"byteSize":22,"index":0,"rawData":"0591202111180387851996","type":5},{"byteSize":32,"index":1,"rawData":"6befb459b2da4ce1a3f9e7e9f3b9dcb4","type":5},{"byteSize":8,"index":2,"rawData":1747613064000,"type":7},{"byteSize":4,"index":3,"rawData":1004,"type":3},{"byteSize":16,"index":4,"rawData":"0591202111180387","type":5},{"byteSize":29,"index":5,"rawData":"05912021111803871741253388198","type":5},{"byteSize":3,"index":6,"rawData":"王枝发","type":5},{"byteSize":0,"index":7,"rawData":"","type":5},{"byteSize":1,"index":8,"rawData":1,"type":3},{"byteSize":28,"index":9,"rawData":"NYN0300010000020000S1F001436","type":5},{"byteSize":5,"index":10,"rawData":"西门M29","type":5},{"byteSize":32,"index":11,"rawData":"fe8e4331e4514d57bec126d6c767cc6c","type":5},{"byteSize":83,"index":12,"rawData":"https://iot.yinxismart.top:8888/group1/M00/02/0D/rBQUNWgqdYiAQR8YAAC_5wURlHI916.jpg","type":5},{"byteSize":0,"index":13,"type":5},{"byteSize":0,"index":14,"type":5},{"byteSize":0,"index":15,"type":7},{"byteSize":0,"index":16,"type":5},{"byteSize":0,"index":17,"type":7},{"byteSize":1,"index":18,"rawData":1,"type":3},{"byteSize":1,"index":19,"rawData":1,"type":3},{"byteSize":1,"index":20,"rawData":1,"type":3},{"byteSize":1,"index":21,"rawData":0,"type":3},{"byteSize":0,"index":22,"type":5},{"byteSize":1,"index":23,"rawData":0,"type":3},{"byteSize":0,"index":24,"type":5},{"byteSize":0,"index":25,"type":5},{"byteSize":0,"index":26,"type":7},{"byteSize":1,"index":27,"rawData":0,"type":3}],"type":"writer"} 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.268 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据: 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] {"exception":"Data truncation: Out of range value for column 'id' at row 1","record":[{"byteSize":22,"index":0,"rawData":"0591202111180387851997","type":5},{"byteSize":32,"index":1,"rawData":"bd4ff446847c483ea12dbc1a746ef9e0","type":5},{"byteSize":8,"index":2,"rawData":1747613074000,"type":7},{"byteSize":4,"index":3,"rawData":1004,"type":3},{"byteSize":16,"index":4,"rawData":"0591202111180387","type":5},{"byteSize":29,"index":5,"rawData":"05912021111803871666857417929","type":5},{"byteSize":3,"index":6,"rawData":"刘必梅","type":5},{"byteSize":0,"index":7,"rawData":"","type":5},{"byteSize":1,"index":8,"rawData":1,"type":3},{"byteSize":28,"index":9,"rawData":"NYN0300010000020000S1F001436","type":5},{"byteSize":5,"index":10,"rawData":"西门M29","type":5},{"byteSize":32,"index":11,"rawData":"09af342659144542a694fc75c8cd1c6f","type":5},{"byteSize":83,"index":12,"rawData":"https://iot.yinxismart.top:8888/group1/M00/02/0C/rBQUN2gqdZKAMtxUAADEYp0rtqk024.jpg","type":5},{"byteSize":0,"index":13,"type":5},{"byteSize":0,"index":14,"type":5},{"byteSize":0,"index":15,"type":7},{"byteSize":0,"index":16,"type":5},{"byteSize":0,"index":17,"type":7},{"byteSize":1,"index":18,"rawData":1,"type":3},{"byteSize":1,"index":19,"rawData":1,"type":3},{"byteSize":1,"index":20,"rawData":1,"type":3},{"byteSize":1,"index":21,"rawData":0,"type":3},{"byteSize":0,"index":22,"type":5},{"byteSize":1,"index":23,"rawData":0,"type":3},{"byteSize":0,"index":24,"type":5},{"byteSize":0,"index":25,"type":5},{"byteSize":0,"index":26,"type":7},{"byteSize":1,"index":27,"rawData":0,"type":3}],"type":"writer"} 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.269 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据: 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] {"exception":"Data truncation: Out of range value for column 'id' at row 1","record":[{"byteSize":22,"index":0,"rawData":"0591202111180387851998","type":5},{"byteSize":32,"index":1,"rawData":"f238007b8c634bb394c0425e9ffc2716","type":5},{"byteSize":8,"index":2,"rawData":1747613150000,"type":7},{"byteSize":4,"index":3,"rawData":1004,"type":3},{"byteSize":16,"index":4,"rawData":"0591202111180387","type":5},{"byteSize":29,"index":5,"rawData":"05912021111803871675924700007","type":5},{"byteSize":3,"index":6,"rawData":"俞玉琴","type":5},{"byteSize":0,"index":7,"rawData":"","type":5},{"byteSize":1,"index":8,"rawData":1,"type":3},{"byteSize":28,"index":9,"rawData":"NYN0300010000020000S1F001436","type":5},{"byteSize":5,"index":10,"rawData":"西门M29","type":5},{"byteSize":32,"index":11,"rawData":"198e3637945e4376a32310eae176b3ef","type":5},{"byteSize":83,"index":12,"rawData":"https://iot.yinxismart.top:8888/group1/M00/02/0C/rBQUN2gqdd6AAFFhAADRKbsR6uM999.jpg","type":5},{"byteSize":0,"index":13,"type":5},{"byteSize":0,"index":14,"type":5},{"byteSize":0,"index":15,"type":7},{"byteSize":0,"index":16,"type":5},{"byteSize":0,"index":17,"type":7},{"byteSize":1,"index":18,"rawData":1,"type":3},{"byteSize":1,"index":19,"rawData":1,"type":3},{"byteSize":1,"index":20,"rawData":1,"type":3},{"byteSize":1,"index":21,"rawData":0,"type":3},{"byteSize":0,"index":22,"type":5},{"byteSize":1,"index":23,"rawData":0,"type":3},{"byteSize":0,"index":24,"type":5},{"byteSize":0,"index":25,"type":5},{"byteSize":0,"index":26,"type":7},{"byteSize":1,"index":27,"rawData":0,"type":3}],"type":"writer"} 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.475 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[1629]ms 2025-05-19 10:33:57 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:33:57.476 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks. 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:34:05.848 [job-0] INFO StandAloneJobContainerCommunicator - Total 361 records, 98167 bytes | Speed 9.59KB/s, 36 records/s | Error 361 records, 98167 bytes | All Task WaitWriterTime 0.001s | All Task WaitReaderTime 1.066s | Percentage 100.00% 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:34:05.849 [job-0] ERROR JobContainer - 运行scheduler 模式[standalone]出错. 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:34:05.849 [job-0] ERROR JobContainer - Exception when job run 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] com.alibaba.datax.common.exception.DataXException: Code:[Framework-14], Description:[DataX传输脏数据超过用户预期,该错误通常是由于源端数据存在较多业务脏数据导致,请仔细检查DataX汇报的脏数据日志信息, 或者您可以适当调大脏数据阈值 .]. - 脏数据条数检查不通过,限制是[100]条,但实际上捕获了[361]条. 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30) ~[datax-common-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.util.ErrorRecordChecker.checkRecordLimit(ErrorRecordChecker.java:58) ~[datax-core-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:89) ~[datax-core-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535) ~[datax-core-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119) ~[datax-core-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.Engine.start(Engine.java:86) [datax-core-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.Engine.entry(Engine.java:168) [datax-core-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.Engine.main(Engine.java:201) [datax-core-0.0.1-SNAPSHOT.jar:na] 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:34:05.850 [job-0] INFO StandAloneJobContainerCommunicator - Total 361 records, 98167 bytes | Speed 95.87KB/s, 361 records/s | Error 361 records, 98167 bytes | All Task WaitWriterTime 0.001s | All Task WaitReaderTime 1.066s | Percentage 100.00% 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:34:05.850 [job-0] ERROR Engine - 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] 经DataX智能分析,该任务最可能的错误原因是: 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] com.alibaba.datax.common.exception.DataXException: Code:[Framework-14], Description:[DataX传输脏数据超过用户预期,该错误通常是由于源端数据存在较多业务脏数据导致,请仔细检查DataX汇报的脏数据日志信息, 或者您可以适当调大脏数据阈值 .]. - 脏数据条数检查不通过,限制是[100]条,但实际上捕获了[361]条. 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30) 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.util.ErrorRecordChecker.checkRecordLimit(ErrorRecordChecker.java:58) 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:89) 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535) 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119) 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.Engine.start(Engine.java:86) 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.Engine.entry(Engine.java:168) 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.Engine.main(Engine.java:201) 2025-05-19 10:34:05 [AnalysisStatistics.analysisStatisticsLog-53] 2025-05-19 10:34:05 [JobThread.run-165] <br>----------- datax-web job execute end(finish) -----------<br>----------- ReturnT:ReturnT [code=500, msg=command exit value(1) is failed, content=null] 2025-05-19 10:34:05 [TriggerCallbackThread.callbackLog-186] <br>----------- datax-web job callback finish. 如何清空脏数据?
05-20
任务描述 本关任务:编写一个清洗QueueStream数据的SparkStreaming程序。 相关知识 为了完成本关任务,你需要掌握:1.如何使用SparkStreaming,2.如何使用 SparkStreaming读取QueueStream。 SparkStreaming 的开发步骤 初始化SparkConf并设置相关参数 val conf = new SparkConf().setMaster(master).setAppName(appName) 说明: appName 是应用程序在集群 UI 上显示的名称。 master 是Spark,Mesos或YARN集群的URL,或在本地模式下运行使用 local[*] 初始化JavaStreamingContext并设置处理批次的时间 val ssc = new StreamingContext(conf, Seconds(1)) 设置数据源 例如: val inputStream = ssc.queueStream(rddQueue) 批次数据处理(使用相关算子完成相应的操作) 算子 含义 map(func) 通过将源DStream的每个元素传递给函数func来返回一个新的DStream flatMap(func) 与map类似,但每个输入项可以映射到0个或更多输出项。 filter(func) 通过仅选择func返回true的源DStream的记录来返回新的DStream repartition(numPartitions) 通过创建更多或更少的分区来更改此DStream中的并行度级别 union(otherStream) 返回一个新的DStream,它包含源DStream和otherDStream中元素的并集 count() 通过计算源DStream的每个RDD中的元素数量,返回单元素RDD的新DStream reduce(func) 通过使用函数func(它接受两个参数并返回一个)聚合源DStream的每个RDD中的元素,返回单元素RDD的新DStream 。该函数应该是关联的和可交换的,以便可以并行计算 countByValue() 当在类型为K的元素的DStream上调用时,返回(K,Long)对的新DStream,其中每个键的值是其在源DStream的每个RDD中的频率 reduceByKey(func,[ numTasks ]) 当在(K,V)对的DStream上调用时,返回(K,V)对的新DStream,其中使用给定的reduce函数聚合每个键的值。注意:默认情况下,这使用Spark的默认并行任务数(本地模式为2,在群集模式下,数量由config属性确定spark.default.parallelism进行分组。您可以传递可选numTasks参数来设置不同数量的任务 join(otherStream, [numTasks]) 当在(K,V)和(K,W)对的两个DStream上调用时,返回(K,(V,W))对的新DStream与每个键的所有元素对 cogroup(otherStream, [numTasks]) 当在(K,V)和(K,W)对的DStream上调用时,返回(K,Seq [V],Seq [W])元组的新DStream transform(func) 通过将RDD-to-RDD函数应用于源DStream的每个RDD来返回新的DStream。这可以用于在DStream上执行任意RDD操作 updateStateByKey(func) 返回一个新的“状态”DStream,其中通过在键的先前状态和键的新值上应用给定函数来更新每个键的状态 foreachRDD(func) 最通用的输出运算符,它将函数func应用于从流生成的每个RDD。此函数应将每个RDD中的数据推送到外部系统,例如将RDD保存到文件,或通过网络将其写入数据库。请注意,函数func在运行流应用程序的驱动程序进程中执行,并且通常会在其中执行RDD操作,这将强制计算流式RDD。 启动流计算 ssc.start(); 等待处理停止 ssc.awaitTermination(); QueueStream QueueStream(队列流):推入队列的每个RDD将被视为DStream中的一批数据,并像流一样处理。 编程要求 在右侧编辑器补充代码,完成以下需求: 将时间戳转换成规定格式的时间形式(格式为:yyyy-MM-dd HH:mm:ss ) 提取数据中的起始URL(切割符为空格) 拼接结果数据,格式如下: Ip:124.132.29.10,visitTime:2019-04-22 11:08:33,startUrl:www/2,targetUrl: https://search.yahoo.com/search?p=反叛的鲁鲁修,statusCode:200 将最终结果写入Mysql数据库 测试说明 平台将对你编写的代码进行评测: 预期输出: 1 Ip:100.143.124.29,visitTime:2017-10-27 14:58:05,startUrl:www/1,targetUrl:https://www.baidu.com/s?wd=反叛的鲁鲁修,statusCode:404 2 Ip:30.132.167.100,visitTime:2018-12-02 11:29:39,startUrl:www/4,targetUrl:-,statusCode:302 3 Ip:30.156.187.132,visitTime:2016-05-17 17:18:56,startUrl:www/2,targetUrl:-,statusCode:200 4 Ip:29.100.10.30,visitTime:2016-10-12 01:25:47,startUrl:www/3,targetUrl:http://cn.bing.com/search?q=游戏人生,statusCode:302 5 Ip:132.187.167.143,visitTime:2017-01-08 23:21:09,startUrl:pianhua/130,targetUrl:-,statusCode:200 6 Ip:143.187.100.10,visitTime:2016-09-21 19:27:39,startUrl:www/1,targetUrl:-,statusCode:302 7 Ip:10.100.124.30,visitTime:2018-09-16 02:49:35,startUrl:www/4,targetUrl:http://cn.bing.com/search?q=来自新世界,statusCode:200 8 Ip:29.10.143.187,visitTime:2017-09-29 15:49:09,startUrl:www/1,targetUrl:-,statusCode:404 9 Ip:29.187.132.100,visitTime:2018-11-27 05:43:17,startUrl:www/1,targetUrl:-,statusCode:200 10 Ip:187.167.124.132,visitTime:2016-01-28 13:34:33,startUrl:www/6,targetUrl:-,statusCode:200 开始你的任务吧,祝你成功! import java.text.SimpleDateFormat import java.util.Date import org.apache.spark.{HashPartitioner, SparkConf} import org.apache.spark.rdd.RDD import org.apache.spark.streaming.{Seconds, StreamingContext} import scala.collection.mutable object QueueStream { def main(args: Array[String]) { val rddQueue = new mutable.SynchronizedQueue[RDD[String]]() val conf = new SparkConf().setMaster("local[2]").setAppName("queueStream") /********** Begin **********/ //1.初始化StreamingContext,设置时间间隔为1s //2.对接队列流 /** * * 数据格式如下: * 100.143.124.29,1509116285000,'GET www/1 HTTP/1.0',https://www.baidu.com/s?wd=反叛的鲁鲁修,404 * 数据从左往右分别代表:用户IP、访问时间戳、起始URL及相关信息(访问方式,起始URL,http版本)、目标URL、状态码 * * * 原始数据的切割符为逗号,(英文逗号) * * 需求: * 1.将时间戳转换成规定时间(格式为:yyyy-MM-dd HH:mm:ss ) * 2.提取数据中的起始URL(切割符为空格) * 3.拼接结果数据,格式如下: * Ip:124.132.29.10,visitTime:2019-04-22 11:08:33,startUrl:www/2,targetUrl:https://search.yahoo.com/search?p=反叛的鲁鲁修,statusCode:200 * 4.将最终结果写入 mysql 数据库, 调用DBUtils.add(line)即可, line:String */ //3.获取队列流中的数据,进行清洗、转换(按照上面的需求) //4.将最终结果写入 mysql 数据库, 调用DBUtils.add(line)即可, line:String //5.启动SparkStreaming /********** End **********/ DBUtils.addQueue(ssc, rddQueue) } }
11-28
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值