ERROR 1406 (22001): Data Too Long, field len 30, data len 48

探讨在 TiDB 4.0 分布式数据库中从 information_schema 复制表结构时遇到的 DataTooLong 错误,并解决因字段长度限制导致的数据迁移问题。

 

今天遇到一个保持问题比较诡异:

执行sql语句insert into test.COLUMNS select * from information_schema.COLUMNS;报错:ERROR 1406 (22001): Data Too Long, field len 30, data len 48

 

问题的背景大概是这样子:

1、TiDB4.0分布式数据库系统,去information_schema.TABLES copy一份表结构,然后建一个一模一样的空表到一个临时库csdn上面。

2、在tidb的命令行里面执行 insert into test.COLUMNS select * from information_schema.COLUMNS;

3、尝试mysqldump从information_schema的TABLES表里面导出数据,导入到csdn库上的TABLES表里面去,还是报错。

4、检查了两个库information_schema和csdn库,表结构都是一样的。

 

看报错,应该是字段只有30的长度,但是录入的数据缺有48个字符串,那么就再仔细检查表结构

CREATE TABLE `TABLES` (
  `TABLE_CATALOG` varchar(512) DEFAULT NULL,
  `TABLE_SCHEMA` varchar(64) DEFAULT NULL,
  `TABLE_NAME` varchar(64) DEFAULT NULL,
  `COLUMN_NAME` varchar(64) DEFAULT NULL,
  `ORDINAL_POSITION` bigint(64) DEFAULT NULL,
  `COLUMN_DEFAULT` text DEFAULT NULL,
  `IS_NULLABLE` varchar(3) DEFAULT NULL,
  `DATA_TYPE` varchar(64) DEFAULT NULL,
  `CHARACTER_MAXIMUM_LENGTH` bigint(21) DEFAULT NULL,
  `CHARACTER_OCTET_LENGTH` bigint(21) DEFAULT NULL,
  `NUMERIC_PRECISION` bigint(21) DEFAULT NULL,
  `NUMERIC_SCALE` bigint(21) DEFAULT NULL,
  `DATETIME_PRECISION` bigint(21) DEFAULT NULL,
  `CHARACTER_SET_NAME` varchar(32) DEFAULT NULL,
  `COLLATION_NAME` varchar(32) DEFAULT NULL,
  `COLUMN_TYPE` text DEFAULT NULL,
  `COLUMN_KEY` varchar(3) DEFAULT NULL,
  `EXTRA` varchar(30) DEFAULT NULL,
  `PRIVILEGES` varchar(80) DEFAULT NULL,
  `COLUMN_COMMENT` varchar(1024) DEFAULT NULL,
  `GENERATION_EXPRESSION` text NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;

 

看到有一个字段EXTRA是`EXTRA` varchar(30) DEFAULT NULL,那么就去检查下这个字段里面的数据的值,是否有超过的。

mysql-17:31:41> select length(EXTRA) l,EXTRA from information_schema.TABLES order by l desc limit 50;
+----+--------------------------------------------------+
| l  | EXTRA                                            |
+----+--------------------------------------------------+
| 48 | DEFAULT_GENERATED on update CURRENT_TIMESTAMP(3) |
| 48 | DEFAULT_GENERATED on update CURRENT_TIMESTAMP(3) |
| 48 | DEFAULT_GENERATED on update CURRENT_TIMESTAMP(3) |
| 48 | DEFAULT_GENERATED on update CURRENT_TIMESTAMP(3) |
| 48 | DEFAULT_GENERATED on update CURRENT_TIMESTAMP(3) |
| 48 | DEFAULT_GENERATED on update CURRENT_TIMESTAMP(3) |
| 48 | DEFAULT_GENERATED on update CURRENT_TIMESTAMP(3) |
| 48 | DEFAULT_GENERATED on update CURRENT_TIMESTAMP(3) |

检查得到length(extra),确实最大长度是48,难道是字符集的原因吗,因为类似mysql的数据库里面,字符集不一样,占据空间是不一样的。

 

先看库的字符集,是一样的:

+----------+------------------------------------------------------------------+
| Database | Create Database                                                  |
+----------+------------------------------------------------------------------+
| csdn     | CREATE DATABASE `csdn` /*!40100 DEFAULT CHARACTER SET utf8mb4 */ |
+----------+------------------------------------------------------------------+

+--------------------+--------------------------------------------------------------------------------+
| Database           | Create Database                                                                |
+--------------------+--------------------------------------------------------------------------------+
| INFORMATION_SCHEMA | CREATE DATABASE `INFORMATION_SCHEMA` /*!40100 DEFAULT CHARACTER SET utf8mb4 */ |
+--------------------+--------------------------------------------------------------------------------+

 

我们再看下表的字符集定义,是否一样,也是一样的:

use information_schema;
CREATE TABLE `TABLES` (
  `TABLE_CATALOG` varchar(512) DEFAULT NULL,
  `TABLE_SCHEMA` varchar(64) DEFAULT NULL,
  `TABLE_NAME` varchar(64) DEFAULT NULL,
  `COLUMN_NAME` varchar(64) DEFAULT NULL,
  `ORDINAL_POSITION` bigint(64) DEFAULT NULL,
  `COLUMN_DEFAULT` text DEFAULT NULL,
  `IS_NULLABLE` varchar(3) DEFAULT NULL,
  `DATA_TYPE` varchar(64) DEFAULT NULL,
  `CHARACTER_MAXIMUM_LENGTH` bigint(21) DEFAULT NULL,
  `CHARACTER_OCTET_LENGTH` bigint(21) DEFAULT NULL,
  `NUMERIC_PRECISION` bigint(21) DEFAULT NULL,
  `NUMERIC_SCALE` bigint(21) DEFAULT NULL,
  `DATETIME_PRECISION` bigint(21) DEFAULT NULL,
  `CHARACTER_SET_NAME` varchar(32) DEFAULT NULL,
  `COLLATION_NAME` varchar(32) DEFAULT NULL,
  `COLUMN_TYPE` text DEFAULT NULL,
  `COLUMN_KEY` varchar(3) DEFAULT NULL,
  `EXTRA` varchar(30) DEFAULT NULL,
  `PRIVILEGES` varchar(80) DEFAULT NULL,
  `COLUMN_COMMENT` varchar(1024) DEFAULT NULL,
  `GENERATION_EXPRESSION` text NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;


use csdn;
CREATE TABLE `TABLES` (
  `TABLE_CATALOG` varchar(512) DEFAULT NULL,
  `TABLE_SCHEMA` varchar(64) DEFAULT NULL,
  `TABLE_NAME` varchar(64) DEFAULT NULL,
  `COLUMN_NAME` varchar(64) DEFAULT NULL,
  `ORDINAL_POSITION` bigint(64) DEFAULT NULL,
  `COLUMN_DEFAULT` text DEFAULT NULL,
  `IS_NULLABLE` varchar(3) DEFAULT NULL,
  `DATA_TYPE` varchar(64) DEFAULT NULL,
  `CHARACTER_MAXIMUM_LENGTH` bigint(21) DEFAULT NULL,
  `CHARACTER_OCTET_LENGTH` bigint(21) DEFAULT NULL,
  `NUMERIC_PRECISION` bigint(21) DEFAULT NULL,
  `NUMERIC_SCALE` bigint(21) DEFAULT NULL,
  `DATETIME_PRECISION` bigint(21) DEFAULT NULL,
  `CHARACTER_SET_NAME` varchar(32) DEFAULT NULL,
  `COLLATION_NAME` varchar(32) DEFAULT NULL,
  `COLUMN_TYPE` text DEFAULT NULL,
  `COLUMN_KEY` varchar(3) DEFAULT NULL,
  `EXTRA` varchar(30) DEFAULT NULL,
  `PRIVILEGES` varchar(80) DEFAULT NULL,
  `COLUMN_COMMENT` varchar(1024) DEFAULT NULL,
  `GENERATION_EXPRESSION` text NOT NULL,
  KEY `idx_pri` (`TABLE_SCHEMA`,`TABLE_NAME`,`COLUMN_KEY`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin

当然了解决办法很简单,就是在csdn库上的TABLES表里面修改字符长度:

alter table csdn.TABLES modify `EXTRA` varchar(128) DEFAULT NULL;

 

但是为什么information_schema库的TABLES表里面,为什么`EXTRA` varchar(30) DEFAULT NULL里面能存取48个字符呢,这问题值得后续研究下。

该代码什么情况会进去catch try { if (fileAllInfo_.erase.empty()) { throw std::invalid_argument("erase para empty."); } printf("func[%s][%d]:fileAllInfo_.erase.size()=%lu \n", __FUNCTION__, __LINE__, fileAllInfo_.erase.size()); _parseFilePath(filePath_, destFolder); std::ofstream outputFile(destFolder + ".pcf", std::ios::out | std::ios::trunc); if (!outputFile.is_open()) { outputFile.close(); throw std::runtime_error("outputFile failed to open."); } /* write pcf file header */ _writePcfField(outputFile, PCF_FLAG); ///< PCF标记 U32 pcf_header_len += PCF_FIELD_LEN; _writePcfField(outputFile, PCF_VERSION); ///< PCF版本 U32 pcf_header_len += PCF_FIELD_LEN; _writePcfField(outputFile, 0); ///< block数目 U32,初始为0读完重写 pcf_header_len += PCF_FIELD_LEN; _writePcfField(outputFile, 0); ///< alldata CRC U32,初始为0读完重写 pcf_header_len += PCF_FIELD_LEN; /* write pcf 自定义数据 */ std::uint32_t data_format = std::stoul(getFileAttribute().data_format_identifier, nullptr, 16); _writePcfField(outputFile, data_format); ///< 数据格式:是否压缩 U32 pcf_header_len += PCF_FIELD_LEN; printf("func[%s][%d]:data_format=0x%x \n", __FUNCTION__, __LINE__, data_format); //逻辑段信息 std::uint32_t erase_addr = 0; std::uint32_t erase_len = 0; _writePcfField(outputFile, fileAllInfo_.erase.size()); pcf_header_len += PCF_FIELD_LEN; for (auto& item : fileAllInfo_.erase) { erase_addr = std::stoul(item.start_address, nullptr, 16); erase_len = std::stoul(item.length, nullptr, 16); printf("func[%s][%d]:erase_addr=%#x, erase_len=%#x \n", __FUNCTION__, __LINE__, erase_addr, erase_len); _writePcfField(outputFile, erase_addr); ///< 逻辑段擦除地址 U32 pcf_header_len += PCF_FIELD_LEN; _writePcfField(outputFile, erase_len); ///< 逻辑段擦除长度 U32 pcf_header_len += PCF_FIELD_LEN; erase_addr = 0; erase_len = 0; } /* write pcf 自定义数据 填充数据 */ std::uint32_t paddingLen = PCF_HEADER_SIZE - pcf_header_len; std::uint8_t* pPaddingData = new std::uint8_t[paddingLen](); outputFile.write(reinterpret_cast<char*>(pPaddingData), paddingLen); delete[] pPaddingData; std::ifstream inputFile(filePath_, std::ios::in | std::ios::binary); if (!inputFile.is_open()) { inputFile.close(); outputFile.close(); throw std::runtime_error("inputFile failed to open."); } /* 获取cbf数据总长度 */ uint64_t total_data_size = fileStructureInfo_.at(FILE_DATA_LENGTH); printf("func[%s][%d]:total_data_size=0x%lx \n", __FUNCTION__, __LINE__, total_data_size); if (0 == total_data_size) { printf("func[%s][%d]:total_data_size=0x%lx \n", __FUNCTION__, __LINE__, total_data_size); inputFile.close(); outputFile.close(); throw std::runtime_error("error:total_data_size is 0"); } /* 定位到cbf数据位置 */ uint64_t positionToRead = fileStructureInfo_[FILE_DATA_POSITION]; printf("func[%s][%d]:positionToRead=0x%lx \n", __FUNCTION__, __LINE__, positionToRead); inputFile.seekg(positionToRead, std::ios::beg); std::uint8_t block_addr_buf[PCF_FIELD_LEN] = {0}; std::uint8_t block_len_buf[PCF_FIELD_LEN] = {0}; std::uint8_t block_crc_buf[PCF_FIELD_LEN] = {0}; std::uint32_t block_addr = 0; std::uint32_t block_len = 0; unsigned short block_crc = 0xFFFF; std::uint32_t block_num = 0; unsigned short all_data_crc = 0xFFFF; ///< crc_ccitt初始值 std::uint32_t read_bufsize = 0;//每次需要读取大小 std::uint32_t has_read_size = 0;//每个block已经读取大小 std::uint32_t pcf_offset = 0;//记录pcf文件偏移 std::uint32_t cbf_block_crc = 0; //cbf配置字节,vdi存4字节 while (true) { inputFile.read(reinterpret_cast<char*>(block_addr_buf), PCF_FIELD_LEN); if (inputFile.eof()) { break; } block_addr = _convert_arr_to_u32(block_addr_buf); printf("func[%s][%d]:block_addr=%#x \n", __FUNCTION__, __LINE__, block_addr); inputFile.read(reinterpret_cast<char*>(block_len_buf), PCF_FIELD_LEN); if (inputFile.eof()) { break; } block_len = _convert_arr_to_u32(block_len_buf); printf("func[%s][%d]:block_len=%#x \n", __FUNCTION__, __LINE__, block_len); _writePcfField(outputFile, block_addr); pcf_offset += PCF_FIELD_LEN; _writePcfField(outputFile, block_len); pcf_offset += PCF_FIELD_LEN; //如果block大小超过文件大小为异常 if (block_len > total_data_size) { bRet = false; break; } _writePcfField(outputFile, 0); //先默认block crc为0 pcf_offset += PCF_FIELD_LEN; //按设置大小循环读取block read_bufsize = block_len < HEAP_MEMORY_MAX_BUFF ? block_len : HEAP_MEMORY_MAX_BUFF; printf("func[%s][%d]:read_bufsize=%#x \n", __FUNCTION__, __LINE__, read_bufsize); readBuff = new char[read_bufsize](); while (has_read_size < block_len) { std::uint32_t onceSize = (block_len - has_read_size) < read_bufsize ? (block_len - has_read_size) : read_bufsize; inputFile.read(readBuff, onceSize); if (inputFile.eof()) { delete[] readBuff; break; } outputFile.write(readBuff, onceSize); pcf_offset += onceSize; has_read_size += onceSize; //block_crc = CalcCRC(block_crc, onceSize, (unsigned char*)readBuff); printf("func[%s][%d]:block_crc=%#x \n", __FUNCTION__, __LINE__, block_crc); //all_data_crc = CalcCRC(all_data_crc, onceSize, (unsigned char*)readBuff); printf("func[%s][%d]:all_data_crc=%#x \n", __FUNCTION__, __LINE__, all_data_crc); tools_time_delay(10); } inputFile.read(reinterpret_cast<char*>(block_crc_buf), CRC_16_LEN); if (inputFile.eof()) { delete[] readBuff; break; } cbf_block_crc = _convert_arr_to_u32(block_crc_buf); printf("func[%s][%d]:cfg cbf_block_crc=%#x \n", __FUNCTION__, __LINE__, cbf_block_crc); outputFile.seekp(PCF_HEADER_SIZE + pcf_offset - block_len - PCF_FIELD_LEN, std::ios::beg); _writePcfField(outputFile, block_crc); //写完后需要将文件指针移动到末尾,待下一个block输入 outputFile.seekp(0, std::ios::end); if (nullptr != readBuff) { delete[] readBuff; } memset(block_addr_buf, 0, sizeof(block_addr_buf)); block_addr = 0; memset(block_len_buf, 0, sizeof(block_len_buf)); block_len = 0; memset(block_crc_buf, 0, sizeof(block_crc_buf)); block_crc = 0xFFFF; has_read_size = 0; block_num++; } //最后写入block num printf("func[%s][%d]:block_num=%u \n", __FUNCTION__, __LINE__, block_num); outputFile.seekp(PCF_FIELD_LEN + PCF_FIELD_LEN, std::ios::beg); _writePcfField(outputFile, block_num); //最后写入all_data_crc printf("func[%s][%d]:all_data_crc=%#x \n", __FUNCTION__, __LINE__, all_data_crc); _writePcfField(outputFile, all_data_crc); inputFile.close(); outputFile.close(); } catch (std::exception& e) { std::cout << e.what() << std::endl; if (nullptr != readBuff) { delete[] readBuff; } return false; }
09-10
/* * Copyright (c) 2019-2024 Siddharth Chandrasekaran <sidcha.dev@gmail.com> * * SPDX-License-Identifier: Apache-2.0 */ #include "osdp_common.h" #include "osdp_diag.h" #define OSDP_PKT_MARK 0xFF #define OSDP_PKT_SOM 0x53 #define PKT_CONTROL_SQN 0x03 #define PKT_CONTROL_CRC 0x04 #define PKT_CONTROL_SCB 0x08 #define PKT_TRACE_MANGLED 0x80 PACK(struct osdp_packet_header { uint8_t som; uint8_t pd_address; uint8_t len_lsb; uint8_t len_msb; uint8_t control; uint8_t data[]; }); static void print_hex(unsigned char *buffer, int len) { int i; for(i = 0;i < len; i++) { if(buffer[i] <= 0x0F) printf("0x%x", buffer[i]); else printf("%x", buffer[i]); } printf("\n"); return; } static inline bool packet_has_mark(struct osdp_pd *pd) { return ISSET_FLAG(pd, PD_FLAG_PKT_HAS_MARK); } static int osdp_channel_send(struct osdp_pd *pd, uint8_t *buf, int len) { int sent, total_sent = 0; /* flush rx to remove any invalid data. */ if (pd->channel.flush) { pd->channel.flush(pd->channel.data); } do { /* send can block; so be greedy */ sent = pd->channel.send(pd->channel.data, buf + total_sent, len - total_sent); if (sent <= 0) { break; } total_sent += sent; } while (total_sent < len); if(ISSET_FLAG(pd, PD_FLAG_PRINT_MASSAGE)) { LOG_ERR("[SENDBUF]:"); print_hex(buf, total_sent); } return total_sent; } static int osdp_channel_receive(struct osdp_pd *pd) { uint8_t buf[64]; int recv, total_recv = 0; #ifdef UNIT_TESTING /** * Some unit tests don't define pd->channel.recv and directly fill * pd->packet_buf to test if everything else work correctly. */ if (!pd->channel.recv) { return 0; } #endif do { recv = pd->channel.recv(pd->channel.data, buf, sizeof(buf)); if (recv <= 0) { break; } if (osdp_rb_push_buf(&pd->rx_rb, buf, recv) != recv) { LOG_EM("RX ring buffer overflow!"); return -1; } total_recv += recv; } while (recv == sizeof(buf)); #if 0 if(pd->address == 0 && total_recv != 0) { LOG_ERR("packet buf:"); print_hex(buf, total_recv); if(pd->rx_rb.head == pd->rx_rb.tail) { LOG_ERR("push buffer again."); osdp_rb_push_buf(&pd->rx_rb, buf, total_recv); } } #endif if(ISSET_FLAG(pd, PD_FLAG_PRINT_MASSAGE) && total_recv != 0) { LOG_ERR("recvie packet buf:"); print_hex(buf, total_recv); } return total_recv; } uint8_t osdp_compute_checksum(uint8_t *msg, int length) { uint8_t checksum = 0; int i, whole_checksum; whole_checksum = 0; for (i = 0; i < length; i++) { whole_checksum += msg[i]; checksum = ~(0xff & whole_checksum) + 1; } return checksum; } static int osdp_phy_get_seq_number(struct osdp_pd *pd, int do_inc) { /* pd->seq_num is set to -1 to reset phy cmd state */ if (do_inc) { pd->seq_number += 1; if (pd->seq_number > 3) { pd->seq_number = 1; } } return pd->seq_number & PKT_CONTROL_SQN; } int osdp_phy_packet_get_data_offset(struct osdp_pd *pd, const uint8_t *buf) { int sb_len = 0, mark_byte_len = 0; struct osdp_packet_header *pkt; ARG_UNUSED(pd); if (packet_has_mark(pd)) { mark_byte_len = 1; buf += 1; } pkt = (struct osdp_packet_header *)buf; if (pkt->control & PKT_CONTROL_SCB) { sb_len = pkt->data[0]; } return mark_byte_len + sizeof(struct osdp_packet_header) + sb_len; } uint8_t *osdp_phy_packet_get_smb(struct osdp_pd *pd, const uint8_t *buf) { struct osdp_packet_header *pkt; pkt = (struct osdp_packet_header *)(buf + packet_has_mark(pd)); return (pkt->control & PKT_CONTROL_SCB) ? pkt->data : NULL; } int osdp_phy_in_sc_handshake(int is_reply, int id) { if (is_reply) { return (id == REPLY_CCRYPT || id == REPLY_RMAC_I); } else { return (id == CMD_CHLNG || id == CMD_SCRYPT); } } int osdp_phy_packet_init(struct osdp_pd *pd, uint8_t *buf, int max_len) { int id, scb_len = 0; struct osdp_packet_header *pkt; if (max_len < OSDP_MINIMUM_PACKET_SIZE) { LOG_ERR("packet_init: packet size too small"); return OSDP_ERR_PKT_FMT; } /** * In PD mode just follow what we received from CP. In CP mode, as we * initiate the transaction, choose based on CONFIG_OSDP_SKIP_MARK_BYTE. */ if ((is_pd_mode(pd) && packet_has_mark(pd)) || (is_cp_mode(pd) && !ISSET_FLAG(pd, PD_FLAG_PKT_SKIP_MARK))) { buf[0] = OSDP_PKT_MARK; buf++; SET_FLAG(pd, PD_FLAG_PKT_HAS_MARK); } /* Fill packet header */ pkt = (struct osdp_packet_header *)buf; pkt->som = OSDP_PKT_SOM; pkt->pd_address = pd->address & 0x7F; /* Use only the lower 7 bits */ if (ISSET_FLAG(pd, PD_FLAG_PKT_BROADCAST)) { pkt->pd_address = 0x7F; CLEAR_FLAG(pd, PD_FLAG_PKT_BROADCAST); } /* PD must reply with MSB of it's address set */ if (is_pd_mode(pd)) { pkt->pd_address |= 0x80; id = pd->reply_id; } else { id = pd->cmd_id; } pkt->control = osdp_phy_get_seq_number(pd, is_cp_mode(pd)); pkt->control |= PKT_CONTROL_CRC; if (sc_is_active(pd)) { pkt->control |= PKT_CONTROL_SCB; pkt->data[0] = scb_len = 2; pkt->data[1] = SCS_15; } else if (osdp_phy_in_sc_handshake(is_pd_mode(pd), id)) { pkt->control |= PKT_CONTROL_SCB; pkt->data[0] = scb_len = 3; pkt->data[1] = SCS_11; } return (packet_has_mark(pd) + sizeof(struct osdp_packet_header) + scb_len); } static int osdp_phy_packet_finalize(struct osdp_pd *pd, uint8_t *buf, int len, int max_len) { uint16_t crc16; struct osdp_packet_header *pkt; uint8_t *data; int data_len; /* Do a sanity check only; we expect header to be pre-filled */ if ((unsigned long)len <= sizeof(struct osdp_packet_header)) { LOG_ERR("PKT_F: Invalid header"); return OSDP_ERR_PKT_FMT; } if (packet_has_mark(pd)) { if (buf[0] != OSDP_PKT_MARK) { LOG_ERR("PKT_F: MARK validation failed! ID: 0x%02x", is_cp_mode(pd) ? pd->cmd_id : pd->reply_id); return OSDP_ERR_PKT_FMT; } /* temporarily get rid of mark byte */ buf += 1; len -= 1; max_len -= 1; } pkt = (struct osdp_packet_header *)buf; if (pkt->som != OSDP_PKT_SOM) { LOG_ERR("PKT_F: header SOM validation failed! ID: 0x%02x", is_cp_mode(pd) ? pd->cmd_id : pd->reply_id); return OSDP_ERR_PKT_FMT; } /* len: with 2 byte CRC */ pkt->len_lsb = BYTE_0(len + 2); pkt->len_msb = BYTE_1(len + 2); if (is_data_trace_enabled(pd)) { uint8_t control; /** * We can potentially avoid having to set PKT_TRACE_MANGLED * here if we can get the dissector accept fully formed SCB * but non-encrypted data block. But that might lead to the * dissector parsing malformed packets as valid ones. * * See the counterpart of this in osdp_phy_decode_packet() for * more details. */ control = pkt->control; pkt->control |= PKT_TRACE_MANGLED; osdp_capture_packet(pd, (uint8_t *)pkt, len + 2); pkt->control = control; } if (sc_is_active(pd) && pkt->control & PKT_CONTROL_SCB && pkt->data[1] >= SCS_15) { if (pkt->data[1] == SCS_17 || pkt->data[1] == SCS_18) { /** * Only the data portion of message (after id byte) * is encrypted. While (en)decrypting, we must skip * header, security block, and cmd/reply ID byte. * * Note: if cmd/reply has no data, we must set type to * SCS_15/SCS_16 and send them. */ data = pkt->data + pkt->data[0] + 1; data_len = len - (sizeof(struct osdp_packet_header) + pkt->data[0] + 1); len -= data_len; /** * check if the passed buffer can hold the encrypted * data where length may be rounded up to the nearest * 16 byte block boundary. */ if (AES_PAD_LEN(data_len + 1) > max_len) { /* data_len + 1 for OSDP_SC_EOM_MARKER */ goto out_of_space_error; } len += osdp_encrypt_data(pd, is_cp_mode(pd), data, data_len); } /* len: with 4bytes MAC; with 2 byte CRC; without 1 byte mark */ if (len + 4 > max_len) { goto out_of_space_error; } /* len: with 2 byte CRC; with 4 byte MAC */ pkt->len_lsb = BYTE_0(len + 2 + 4); pkt->len_msb = BYTE_1(len + 2 + 4); /* compute and extend the buf with 4 MAC bytes */ osdp_compute_mac(pd, is_cp_mode(pd), buf, len); data = is_cp_mode(pd) ? pd->sc.c_mac : pd->sc.r_mac; memcpy(buf + len, data, 4); len += 4; } /* fill crc16 */ if (len + 2 > max_len) { goto out_of_space_error; } crc16 = osdp_compute_crc16(buf, len); buf[len + 0] = BYTE_0(crc16); buf[len + 1] = BYTE_1(crc16); len += 2; return len + packet_has_mark(pd); out_of_space_error: LOG_ERR("PKT_F: Out of buffer space! CMD(%02x)", pd->cmd_id); return OSDP_ERR_PKT_FMT; } int osdp_phy_send_packet(struct osdp_pd *pd, uint8_t *buf, int len, int max_len) { int ret; /* if it has been more than TOUT_MS, then CP will not process. Do not send */ if(is_pd_mode(pd) && (osdp_millis_since(pd->tstamp) > OSDP_RESP_TOUT_MS)) { return OSDP_ERR_PKT_BUSY; } /* finalize packet */ len = osdp_phy_packet_finalize(pd, buf, len, max_len); if (len < 0) { return OSDP_ERR_PKT_BUILD; } if (is_packet_trace_enabled(pd)) { osdp_capture_packet(pd, buf, len); } ret = osdp_channel_send(pd, buf, len); if (ret != len) { LOG_ERR("Channel send for %d bytes failed! ret: %d", len, ret); return OSDP_ERR_PKT_BUILD; } return OSDP_ERR_PKT_NONE; } static bool phy_rescan_packet_buf(struct osdp_pd *pd) { unsigned long j = packet_has_mark(pd); unsigned long i = j + 1; /* +1 to skip current SoM */ while (i < pd->packet_buf_len && pd->packet_buf[i] != OSDP_PKT_SOM) { i++; } if (i < pd->packet_buf_len) { /* found another SoM; move the rest of the bytes down */ if (i && pd->packet_buf[i - 1] == OSDP_PKT_MARK) { pd->packet_buf[0] = OSDP_PKT_MARK; j = 1; SET_FLAG(pd, PD_FLAG_PKT_HAS_MARK); } else { j = 0; CLEAR_FLAG(pd, PD_FLAG_PKT_HAS_MARK); } while (i < pd->packet_buf_len) { pd->packet_buf[j++] = pd->packet_buf[i++]; } pd->packet_buf_len = j; return true; } /* nothing found, discarded all */ pd->packet_buf_len = 0; return false; } static int phy_check_header(struct osdp_pd *pd) { unsigned long pkt_len; int len; struct osdp_packet_header *pkt; uint8_t cur_byte = 0, prev_byte = 0; uint8_t *buf = pd->packet_buf; /* Scan for packet start */ while (pd->packet_buf_len == 0) { if (osdp_rb_pop(&pd->rx_rb, &cur_byte)) { if(ISSET_FLAG(pd, PD_FLAG_PRINT_MASSAGE)) LOG_ERR("NO data now."); return OSDP_ERR_PKT_NO_DATA; } if (cur_byte == OSDP_PKT_SOM) { if (prev_byte == OSDP_PKT_MARK) { buf[0] = OSDP_PKT_MARK; buf[1] = OSDP_PKT_SOM; pd->packet_buf_len = 2; SET_FLAG(pd, PD_FLAG_PKT_HAS_MARK); } else { buf[0] = OSDP_PKT_SOM; pd->packet_buf_len = 1; CLEAR_FLAG(pd, PD_FLAG_PKT_HAS_MARK); } break; } if (cur_byte != OSDP_PKT_MARK) { pd->packet_scan_skip++; } prev_byte = cur_byte; } /* Found start of a new packet; wait until we have atleast the header */ len = osdp_rb_pop_buf(&pd->rx_rb, buf + pd->packet_buf_len, sizeof(struct osdp_packet_header) - 1); pd->packet_buf_len += len; if (pd->packet_buf_len < sizeof(struct osdp_packet_header)) { LOG_ERR("wrong packet data now."); return OSDP_ERR_PKT_WAIT; } pkt = (struct osdp_packet_header *)(buf + packet_has_mark(pd)); /* validate packet header */ if (pkt->som != OSDP_PKT_SOM) { LOG_ERR("Invalid SOM 0x%02x", pkt->som); return OSDP_ERR_PKT_FMT; } /* validate packet */ pkt_len = (pkt->len_msb << 8) | pkt->len_lsb; if (pkt_len > OSDP_PACKET_BUF_SIZE || pkt_len < sizeof(struct osdp_packet_header) + 1 || (is_cp_mode(pd) && !(pkt->pd_address & 0x80))) { /* * Since SoM byte was encountered and the packet structure is * invalid, we cannot just discard all bytes extracted so far * as there may another valid SoM in the subsequent stream. So * we need to re-scan rest of the extracted bytes for another * SoM before we can discard the extracted bytes. */ if(ISSET_FLAG(pd, PD_FLAG_PRINT_MASSAGE)) { LOG_ERR("packet_buf_len:%d, packet_Buf:", pd->packet_buf_len); print_hex(pd->packet_buf, pd->packet_buf_len); } if (phy_rescan_packet_buf(pd)) { LOG_DBG("Found nested SoM in re-scan; re-parsing"); } return OSDP_ERR_PKT_WAIT; } return pkt_len + packet_has_mark(pd); } static int phy_check_packet(struct osdp_pd *pd, uint8_t *buf, int pkt_len) { int pd_addr; uint16_t comp, cur; struct osdp_packet_header *pkt; if (packet_has_mark(pd)) { buf += 1; pkt_len -= 1; } pkt = (struct osdp_packet_header *)buf; /* validate CRC/checksum */ if (pkt->control & PKT_CONTROL_CRC) { pkt_len -= 2; /* consume CRC */ cur = (buf[pkt_len + 1] << 8) | buf[pkt_len]; comp = osdp_compute_crc16(buf, pkt_len); if (comp != cur) { LOG_ERR("Invalid crc 0x%04x/0x%04x", comp, cur); return OSDP_ERR_PKT_FMT; } } else { pkt_len -= 1; /* consume checksum */ cur = buf[pkt_len]; comp = osdp_compute_checksum(buf, pkt_len); if (comp != cur) { LOG_ERR("Invalid checksum %02x/%02x", comp, cur); return OSDP_ERR_PKT_FMT; } } /* validate PD address */ if(is_pd_mode(pd) && pkt->pd_address & 0x80) { if(ISSET_FLAG(pd, PD_FLAG_PRINT_MASSAGE)) { LOG_ERR("anther pd reply message, ignore it."); } return OSDP_ERR_PKT_SKIP; } pd_addr = pkt->pd_address & 0x7F; if (pd_addr != pd->address && pd_addr != 0x7F) { /* not addressed to us and was not broadcasted */ if (is_cp_mode(pd)) { LOG_ERR("Invalid pd address %d", pd_addr); return OSDP_ERR_PKT_CHECK; } return OSDP_ERR_PKT_SKIP; } /* validate sequence number */ comp = pkt->control & PKT_CONTROL_SQN; if (is_pd_mode(pd)) { if (comp == 0) { /** * CP is trying to restart communication by sending a 0. * The current PD implementation does not hold any state * between commands so we can just set seq_number to -1 * (so it gets incremented to 0 with a call to * phy_get_seq_number()) and invalidate any established * secure channels. */ pd->seq_number = -1; sc_deactivate(pd); } else if (comp == pd->seq_number) { /** * Sometimes, a CP re-sends the same command without * incrementing the sequence number. To handle such cases, * we will move the sequence back one step (with do_inc * set to -1) and then process the packet all over again * as if it was the first time we are seeing it. */ pd->seq_number -= 1; LOG_INF("Received a sequence repeat packet!"); } /** * For packets addressed to the broadcast address, the reply * must have address set to 0x7F and not the current PD's * address. To handle this case, let's capture this state in PD * flags. */ if (pd_addr == 0x7F) { SET_FLAG(pd, PD_FLAG_PKT_BROADCAST); } } else { if (comp == 0) { /** * Check for receiving a busy reply from the PD which would * have a sequence number of 0, come in an unsecured packet * of minimum length, and have the reply ID REPLY_BUSY. */ if ((pkt_len == 6) && (pkt->data[0] == REPLY_BUSY)) { pd->seq_number -= 1; LOG_INF("packet busy now."); return OSDP_ERR_PKT_BUSY; } } } cur = osdp_phy_get_seq_number(pd, is_pd_mode(pd)); if (cur != comp && !ISSET_FLAG(pd, PD_FLAG_SKIP_SEQ_CHECK)) { LOG_ERR("Packet sequence mismatch (%d/%d)", cur, comp); pd->reply_id = REPLY_NAK; pd->ephemeral_data[0] = OSDP_PD_NAK_SEQ_NUM; return OSDP_ERR_PKT_NACK; } return OSDP_ERR_PKT_NONE; } int osdp_phy_check_packet(struct osdp_pd *pd) { int ret = OSDP_ERR_PKT_FMT; ret = osdp_channel_receive(pd); /* always pull new bytes first */ /** * PD mode does not maintain state. When we receive the anything * from CP, we need to capture the timestamp so we can timeout and * clear the buffer on errors and stray RX data. */ if (is_pd_mode(pd) && pd->packet_buf_len == 0 && ret > 0) { pd->tstamp = osdp_millis_now(); if(ISSET_FLAG(pd, PD_FLAG_PRINT_MASSAGE)) { LOG_ERR("time stamp now:%lld", pd->tstamp); } } if (pd->packet_len == 0) { ret = phy_check_header(pd); if (ret < 0) { return ret; } pd->packet_len = ret; if (pd->packet_scan_skip) { LOG_DBG("Packet scan skipped:%u mark:%d", pd->packet_scan_skip, ISSET_FLAG(pd, PD_FLAG_PKT_HAS_MARK)); pd->packet_scan_skip = 0; } } /* We have a valid header, collect one full packet */ ret = osdp_rb_pop_buf(&pd->rx_rb, pd->packet_buf + pd->packet_buf_len, pd->packet_len - pd->packet_buf_len); pd->packet_buf_len += ret; if (pd->packet_buf_len != pd->packet_len) { return OSDP_ERR_PKT_WAIT; } if (is_packet_trace_enabled(pd)) { osdp_capture_packet(pd, pd->packet_buf, pd->packet_buf_len); } return phy_check_packet(pd, pd->packet_buf, pd->packet_len); } int osdp_phy_decode_packet(struct osdp_pd *pd, uint8_t **pkt_start) { uint8_t *data, *mac, *buf = pd->packet_buf; int mac_offset, is_cmd, len = pd->packet_buf_len; struct osdp_packet_header *pkt; bool is_sc_active = sc_is_active(pd); if (packet_has_mark(pd)) { buf += 1; len -= 1; } pkt = (struct osdp_packet_header *)buf; len -= pkt->control & PKT_CONTROL_CRC ? 2 : 1; mac_offset = len - 4; data = pkt->data; len -= sizeof(struct osdp_packet_header); if (pkt->control & PKT_CONTROL_SCB) { if (is_pd_mode(pd) && !sc_is_capable(pd)) { LOG_ERR("PD is not SC capable"); pd->reply_id = REPLY_NAK; pd->ephemeral_data[0] = OSDP_PD_NAK_SC_UNSUP; return OSDP_ERR_PKT_NACK; } if (pkt->data[1] < SCS_11 || pkt->data[1] > SCS_18) { LOG_ERR("Invalid SB Type"); pd->reply_id = REPLY_NAK; pd->ephemeral_data[0] = OSDP_PD_NAK_SC_COND; return OSDP_ERR_PKT_NACK; } if (!is_sc_active && pkt->data[1] > SCS_14) { LOG_ERR("Invalid SCS type (%x)", pkt->data[1]); pd->reply_id = REPLY_NAK; pd->ephemeral_data[0] = OSDP_PD_NAK_SC_COND; return OSDP_ERR_PKT_NACK; } if (pkt->data[1] == SCS_11 || pkt->data[1] == SCS_13) { /** * CP signals PD to use SCBKD by setting SCB data byte * to 0. But since SCBK-D is insecure access, it's * usage is limited to install mode (a provisioning time * mode) only. */ if (ISSET_FLAG(pd, OSDP_FLAG_INSTALL_MODE) && pkt->data[2] == 0) { SET_FLAG(pd, PD_FLAG_SC_USE_SCBKD); } } data = pkt->data + pkt->data[0]; len -= pkt->data[0]; /* consume security block */ } else { if (is_cp_mode(pd)) { /** * If the current packet is an ACK for a KEYSET, the PD * might have discarded the secure channel session keys * in favour of the new key we sent and hence this packet * may reach us in plain text. To work with such PDs, we * must also discard our secure session. * * For now, let's just pretend that SC is deactivated so * the rest of this method finishes normally. The actual * secure channel is actually discarded from the CP * state machine. */ if (pd->cmd_id == CMD_KEYSET && pkt->data[0] == REPLY_ACK) { is_sc_active = false; } /** * When the PD discards it's secure channel for some * reason, it responds with NACK(6) in plaintext. There * may be other cases too. So we will allow NAKs in */ if (is_sc_active && pkt->data[0] == REPLY_NAK) { is_sc_active = false; } } if (is_sc_active) { LOG_ERR("Received plain-text message in SC"); pd->reply_id = REPLY_NAK; pd->ephemeral_data[0] = OSDP_PD_NAK_SC_COND; return OSDP_ERR_PKT_NACK; } } if (is_sc_active && pkt->control & PKT_CONTROL_SCB && pkt->data[1] >= SCS_15) { /* validate MAC */ is_cmd = is_pd_mode(pd); osdp_compute_mac(pd, is_cmd, buf, mac_offset); mac = is_cmd ? pd->sc.c_mac : pd->sc.r_mac; if (memcmp(buf + mac_offset, mac, 4) != 0) { LOG_ERR("Invalid MAC; discarding SC"); sc_deactivate(pd); pd->reply_id = REPLY_NAK; pd->ephemeral_data[0] = OSDP_PD_NAK_SC_COND; return OSDP_ERR_PKT_NACK; } len -= 4; /* consume MAC */ /* decrypt data block */ if (pkt->data[1] == SCS_17 || pkt->data[1] == SCS_18) { /** * Only the data portion of message (after id byte) * is encrypted. While (en)decrypting, we must skip * header (6), security block (2) and cmd/reply id (1) * bytes if cmd/reply has no data, use SCS_15/SCS_16. * * At this point, the header and security block is * already consumed. So we can just skip the cmd/reply * ID (data[0]) when calling osdp_decrypt_data(). */ len = osdp_decrypt_data(pd, is_cmd, data + 1, len - 1); if (len < 0) { LOG_ERR("Failed at decrypt; discarding SC"); sc_deactivate(pd); pd->reply_id = REPLY_NAK; pd->ephemeral_data[0] = OSDP_PD_NAK_SC_COND; return OSDP_ERR_PKT_NACK; } if (len == 0) { /** * If cmd/reply has no data, PD "should" have * used SCS_15/SCS_16 but we will be tolerant * towards those faulty implementations. */ LOG_WRN_ONCE("Received encrypted data block with 0 " "length; tolerating non-conformance!"); } len += 1; /* put back cmd/reply ID */ } } if (is_data_trace_enabled(pd)) { int ret = len; /** * Here, we move the decrypted data block right after the * header so we can pass the entire packet to the tracing * infrastructure allowing us to reuse the same protocol * dissector for regular packet trace as well as data trace * files. * * We also touch up the packet so that it can be parsed/decoded * correctly. Since none of the later states require the * header, we are free to mangle it as needed for our * convenience. * * The following changes performed: * - Update the length field with the modifications * - Remove `PKT_CONTROL_SCB` bit to show no sign of a secure * channels' existence. * - Set a new bit `PKT_TRACE_MANGLED` so the dissector can * skip the PacketCheck bytes (maybe other housekeeping?). */ memmove(pkt->data, data, len); *pkt_start = pkt->data; len += sizeof(struct osdp_packet_header); pkt->control &= ~PKT_CONTROL_SCB; pkt->control |= PKT_TRACE_MANGLED; pkt->len_lsb = BYTE_0(len); pkt->len_msb = BYTE_1(len); osdp_capture_packet(pd, (uint8_t *)pkt, len); return ret; } *pkt_start = data; return len; } void osdp_phy_state_reset(struct osdp_pd *pd, bool is_error) { pd->packet_buf_len = 0; pd->packet_len = 0; pd->phy_state = 0; if (is_error) { pd->phy_retry_count = 0; pd->seq_number = -1; if (pd->channel.flush) { pd->channel.flush(pd->channel.data); } } } #ifdef UNIT_TESTING int (*test_osdp_phy_packet_finalize)(struct osdp_pd *pd, uint8_t *buf, int len, int max_len) = osdp_phy_packet_finalize; #endif
09-27
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值