在把一个大表从 MySQL 迁移到 Redis 时,你可能会发现,每次提取、转换、导入一条数据是让人难以忍受的慢!这里有一个技巧,你可以通过使用管道把 MySQL 的输出直接输入到 redis-cli输入端,这可以使两个数据库都能以他们的最顶级速度来运行。
t_loan_risk_program (658.1461G),9999204295 条 MySQL 数据导入到 Redis 的时间从 12 小时缩短到了30分钟。
MySQL数据表结构:
CREATE TABLE `t_loan_risk_program` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`call_histories` longtext,
`contacts` longtext,
`create_time` datetime DEFAULT NULL,
`current_version` varchar(32) CHARACTER SET utf8 DEFAULT NULL,
`customer_id` bigint(20) DEFAULT NULL,
`device` varchar(10) CHARACTER SET utf8 DEFAULT NULL,
`device_info` longtext CHARACTER SET utf8,
`gps` text CHARACTER SET utf8,
`id_card_correct` bit(1) DEFAULT NULL,
`imei` varchar(64) CHARACTER SET utf8 DEFAULT NULL,
`in_black_list` bit(1) DEFAULT NULL,
`jailbreak_status` bit(1) NOT NULL,
`like_screenshot_score` double DEFAULT NULL,
`live_score` double DEFAULT NULL,
`loan_id` bigint(20) DEFAULT NULL,
`messages` longtext,
`packages` longtext,
`simulator_status` bit(1) DEFAULT NULL,
`zhima_black_list` bit(1) DEFAULT NULL,
`zhima_score` varchar(15) CHARACTER SET utf8 DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `cusIdAndloanId` (`customer_id`,`loan_id`) USING BTREE,
KEY `cusId` (`customer_id`) USING BTREE,
KEY `loanId` (`loan_id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=8289585 DEFAULT CHARSET=utf8mb4 ;
Redis存储结构:
HSET events_all_time [action] [count]
下面是重点,能过下面SQL语句将MySQL输出直接变更成redis-cli可接收的格式:
root@storm-master-01:/home#more /home/t_loan_rask_programredis.sql
SELECT CONCAT(
"*4\r\n",
'$', LENGTH(redis_cmd), '\r\n',
redis_cmd, '\r\n',
'$', LENGTH(redis_key), '\r\n',
redis_key, '\r\n',
'$', LENGTH(hkey), '\r\n',
hkey, '\r\n',
'$', LENGTH(hval), '\r\n',
hval, '\r'
)
FROM (
SELECT
'HSET' as redis_cmd,
't_loan_risk_program ' AS redis_key,
id AS hkey,
call_histories AS hval,
contacts as co,
create_time as crtim,
current_version as cuver,
customer_id as cuid,
device as dev,
device_info as devinfo,
gps as gps,
id_card_correct as idcc,
imei as im,
in_black_list as inbli,
jailbreak_status as ja,
like_screenshot_score as lik,
live_score as li,
loan_id as loid,
messages as mess,
packages as pak,
simulator_status as sim,
zhima_black_list as zh,
zhima_score as zhsc
FROM t_loan_risk_program
) AS ttt
ok, 用下面的命令执行:
mysql -h192.168.1.247 -uloan -ploan loan -P3309 --skip-column-names --raw < /home/t_loan_rask_programredis.sql | redis-cli -h 192.168.1.247 --pipe
很重要的mysql参数说明:
--raw: 使mysql不转换字段值中的换行符。
--skip-column-names: 使mysql输出的每行中不包含列名。