把数据库里的大表放到REDIS中(从Mysql将数据快速导入到redis中 )


在把一个大表从 MySQL 迁移到 Redis 时,你可能会发现,每次提取、转换、导入一条数据是让人难以忍受的慢!这里有一个技巧,你可以通过使用管道把 MySQL 的输出直接输入到 redis-cli输入端,这可以使两个数据库都能以他们的最顶级速度来运行。
t_loan_risk_program (658.1461G),9999204295 条 MySQL 数据导入到 Redis 的时间从 12 小时缩短到了30分钟。
 
MySQL数据表结构:
CREATE TABLE `t_loan_risk_program` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `call_histories` longtext,
  `contacts` longtext,
  `create_time` datetime DEFAULT NULL,
  `current_version` varchar(32) CHARACTER SET utf8 DEFAULT NULL,
  `customer_id` bigint(20) DEFAULT NULL,
  `device` varchar(10) CHARACTER SET utf8 DEFAULT NULL,
  `device_info` longtext CHARACTER SET utf8,
  `gps` text CHARACTER SET utf8,
  `id_card_correct` bit(1) DEFAULT NULL,
  `imei` varchar(64) CHARACTER SET utf8 DEFAULT NULL,
  `in_black_list` bit(1) DEFAULT NULL,
  `jailbreak_status` bit(1) NOT NULL,
  `like_screenshot_score` double DEFAULT NULL,
  `live_score` double DEFAULT NULL,
  `loan_id` bigint(20) DEFAULT NULL,
  `messages` longtext,
  `packages` longtext,
  `simulator_status` bit(1) DEFAULT NULL,
  `zhima_black_list` bit(1) DEFAULT NULL,
  `zhima_score` varchar(15) CHARACTER SET utf8 DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `cusIdAndloanId` (`customer_id`,`loan_id`) USING BTREE,
  KEY `cusId` (`customer_id`) USING BTREE,
  KEY `loanId` (`loan_id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=8289585 DEFAULT CHARSET=utf8mb4 ;

Redis存储结构:
    HSET events_all_time [action] [count]
下面是重点,能过下面SQL语句将MySQL输出直接变更成redis-cli可接收的格式:

root@storm-master-01:/home#more /home/t_loan_rask_programredis.sql
 SELECT CONCAT(
 "*4\r\n",
 '$', LENGTH(redis_cmd), '\r\n',
 redis_cmd, '\r\n',
 '$', LENGTH(redis_key), '\r\n',
 redis_key, '\r\n',
 '$', LENGTH(hkey), '\r\n',
 hkey, '\r\n',
 '$', LENGTH(hval), '\r\n',
 hval, '\r'
)
FROM (
 SELECT
 'HSET' as redis_cmd,
 't_loan_risk_program ' AS redis_key,
 id AS hkey,
 call_histories AS hval,
 contacts as co,
 create_time as crtim,
 current_version as cuver,
customer_id as cuid,
device as dev,
device_info as devinfo,
gps as gps,
id_card_correct as idcc,
imei as im,
in_black_list as inbli,
jailbreak_status as ja,
like_screenshot_score as lik,
live_score as li,
loan_id as loid,
messages as mess,
packages as pak,
simulator_status as sim,
zhima_black_list as zh,
zhima_score as zhsc
FROM t_loan_risk_program
) AS ttt

ok, 用下面的命令执行:
mysql -h192.168.1.247 -uloan -ploan  loan -P3309 --skip-column-names --raw  < /home/t_loan_rask_programredis.sql | redis-cli -h 192.168.1.247 --pipe
 很重要的mysql参数说明:
  --raw: 使mysql不转换字段值中的换行符。
  --skip-column-names: 使mysql输出的每行中不包含列名。

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值