python编写Mongodb同步实时数据到Tidb或mysql

                                                                                               Mongodb同步实时数据到Tidb

架构图

1.Go环境安装

yum install golang golang-bin golang-src  –y

 

2.mongo-shake安装

直接下载最新的mongo-shake安装包不用编译

开源下载地址

https://github.com/alibaba/MongoShake?spm=a2c4e.10696291.0.0.316619a4JXuDrN

二进制版本下载地址

https://github.com/alibaba/MongoShake/releases

 

下载mongo-shake-v2.4.16.tar.gz

cd /home/gocode/

tar –zxvf mongo-shake-v2.4.16.tar.gz

mv mongo-shake-v2.4.16 mongo-shake

cd mongo-shake

更改配置vim collector.conf

 

mongo_urls = mongodb://user:password@x.x.x.x:0000,x.x.x.x:0000

tunnel = kafka

tunnel.address = testGroup1@x.x.x.x:0000,x.x.x.x:0000,x.x.x.x:0000

tunnel.message = json

incr_sync.mongo_fetch_method = oplog

#黑白名单设置

filter.namespace.black =

#filter.namespace.white = test1215

 

启动

nohup ./start.sh collector.conf > nohup.log 2>&1 &

 

3. 从kafka获取数据存入Tidb

#Python脚本

#!/usr/bin/env python

# -*- coding:utf-8 -*-

from kafka import KafkaConsumer

import pymysql

import json

import time

import logging

import datetime

import os

import sys

#写日志

logging.basicConfig(level=logging.INFO,

                    format='%(asctime)s %(filename)s[line:%(lineno)d] %(levelname)s %(message)s',

                    datefmt='%Y/%m/%d %H:%M:%S',

                    filename='log_tracking.log',

                    filemode='a')

def writeErrorLog(errSrc, errType, errMsg):

    """

    ERROR日志

    :param errSrc:

    :param errType:

    :param errMsg:

    :