字段更改造成ogg进程停止解决办法

收到开发人员的变更邮件申请，需要在某个表增加字段，由于我们的环境DDL是没有开启的。用的消息中间件Kafka做传输消费，应用到目标库。由于紧急变更，先做字段变更吧。

变更完成后，查看ogg状态；
[oracle@bapdb1 ggs12]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.1.2.1.0 OGGCORE_12.1.2.1.0_PLATFORMS_140727.2135.1_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Aug 7 2014 09:14:25
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2014, Oracle and/or its affiliates. All rights reserved.

GGSCI (bapdb1) 2> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING
JAGENT RUNNING
EXTRACT RUNNING EXTDMQ 00:00:03 00:00:07
EXTRACT RUNNING EXTNEWP 00:00:04 00:00:09
EXTRACT RUNNING EXT_KAF 00:00:03 00:00:00
EXTRACT RUNNING EXT_KAF2 00:00:03 00:00:03
EXTRACT RUNNING EXT_KAF4 00:00:03 00:00:03
EXTRACT RUNNING EXT_KAF5 00:00:01 00:00:04
EXTRACT RUNNING EXT_KAF6 00:00:02 00:00:01
EXTRACT RUNNING EXT_KAF7 00:00:03 00:00:08
EXTRACT RUNNING EXT_KAF8 00:00:03 00:00:06
EXTRACT RUNNING PMP_KAF 00:00:00 00:00:04
EXTRACT RUNNING PMP_KAF2 00:00:00 00:00:08
EXTRACT RUNNING PMP_KAF4 00:00:00 00:00:10
EXTRACT RUNNING PMP_KAF5 00:00:03 00:00:06
EXTRACT RUNNING PMP_KAF6 00:00:04 00:00:02
EXTRACT RUNNING PMP_KAF7 00:00:03 00:00:03
EXTRACT RUNNING PMP_KAF8 00:00:04 00:00:05
EXTRACT RUNNING PUDMQ 00:00:00 00:00:04
EXTRACT RUNNING PUMPNEWP 00:00:00 00:00:01
REPLICAT RUNNING REPZW1 00:00:00 00:00:01

手工停止该表用到的进程，抽取和投递均停止后，等待邮件报错，找到对应的目标端机器；

找到kafka地址为10.1.x.9的地址，登陆机器

[systemusers@PBSDATASKAFKAGROUP2-1 ~]$ sudo su - app
[app@PBSDATASKAFKAGROUP2-1 ~]$ ls
data datalogs glt jdk1.8.0_101 jdk1.8.0_25 kafka_2.12-1.0.0 logs nohup.out ogg scripts shell tools zookeeper-3.4.11
[app@PBSDATASKAFKAGROUP2-1 ~]$ ps -ef |grep mgr
root 22 2 0 Mar19 ? 00:00:00 [async/mgr]
app 11451 1 0 Mar22 ? 00:43:43 ./mgr PARAMFILE /home/app/ogg/ggs12/dirprm/mgr.prm REPORTFILE /home/app/ogg/ggs12/dirrpt/MGR.rpt PROCESSID MGR
app 71909 71834 0 17:36 pts/0 00:00:00 grep mgr
[app@PBSDATASKAFKAGROUP2-1 ~]$ pwd
/home/app
[app@PBSDATASKAFKAGROUP2-1 ~]$ cd ogg/ggs12/
[app@PBSDATASKAFKAGROUP2-1 ggs12]$ ./ggsci
Oracle GoldenGate for Big Data
Version 12.3.1.1.1

Oracle GoldenGate Command Interpreter
Version 12.3.0.1.0 OGGCORE_OGGADP.12.3.0.1.0GA_PLATFORMS_170828.1608
Linux, x64, 64bit (optimized), Generic on Aug 28 2017 17:13:45
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.

GGSCI (PBSDATASKAFKAGROUP2-1) 1> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING
REPLICAT RUNNING REP_75 00:00:00 00:00:06
REPLICAT RUNNING REP_75_2 00:00:03 00:00:04
REPLICAT ABENDING REP_75_3 00:00:03 00:00:02
REPLICAT RUNNING REP_90 00:00:00 00:00:00
REPLICAT RUNNING REP_90_1 00:00:05 00:00:08
REPLICAT RUNNING REP_90_2 00:00:00 00:00:00
REPLICAT RUNNING REP_90_3 00:00:00 00:00:01

2018-05-31 17:04:28 ERROR OGG-01161 Bad column index (31) specified for table CFS.T_CFS_ACCT_ITEM, max columns = 31.

2018-05-31 17:04:28 ERROR OGG-01668 PROCESS ABENDING.

源端进行操作，先将原来的defgen文件转存一份加时间日期。
[oracle@bapdb1 ggs12]$cd dirdef/
[oracle@bapdb1 ggs12]$mv defgen75_3.def defgen75_3.def_0531
[oracle@bapdb1 ggs12]$ pwd
/ggs/ggs12
[oracle@bapdb1 ggs12]$defgen paramfile ./dirprm/defgen75_3.prm
生成新的defgen75_3.def文件，传送到目标kafka服务器中，

kafka端操作：
[app@PBSDATASKAFKAGROUP2-1 ggs12]$cd dirdef/
[app@PBSDATASKAFKAGROUP2-1 dirdef]$ mv defgen75_3.def defgen75_3.def_0531

然后在目标库中添加字段操作！

最后kafka目标机中ogg进程启动即可，
GGSCI (PBSDATASKAFKAGROUP2-1) 1> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING
REPLICAT RUNNING REP_75 00:00:05 00:00:01
REPLICAT RUNNING REP_75_2 00:00:03 00:00:02
REPLICAT RUNNING REP_75_3 00:00:04 00:00:05
REPLICAT RUNNING REP_90 00:00:00 00:00:00
REPLICAT RUNNING REP_90_1 00:00:05 00:00:05
REPLICAT RUNNING REP_90_2 00:00:00 00:00:05
REPLICAT RUNNING REP_90_3 00:00:00 00:00:07