Oracle 11.2 OGG-01117 Received signal: OMVS Shutdown (33)

AIX: 'GGS ERROR 190 ... : Received Signal: OMVS Shutdown (33).' (Doc ID 969596.1)

 

This message means that AIX had a system-wide paging problem, which it reported to the GoldenGate component, in this case the Collector process. The AIX system sent an unexpected UNIX SIGDANGER signal to the Collector process. Solution Details:Current GoldenGate versions are aware of this UNIX signal only at shutdown of a z/OS UNIX System Services (OMVS) environment. However, AIX sends SIGDANGER before an imminent system failure that is caused by a shortage of free paging-space pages. 

 

metalink上的文档说是pagespace的问题，看了一下errpt，发现确实有相关报错。

 

进一步用topas发现

oraagent的pagespace用的非常高，高过同类进程100倍以上。

 

于是又发现如下文章：oraagent.bin High Memory Usage as Dependent Listener was Removed/Renamed (Doc ID 1640721.1)

 

• <GI_HOME>/log/<node>/agent/crsd/oraagent_grid/oraagent_grid.log

2014-02-12 08:08:10.132: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 CLSCRS_STAT ret: 200
2014-02-12 08:08:10.132: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 clscrs_res_get_op_status CLSCRS_STAT status 210 err_msg CRS-0210: Could not find resource ' ora.LISTENER.lsnr'.    == ==>> resource missing
2014-02-12 08:08:10.132: [ USRTHRD][417280]{0:13:2} CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0 
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 CLSCRS_STAT ret: 200
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} checkCrsStat 2 clscrs_res_get_op_status CLSCRS_STAT status 210 err_msg CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} AsmCommonAgent::setLocalListener cls::Exception CRS-0210: Could not find resource 'ora.LISTENER.lsnr'.
2014-02-12 08:08:10.476: [ USRTHRD][417280]{0:13:2} ASM Dedicated Thread }

 

• "<GI_HOME>/bin/crsctl stat res -t" shows no 'ora.LISTENER.lsnr' resource

 

• "<GI_HOME>/bin/crsctl stat res ora.asm -p" shows ora.asm depends on 'ora.LISTENER.lsnr' resource

NAME=ora.asm
....
START_DEPENDENCIES=hard(ora.cssd) weak( ora.LISTENER.lsnr)

  

 

Cause

Dependent resource ' ora.LISTENER.lsnr' was removed.

The issue is being investigated in bug 18261183 and bug 18427298

Solution

While the bugs are still being worked by Development, the workaround is to add the missing "ora.LISTENER.lsnr" listener or set correct "listener" resource for ora.asm as grid user:

In the case of "ora.LISTENER.lsnr" is gone, add the listener back:

$ srvctl add listener -l LISTENER -p <port>

 

In the case of "ora.LISTENER.lsnr" was renamed to something else, i.e. "ora.ASM_LISTENER.lsnr"

$ srvctl modify asm -l ASM_LISTENER

 

确实如上文所说，缺少ora.LISTENER.lsnr，其实名字改了，但asm依赖的listener名字还是LISTENER

尝试srvctl modify asm -l <mylistener>

 

发现报错：

PRCA-1032: ASM Listener LISTENER Does Not Exist On 11.2 SIHA/OHAS Configuration.

 

只好继续查，找到 (Doc ID 1428284.1)

发现要解决这个问题，需要先添加名为listener的listener，再改，改完只好再删除这个新加的listener

 

srvctl add listener -l LISTENER -p 1522                        因为1521的已经被占用了。

srvctl modify asm -l <mylistener>    

现在可以了。

srvctl remove listener -l LISTENER

 

再将has重启一下，发现oraagent的pagespace就正常了。