ASMB process grows raising ora-4030 intermittently (Doc ID 735180.1)

1.1 常熟项目数据库打补丁

1.1.1 项目名称

常熟项目

1.1.2 问题描述

针对 11月19号早上9数据库故障修复

1.1.3 影响范围

申请时间凌晨停机维护

1.1.4 问题分析

问题1：

Wed Nov 19 06:26:30 2014

Errors in file/u01/app/oracle/diag/rdbms/rac/rac2/trace/rac2_asmb_7546.trc (incident=108177):

ORA-04030: out of process memory when trying to allocate 592 bytes(callheap,kfmditer)

Incident details in:/u01/app/oracle/diag/rdbms/rac/rac2/incident/incdir_108177/rac2_asmb_7546_i108177.trc

Errors in file/u01/app/oracle/diag/rdbms/rac/rac2/trace/rac2_asmb_7546.trc (incident=108178):

ORA-04030: out of process memory when trying to allocate 528 bytes(kewa_dump_sess,kewa_ash_cb_nextsample: sample buffer)

ORA-04030: out of process memory when trying to allocate 592 bytes(callheap,kfmditer)

Incident details in:/u01/app/oracle/diag/rdbms/rac/rac2/incident/incdir_108178/rac2_asmb_7546_i108178.trc

Wed Nov 19 06:26:33 2014

Trace dumping is performing id=[cdmp_20141119062633]

Trace dumping is performing id=[cdmp_20141119062635]

Errors in file/u01/app/oracle/diag/rdbms/rac/rac2/trace/rac2_asmb_7546.trc:

ORA-04030: out of process memory when trying to allocate 592 bytes(callheap,kfmditer)

ASMB (ospid: 7546): terminating the instance due to error 4030

System state dump is made for local instance

System State dumped to trace file/u01/app/oracle/diag/rdbms/rac/rac2/trace/rac2_diag_7500.trc

Wed Nov 19 06:26:38 2014

ORA-1092 : opiodr aborting process unknown ospid(1159_47948857381328)

Wed Nov 19 06:26:38 2014

ORA-1092 : opitsk aborting process

Wed Nov 19 06:26:38 2014

Errors in file /u01/app/oracle/diag/rdbms/rac/rac2/trace/rac2_lms1_7520.trc (incident=108105):

通过查看Oracle官方文档这个异常是Oracle BUG引起

Bug 6851110
Abstract: ASMB PROCESS HAS GROWIN IN SIZE AND IS NOT RELEASING MEMORY.

ASMB process grows raising ora-4030 intermittently (Doc ID 735180.1)

In this Document   Symptoms   Cause   Solution   References APPLIES TO: Oracle Server - Standard Edition - Version: 11.1.0.6 and later   [Release: 11.1 and later ] Information in this document applies to any platform. SYMPTOMS ASMB process grows on memory, eventually leading to ora-4030 errors which causes DB crash. The reported error: ORA-04030: out of process memory when trying to allocate 552 Bytes (callheap,kfmditer) In the ASMB process heapdump we can see most of memory chunks are for 'kfmditer', example: BreakDown ~~~~~~~~~ Type     Count   Sum        Average ~~~~     ~~~~~   ~~~        ~~~~~~~ Free     285684  142841492  500.00 kfmditer 285685  157698132  552.00 Total = 300539624 bytes 293495.73k 286.62MB CAUSE Bug 6851110 Abstract: ASMB PROCESS HAS GROWIN IN SIZE AND IS NOT RELEASING MEMORY. Fixed In Ver: 11.2 SOLUTION This issue is fixed in 11.2.0.1 (Base Release)  11.1.0.7.1 (Patch Set Update)  10.2.0.5 (Server Patch Set)  11.1.0.7 Patch 11 on Windows Platforms  11.1.0.7 RAC Recommended Patch Bundle #1  11.1.0.6 Patch 11 on Windows Platforms    Applying the patch for the bug will solve this issue in lower patchset versions. You can check if Patch 6851110 is available for your patchset release and  O/S environment.:  Patch 6851110

1.1.5 解决方案

1：备份u01目录

su - oracle

  tar -cvf  /u01/backup/oraclehome.tar   /u01/

 2：确认Opatch版本号

 3：使用最新的Opatch 去打此补丁

#-------------------------------------------------------------------------

#  Interim Patch for BaseBugs: 6851110

#-------------------------------------------------------------------------

#

#  DATE:  Mon Dec 8 06:13:27 2008

# -------------------------------

#  Platform Patch for : LinuxAMD

#  Product Version #  : 11.1.0.7

#  Product Patched    : RDBMS

#

#  Bugs Fixed by this patch:

#  -------------------------

#  6851110:ASMB PROCESS HASGROWIN IN SIZE AND IS NOT RELEASING MEMORY.

#

#  Patch InstallationInstructions:

# --------------------------------

#  To apply the patch, unzipthe PSE container file:

#

# p6851110_11107_Linux-x86-64.zip

#

#  Set your current directoryto the directory where the patch

#  is located:

#

#    % cd 6851110

#  On AIX platforms only for10201 Patches  :

#

#  Ensure environment variableOBJECT_MODE is set to 32_64

#  e.g.

#

#    % setenv OBJECT_MODE32_64

#    or

#    $ exportOBJECT_MODE=32_64

#

#  Ensure that the directorycontaining the opatch script appears in

#  your $PATH; then enter thefollowing command:

#

#    % opatch apply -local 

#

#  Patch Special Instructions:

#  ---------------------------

#  Make sure all instancesrunning under the ORACLE_HOME being patched

#  are cleanly shutdown beforeinstalling this patch. Also ensure that

#  the tool used to terminatethe instance(s) has exited cleanly.

#

#  (a) Pre-install steps:  stop DB instance and then stop ASM instance

#  (b) Post-install steps:restart ASM instance and then DB instance

#

#  If the Oracle inventory isnot setup correctly this utility will

#  fail. To checkaccessibility to the inventory you can use the

#  command

#

#    % opatch lsinventory

#

#  If you have any problemsinstalling this PSE or are not sure

#  about inventory setupplease call Oracle support.

#

#  Patch DeinstallationInstructions:  --补丁回滚

# ----------------------------------

#  Use the following command:

#

#    % cd 6851110

#    % opatch rollback -id6851110 

1.1.6 规避方案

1.1.7 后续计划

1.1.8 参考资料

ASMB processgrows raising ora-4030 intermittently (Doc ID 735180.1)