环境:11.2.0.3.6 双节点RAC
系统:aix power system 6.1
问题:一个节点在凌晨出现ORA-600错误
alert日志:
Global Enqueue Services Deadlock detected. More info in file
/u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_lmd0_20513176.trc.
Wed Jul 16 23:59:59 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_10420630.trc (incident=6917166):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x1109BAFC8], [keybuf], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/trsen/trsen2/incident/incdir_6917166/trsen2_ora_10420630_i6917166.trc
Thu Jul 17 00:00:04 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_21430462.trc (incident=6918870):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x1109DAFC8], [keybuf], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/trsen/trsen2/incident/incdir_6918870/trsen2_ora_21430462_i6918870.trc
Thu Jul 17 00:00:06 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_3802044.trc (incident=6919270):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x11091AFC8], [keybuf], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/trsen/trsen2/incident/incdir_6919270/trsen2_ora_3802044_i6919270.trc
Thu Jul 17 00:00:06 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_22872348.trc (incident=6916846):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x1108AAFA8], [keybuf], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/trsen/trsen2/incident/incdir_6916846/trsen2_ora_22872348_i6916846.trc
Thu Jul 17 00:00:06 2014
Dumping diagnostic data in directory=[cdmp_20140717000006], requested by (instance=2, osid=21430462), summary=[incident=6918870].
Thu Jul 17 00:00:06 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_20710582.trc (incident=6919326):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x11099AFA8], [keybuf], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/trsen/trsen2/incident/incdir_6919326/trsen2_ora_20710582_i6919326.trc
Thu Jul 17 00:00:07 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_4588318.trc (incident=6918646):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x11098AFA8], [keybuf], [], [], [], [], [], [], [], [], []
Thu Jul 17 00:00:07 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_12650346.trc (incident=6924446):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x1108B74D0], [keybuf], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
从alert日志里可以看到由于调用这个函数[kghstack_underflow_internal_3],报错,接下来,看一下提示的详细trc文件
trc文件内容:
Dump continued from file: /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_10420630.trc
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x1109BAFC8], [keybuf], [], [], [], [], [], [], [], [], []
========= Dump for incident 6917166 (ORA 600 [kghstack_underflow_internal_3]) ========
*** 2014-07-16 23:59:59.721
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=11va1wb1z3hdm) -----
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+40 bl 107c70e00 000000000 ? 000000001 ?
000000003 ? 000000000 ?
000000000 ? 000000001 ?
000000003 ? 000000000 ?
ksedst1()+112 call skdstdst() 13816ED051D6EE31 ?
484428412F737464 ?
FFFFFFFFFFEF900 ? 000002004 ?
11050CC00 ? 10A252BD4 ?
000000000 ? 11050CC00 ?
ksedst()+40 call ksedst1() 30300000000 ? 002050033 ?
10A252BC8 ? 700000000025C ?
000000000 ? 000000000 ?
10A252228 ? 000000000 ?
dbkedDefDump()+1516 call ksedst() 000000000 ? 000000000 ?
FFFFFFFFFFEFAF0 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 300000003 ?
ksedmp()+72 call dbkedDefDump() 300000000 ? 1100009F8 ?
FFFFFFFFFFF00E0 ?
4844424409A07784 ?
10011CA24 ? 109A07778 ?
FFFFFFFFFFF0130 ? 110454B90 ?
ksfdmp()+100 call ksedmp() 000000000 ? 000000000 ?
009A07793 ? 10AC18EC0 ?
200000000000000 ? 000000000 ?
1106DC4E8 ? 11050CC00 ?
dbgexPhaseII()+1940 call ksfdmp() 000002004 ? 11050CC00 ?
系统:aix power system 6.1
问题:一个节点在凌晨出现ORA-600错误
alert日志:
Global Enqueue Services Deadlock detected. More info in file
/u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_lmd0_20513176.trc.
Wed Jul 16 23:59:59 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_10420630.trc (incident=6917166):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x1109BAFC8], [keybuf], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/trsen/trsen2/incident/incdir_6917166/trsen2_ora_10420630_i6917166.trc
Thu Jul 17 00:00:04 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_21430462.trc (incident=6918870):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x1109DAFC8], [keybuf], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/trsen/trsen2/incident/incdir_6918870/trsen2_ora_21430462_i6918870.trc
Thu Jul 17 00:00:06 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_3802044.trc (incident=6919270):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x11091AFC8], [keybuf], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/trsen/trsen2/incident/incdir_6919270/trsen2_ora_3802044_i6919270.trc
Thu Jul 17 00:00:06 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_22872348.trc (incident=6916846):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x1108AAFA8], [keybuf], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/trsen/trsen2/incident/incdir_6916846/trsen2_ora_22872348_i6916846.trc
Thu Jul 17 00:00:06 2014
Dumping diagnostic data in directory=[cdmp_20140717000006], requested by (instance=2, osid=21430462), summary=[incident=6918870].
Thu Jul 17 00:00:06 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_20710582.trc (incident=6919326):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x11099AFA8], [keybuf], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/trsen/trsen2/incident/incdir_6919326/trsen2_ora_20710582_i6919326.trc
Thu Jul 17 00:00:07 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_4588318.trc (incident=6918646):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x11098AFA8], [keybuf], [], [], [], [], [], [], [], [], []
Thu Jul 17 00:00:07 2014
Errors in file /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_12650346.trc (incident=6924446):
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x1108B74D0], [keybuf], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
从alert日志里可以看到由于调用这个函数[kghstack_underflow_internal_3],报错,接下来,看一下提示的详细trc文件
trc文件内容:
Dump continued from file: /u01/app/oracle/diag/rdbms/trsen/trsen2/trace/trsen2_ora_10420630.trc
ORA-00600: 内部错误代码, 参数: [kghstack_underflow_internal_3], [0x1109BAFC8], [keybuf], [], [], [], [], [], [], [], [], []
========= Dump for incident 6917166 (ORA 600 [kghstack_underflow_internal_3]) ========
*** 2014-07-16 23:59:59.721
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=11va1wb1z3hdm) -----
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+40 bl 107c70e00 000000000 ? 000000001 ?
000000003 ? 000000000 ?
000000000 ? 000000001 ?
000000003 ? 000000000 ?
ksedst1()+112 call skdstdst() 13816ED051D6EE31 ?
484428412F737464 ?
FFFFFFFFFFEF900 ? 000002004 ?
11050CC00 ? 10A252BD4 ?
000000000 ? 11050CC00 ?
ksedst()+40 call ksedst1() 30300000000 ? 002050033 ?
10A252BC8 ? 700000000025C ?
000000000 ? 000000000 ?
10A252228 ? 000000000 ?
dbkedDefDump()+1516 call ksedst() 000000000 ? 000000000 ?
FFFFFFFFFFEFAF0 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 300000003 ?
ksedmp()+72 call dbkedDefDump() 300000000 ? 1100009F8 ?
FFFFFFFFFFF00E0 ?
4844424409A07784 ?
10011CA24 ? 109A07778 ?
FFFFFFFFFFF0130 ? 110454B90 ?
ksfdmp()+100 call ksedmp() 000000000 ? 000000000 ?
009A07793 ? 10AC18EC0 ?
200000000000000 ? 000000000 ?
1106DC4E8 ? 11050CC00 ?
dbgexPhaseII()+1940 call ksfdmp() 000002004 ? 11050CC00 ?