Oracle报错实例分析：运维DBA反映Oracle数据库出现报错ORA-00600: internal error code, arguments: [4400], [48]，分析原因为跟分布式事务有关。
分析一个ORA-600错误，用UE打开trace，看到如下错误：
Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.8.0 - Production
ORACLE_HOME = /install1/oracle/bill1/product/9.2.0
System name: AIX
Node name: billing1
Release: 3
Version: 5
Machine: 00CB104D4C00
Instance name: bill1
Redo thread mounted by this instance: 1
Oracle process number: 148
Unix process pid: 1363982, image: oracle@billing1 (TNS V1-V3)
*** SESSION ID:(578.13852) 2012-06-04 12:08:44.492
*** 2012-06-04 12:08:44.492
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [4400], [48], [], [], [], [], [], []
可以看到，该系统是aix 5.3，db version是9208 单实例。
继续查看下面的call stack，内容如下：
----- Call Stack Trace -----
calling 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎call 错误的粉碎 错误的粉碎 entry 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎argument values in hex 错误的粉碎 错误的粉碎
location 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 type 错误的粉碎 错误的粉碎 point 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎( 错误的粉碎means dubious value) 错误的粉碎 错误的粉碎
-------------------- -------- -------------------- ----------------------------
ksedmp+0148 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 ksedst 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 1029555CC
ksfdmp+0018 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 01FD46A8 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
kgeriv+0118 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 _ptrgl 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
kgeasi+00cc 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 kgeriv 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 000000002 错误的粉碎 1100A2128
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 102954750 错误的粉碎 7000001B2B78F30
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 000000000
ktcddt+013c 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 kgeasi 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 110006838 错误的粉碎 1103923A8
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 113000001130 错误的粉碎 200000002
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 100000001 错误的粉碎 000000004
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 000000030 错误的粉碎 7000001B1B74EE8
ktcsod+01f8 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 ktcddt 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 110006838 错误的粉碎 110006978
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 000000000
kssdch_stage+02b8 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 _ptrgl 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
kssdch+0014 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 kssdch_stage 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 000000000 错误的粉碎 700000000007D98
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 110002F50
ktcbod+030c 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 kssdch 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 11000D618 错误的粉碎 000000008
kssdch_stage+02b8 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 _ptrgl 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
kssdch+0014 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 kssdch_stage 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 110002F50 错误的粉碎 110061758
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 000000009
ksuxds+1118 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 kssdch 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 1000E82E4 错误的粉碎 000000000
ksudel+006c 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 ksuxds 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 7000001AB716A20 错误的粉碎 100000001
opilof+03dc 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 01FD4914 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
opiodr+08cc 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 _ptrgl 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
ttcpip+0cc4 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 _ptrgl 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
opitsk+0d60 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 ttcpip 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎http://www.oracleplus.net 11000D4C0 错误的粉碎 000000000
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 000000000 错误的粉碎 000000000
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 000000000 错误的粉碎 000000000
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 000000000 错误的粉碎 000000000
opiino+0758 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 opitsk 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 000000000 错误的粉碎 000000000
opiodr+08cc 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 _ptrgl 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
opidrv+032c 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 opiodr 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 3C00000018 错误的粉碎 4101FAA40
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 FFFFFFFFFFFF8F0 错误的粉碎 0A0012010
sou2o+0028 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 bl 错误的粉碎 错误的粉碎 错误的粉碎 opidrv 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 3C0C000000 错误的粉碎 4A0059B20
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 FFFFFFFFFFFF8F0
main+0138 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎bl 错误的粉碎 错误的粉碎 错误的粉碎 01FD40E8 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
__start+0098 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 bl 错误的粉碎 错误的粉碎 错误的粉碎 main 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 000000000 错误的粉碎 000000000
mos上关于该错误的描述是这样的:
PURPOSE: 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
错误的粉碎This article discusses the internal error "ORA-600 [4400]", what 错误的粉碎it means and possible actions. The information here is only applicable 错误的粉碎to the versions listed and is provided only for guidance.
ERROR: 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
错误的粉碎ORA-600 [4400] [a] [b] [c] [d] [e]
VERSIONS:
错误的粉碎versions 6.0 to 11
DESCRIPTION:
错误的粉碎Internal error 4400 means that we are trying to delete a transaction (for 错误的粉碎example at logoff time) but the transaction has not yet been marked 错误的粉碎completed.
错误的粉碎This can happen at the remote site in a distributed transaction if the 错误的粉碎first part of the first stage of a two phase commit gets an error before 错误的粉碎it really starts the protocol.
FUNCTIONALITY: 错误的粉碎 错误的粉碎
错误的粉碎TRANSACTION CONTROL
IMPACT: 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎
错误的粉碎PROCESS FAILURE - but only at logoff so minimal impact
错误的粉碎NON CORRUPTIVE - No underlying data corruption.
该文档描述说4400错误是跟分布式事务有关，曾经也遇到不少关于分布式事务的问题，以前也写过一篇：ORA-01591: lock held by in-doubt distributed transaction
针对该错误，对比call stack 可以发现，基本上完全一致，该文档说该错误完全可以忽略，如下：
ORA-00600 [4400], [48], [], [], [], [] From a Distributed Transaction [ID 464861.1]
Symptoms
The following error is reported on 9.2.0.5:
ORA-00600: internal error code, arguments: [4400], [48], [], [], [], [], [], []
The call stack is:
ksedmp ksfdmp kgeriv kgeasi ktcddt ktcsod kssdch_stage kssdch ktcbod
kssdch_stage kssdch PGOSF40__ksuxds ksudel kxfprdp opirip opidrv sou2o
Cause
The error is encountered due to Bug 3840810 which was fixed in version 10.1.0.3.
The error is encountered when there is a dblink between 8i and 9i/10g databases. This error is only raised
in the log-off of the local session while trying to delete a transaction but the transaction has not yet
been marked completed. This lack of information is caused by the bug and if there is no process failure due
to this error, it can be ignored since there is no SQL statement/session affected.
This bug has been fixed by architectural changes in 10g and unfortunately is not backportable to 9.2.
If this is an one time occurrence then it can be safely ignored.
我们可以从trace里面找到如下信息：
BH (0x700000134fdd900) file#: 405 rdba: 0x65403498 (405/13464) class 1 ba: 0x700000134764000
错误的粉碎set: 84 dbwrid: 3 obj: 343488 objn: 343488
错误的粉碎hash: [700000070fead00,700000163feec00] lru: [7000000e8fe6d68,70000013ffe8468]
错误的粉碎LRU flags: hot_buffer
错误的粉碎ckptq: [7000000aefe93d8,700000088fe2dd8] fileq: [7000001b1174040,7000000eefc73e8]
错误的粉碎st: XCURRENT md: NULL rsop: 0x0 tch: 5
错误的粉碎flags: buffer_dirty gotten_in_current_mode block_written_once
错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎redo_since_read
错误的粉碎LRBA: [0x3e7cf.396e5.0] HSCN: [0x0bac.2f283f13] HSUB: [1] RRBA: [0x0.0.0]
错误的粉碎buffer tsn: 32 rdba: 0x65403498 (405/13464)
错误的粉碎scn: 0x0bac.2f283f13 seq: 0x01 flg: 0x02 tail: 0x3f130601
错误的粉碎frmt: 0x02 chkval: 0x0000 type: 0x06=trans data
Block header dump: 错误的粉碎0x65403498
Itl 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 Xid 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎Uba 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎 Flag 错误的粉碎Lck 错误的粉碎 错误的粉碎 错误的粉碎 错误的粉碎Scn/Fsc
0x01 错误的粉碎 0x0040.026.00167771 错误的粉碎0x7203e25c.0f91.02 错误的粉碎C--- 错误的粉碎 错误的粉碎0 错误的粉碎scn 0x0bac.2f26d569
0x02 错误的粉碎 0x008c.04b.002ae01f 错误的粉碎0x7242e607.0089.07 错误的粉碎C--- 错误的粉碎 错误的粉碎0 错误的粉碎scn 0x0bac.2f26d709
0x03 错误的粉碎 0x0043.04f.00184d78 错误的粉碎0x720020a7.e5ff.2f 错误的粉碎C--- 错误的粉碎 错误的粉碎0 错误的粉碎scn 0x0bac.2f28047f
.......
0x30 错误的粉碎 0x0005.041.001d254f 错误的粉碎0x228560f5.c941.0e 错误的粉碎--U- 错误的粉碎 错误的粉碎1 错误的粉碎fsc 0x0075.2f282d74
........
0x3f 错误的粉碎 0x008e.012.0029e870 错误的粉碎0x22801e5b.0444.06 错误的粉碎C--- 错误的粉碎 错误的粉碎0 错误的粉碎scn 0x0bac.2f26d4de
0x40 错误的粉碎 0x0005.000.001d261c 错误的粉碎0x228560f5.c941.0a 错误的粉碎--U- 错误的粉碎 错误的粉碎1 错误的粉碎fsc 0x007b.2f282d25
.........
0x61 错误的粉碎 0x0020.004.000bc447 错误的粉碎0x7203ce74.e107.12 错误的粉碎C--- 错误的粉碎 错误的粉碎0 错误的粉碎scn 0x0bac.2f26d66a
0x62 错误的粉碎 0x0049.04c.001e453a 错误的粉碎0x2a42eb96.ffe3.06 错误的粉碎C--- 错误的粉碎 错误的粉碎0 错误的粉碎scn 0x0bac.2f26d647
LRBA 是recover的起点，这个是checkpoint东西，大家可以参考这里：详解oracle checkpoint
从上面可以看到，所有事务falg都是C或U，表示事务都是提交了的，说明这个错误确实没有任何影响。
更多Oracle相关信息见Oracle 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=12