转帖-大对象导致JVM Crash (Jboss) 分析及解决

本文记录了一次JVM Crash的故障排查过程，详细分析了hs_err_pid.log文件，最终定位到Young GC期间发生的异常，并给出了通过设置-XX:-ReduceInitialCardMarks参数来规避问题的方法。

最近在一个项目中，web 应用跑一段时间后, JBoss JVM crash ，web日志中没有任何异常。

存放日志的地方发现有 hs_err_pid25052.log，发现这个文件，就知道是JVM crash了。

打开这个文件然后分析：

Log代码

--------------- T H R E A D ---------------
Current thread (0x0000000050682000): GCTaskThread [stack: 0x00000000413fb000,0x00000000414fc000] [id=25059]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000018
Registers:
此处省略.........
Top of Stack: (sp=0x00000000414fae70)
此处省略.........
Instructions: (pc=0x00002aaffd33163c)
此处省略.........
Stack: [0x00000000413fb000,0x00000000414fc000], sp=0x00000000414fae70, free space=3ff0000000000000018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x62263c]
V [libjvm.so+0x6230b0]
V [libjvm.so+0x625672]
V [libjvm.so+0x365dda]
V [libjvm.so+0x5da2af]

分析1：JVM crash时，执行的线程GCTaskThread；V [libjvm.so+0x62263c]
代表执行的是VM代码，也就是说执行虚拟机的代码时JVM crash的。

Log代码

--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00000000517fe800 JavaThread "notifier[172.29.63.18:9090]104" daemon [_thread_blocked, id=27922, stack(0x000000004c36d000,0x000000004c46e000)]
0x0000000050f35000 JavaThread "RpcClientIoHandlerImpl-processor-106" daemon [_thread_blocked, id=27921, stack(0x000000004db85000,0x000000004dc86000)]
此处省略.........
Other Threads:
0x00000000506d2000 VMThread [stack: 0x0000000040b6a000,0x0000000040c6b000] [id=25062]
0x0000000050713800 WatcherThread [stack: 0x000000004085f000,0x0000000040960000] [id=25069]
=>0x0000000050682000 (exited) GCTaskThread [stack: 0x00000000413fb000,0x00000000414fc000] [id=25059]

分析2：这里日志是当前所有的Thread的列表。exited标示说明这个是执行线程。

Java代码

VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x0000000050663370] Threads_lock - owner thread: 0x00000000506d2000
[0x0000000050663870] Heap_lock - owner thread: 0x00000000517c4800
Heap
PSYoungGen total 43328K, used 43136K [0x00002aaacb760000, 0x00002aaace200000, 0x00002aaad6200000)
eden space 43008K, 100% used [0x00002aaacb760000,0x00002aaace160000,0x00002aaace160000)
from space 320K, 40% used [0x00002aaace1b0000,0x00002aaace1d0000,0x00002aaace200000)
to space 320K, 40% used [0x00002aaace160000,0x00002aaace180000,0x00002aaace1b0000)
PSOldGen total 349568K, used 119371K [0x00002aaab6200000, 0x00002aaacb760000, 0x00002aaacb760000)
object space 349568K, 34% used [0x00002aaab6200000,0x00002aaabd692e50,0x00002aaacb760000)
PSPermGen total 131072K, used 62813K [0x00002aaaae200000, 0x00002aaab6200000, 0x00002aaab6200000)
object space 131072K, 47% used [0x00002aaaae200000,0x00002aaab1f57670,0x00002aaab6200000)

分析3上，从上面的日志可以肯定是Young GC的时候发生了异常，导致JVM crash。

在网上查询这个异常，发现很多人碰到了这个问题；然后看到sun jdk 官网上，这个版本（1.6_18）有已知的问题，其中一条讲的就是我们的问题：

引用代码

Card-Marking Optimization Issue
• A flaw in the implementation of a card-marking performance optimization in the JVM can cause heap corruption under some circumstances. This issue affects the CMS garbage collector prior to 6u18, and the CMS, G1 and Parallel Garbage Collectors in 6u18. The serial garbage collector is not affected. Applications most likely to be affected by this issue are those that allocate very large objects which would not normally fit in Eden, or those that make extensive use of JNI Critical Sections (JNI Get/Release*Critical).
This issue will be fixed in the next Java SE 6 update.
Meanwhile, as a workaround to the issue, users should disable this performance optimization by -XX:-ReduceInitialCardMarks.

解决方案：通过jdk 增加这个 -XX:-ReduceInitialCardMarks 项，避免这个问题。

总结:

JVM crash时，充分分析JVM自动生成的hs_err_pid.log文件；如果确定是JVM的问题后，去网上google，并且上Sun的官网。

[@more@]

目前正在根据这篇文章进行客户问题的处理。

重点是如何根据出错的日志分析出来GC的问题。后续的跟进才是重点。

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/8304426/viewspace-1054170/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/8304426/viewspace-1054170/