蚂蚁小窝

半个月来，新的网站服务器运行可靠性急遽下降，主要原因是jvm crash。而引起jvm crash的原因也都与java GC(Garbage Collection）相关。
  查了一下sun的文档“Trouble-Shooting and
Diagnostic Guide",总算有了点初步的认识。检查4个jvm 崩溃时的文档(hs_err_pidXXXX)，原因完全一致:
An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0x010d2d9e, pid=8089, tid=5417904
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_07-b03 mixed mode)
# Problematic frame:
# V  [libjvm.so+0x3efd9e]
#

---------------  T H R E A D  ---------------

Current thread (0x080e34a0):  VMThread [id=8090]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000058

Registers:
EAX=0x00000000, EBX=0x01292970, ECX=0x00670076, EDX=0xc82db880
ESP=0x00529d30, EBP=0x00529d48, ESI=0x01283c40, EDI=0x012a2094
EIP=0x010d2d9e, CR2=0x00000058, EFLAGS=0x00210216

Top of Stack: (sp=0x00529d30)
0x00529d30:   00670076 c82db880 0806bc60 0806bc60
0x00529d40:   01292970 0128a3a4 00529d68 010d30ac
0x00529d50:   0128a3a4 0047c800 0047aff4 00000000
0x00529d60:   fffffff0 01292970 00529d98 011c2584
0x00529d70:   012a2094 0128a3a4 00529d98 011b3b91
0x00529d80:   005b7cb7 0047cbd0 011f0373 01292970
0x00529d90:   00000002 0806bb40 00529dd8 00ef1bc0
0x00529da0:   012a2094 00000000 00000001 00000001

Instructions: (pc=0x010d2d9e)
0x010d2d8e:   40 08 8b 14 88 8b 42 04 8d 48 08 8b 40 08 52 51
0x010d2d9e:   ff 50 58 8b 06 83 c4 10 8b 08 85 c9 75 d9 eb d0

Stack: [0x004aa000,0x0052b000),  sp=0x00529d30,  free space=511k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0x010d2d9e, pid=8089, tid=5417904
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_07-b03 mixed mode)
# Problematic frame:
# V  [libjvm.so+0x3efd9e]
#

---------------  T H R E A D  ---------------

Current thread (0x080e34a0):  VMThread [id=8090]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000058

Registers:
EAX=0x00000000, EBX=0x01292970, ECX=0x00670076, EDX=0xc82db880
ESP=0x00529d30, EBP=0x00529d48, ESI=0x01283c40, EDI=0x012a2094
EIP=0x010d2d9e, CR2=0x00000058, EFLAGS=0x00210216

Top of Stack: (sp=0x00529d30)
0x00529d30:   00670076 c82db880 0806bc60 0806bc60
0x00529d40:   01292970 0128a3a4 00529d68 010d30ac
0x00529d50:   0128a3a4 0047c800 0047aff4 00000000
0x00529d60:   fffffff0 01292970 00529d98 011c2584
0x00529d70:   012a2094 0128a3a4 00529d98 011b3b91
0x00529d80:   005b7cb7 0047cbd0 011f0373 01292970
0x00529d90:   00000002 0806bb40 00529dd8 00ef1bc0
0x00529da0:   012a2094 00000000 00000001 00000001

Instructions: (pc=0x010d2d9e)
0x010d2d8e:   40 08 8b 14 88 8b 42 04 8d 48 08 8b 40 08 52 51
0x010d2d9e:   ff 50 58 8b 06 83 c4 10 8b 08 85 c9 75 d9 eb d0

Stack: [0x004aa000,0x0052b000),  sp=0x00529d30,  free space=511k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

..............

VM_Operation (0x0612cf40): full generation collection, mode: safepoint, requested by thread 0x08a11b30

...............................

Heap
def new generation   total 52480K, used 26959K [0xb6800000, 0xba0f0000, 0xbce30000)
  eden space 46656K,  55% used [0xb6800000, 0xb811aa28, 0xb9590000)
  from space 5824K,  21% used [0xb9590000, 0xb96c92f0, 0xb9b40000)
  to   space 5824K,   0% used [0xb9b40000, 0xb9b40000, 0xba0f0000)
tenured generation   total 466048K, used 242672K [0xbce30000, 0xd9550000, 0xf0000000)
   the space 466048K,  52% used [0xbce30000, 0xcbb2c398, 0xcbb2c400, 0xd9550000)

compacting perm gen  total 50432K, used 50281K [0xf0000000, 0xf3140000, 0xf4000000)
   the space 50432K,  99% used [0xf0000000, 0xf311a608, 0xf311a800, 0xf3140000)
No shared spaces configured.

.......................

再查jboss的文档，初步判断jvm崩溃的原因是permsize不足所致。因为系统大量使用了spring beans，jvm需要较多地使用 Permanent Generation Heap来存储reflective data。
解决方案是：在JAVA_OPTS加上-XX:MaxPermSize=128m，jdk5.0默认的初始值为8Mb(client)/16Mb(server),最大值为64Mb.

再使用jmap观察调整的效果：
cd /opt/jdk/bin
./jmap -heap 9657
Attaching to process ID 9657, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.5.0_07-b03

using thread-local object allocation.
Mark Sweep Compact GC

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 964689920 (920.0MB)
   NewSize          = 655360 (0.625MB)
   MaxNewSize       = 4294901760 (4095.9375MB)
   OldSize          = 1441792 (1.375MB)
   NewRatio         = 8
   SurvivorRatio    = 8
   PermSize         = 67108864 (64.0MB)
   MaxPermSize      = 134217728 (128.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 96468992 (92.0MB)
   used     = 4733136 (4.5138702392578125MB)
   free     = 91735856 (87.48612976074219MB)
   4.906380694845448% used
Eden Space:
   capacity = 85786624 (81.8125MB)
   used     = 4733136 (4.5138702392578125MB)
   free     = 81053488 (77.29862976074219MB)
   5.517335662958365% used
From Space:
   capacity = 10682368 (10.1875MB)
   used     = 0 (0.0MB)
   free     = 10682368 (10.1875MB)
   0.0% used
To Space:
   capacity = 10682368 (10.1875MB)
   used     = 0 (0.0MB)
   free     = 10682368 (10.1875MB)
   0.0% used
tenured generation:
   capacity = 857538560 (817.8125MB)
   used     = 193104168 (184.15848541259766MB)
   free     = 664434392 (633.6540145874023MB)
   22.51842389454767% used
Perm Generation:
   capacity = 67108864 (64.0MB)
   used     = 45904968 (43.77838897705078MB)
   free     = 21203896 (20.22161102294922MB)
   68.40373277664185% used

现在Perm Generation只是使用了68%，而之前4次jvm崩溃是，该值均为99%.
新服务器的可用性问题是否因此得到解决，还有待观察。