一次.net托管内存泄露分析

原创

于 2021-06-30 15:39:32 发布 · 1.8k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#开发框架 #.NET #Java #Windows

针对一个.NET服务进程的CPU抖动问题，通过抓取进程dump并分析，发现大量托管对象堆积于Gen2内存中，疑似存在内存泄露。进一步追踪发现，某些客户端对象间接引用自ShowHand.NetSharp.Client，导致大量对象无法正常回收。

简介：一次.net托管内存泄露分析

最近协助分析了一个.net进程内存泄露的问题，过程分享给大家。

症状：客户的服务端.net进程出现分钟级的cpu抖动，接近100%后落回。

图1

分析：支持同学通过procdump.exe工具抓取了进程dump, 设定触发dump的条件为，若进程的CPU使用量超过80%持续1秒，则开始抓取。

procdump.exe -ma -s 1 -c 80 10672 f:\aliyun

Loading Dump File [E:\temp\201127\GameServer.exe_201119_171245.dmp\GameServer.exe_201119_171245.dmp]

User Mini Dump File with Full Memory: Only application data is available

Comment: '

*** e:\soft\procdump\procdump.exe -ma -s 1 -c 80 10672 f:\aliyun

*** Process exceeded 80% CPU (system scale) for 1 second. Value: 88%. Hottest Thread: 4196 (0x1064).'

************* Path validation summary **************

Response Time (ms) Location

Deferred srv*F:\symbols*https://msdl.microsoft.com/download/symbols

Symbol search path is: srv*F:\symbols*https://msdl.microsoft.com/download/symbols

Executable search path is:

Windows 8.1 Version 9600 MP (32 procs) Free x64

Product: Server, suite: TerminalServer DataCenter SingleUserTS

6.3.9600.18217 (winblue_ltsb.160124-0053)

Machine Name:

Debug session time: Thu Nov 19 17:12:45.000 2020 (UTC + 8:00)

System Uptime: 38 days 3:36:48.460

Process Uptime: 0 days 0:33:22.000

在dump抓取时，所采样的系统CPU负载高达91%。

0:059> .loadby sos clr

0:059> !threadpool

CPU utilization: 91%

Worker Thread: Total: 57 Running: 3 Idle: 49 MaxLimit: 32767 MinLimit: 32

Work Request in Queue: 0

--------------------------------------

Number of Timers: 1

--------------------------------------

Completion Port Thread:Total: 89 Free: 88 MaxFree: 64 CurrentLimit: 89 MaxLimit: 1000 MinLimit: 65

查看dump抓取瞬间，有为数不多的几个线程在使用CPU。

1) Thread 37

0:059> ~37s

mscorlib_ni!System.IO.FileStream.WriteFileNative(Microsoft.Win32.SafeHandles.SafeFileHandle, Byte[], Int32, Int32, System.Threading.NativeOverlapped*, Int32 ByRef)$##600184B+0x86:

00007ffb`c5923d76 48894de0 mov qword ptr [rbp-20h],rcx ss:000000e2`16b0ee20=000000e3cde2fc10

0:037> kL

# Child-SP RetAddr Call Site

00 000000e2`16b0edf0 00007ffb`c5923cc2 mscorlib_ni!System.IO.FileStream.WriteFileNative(Microsoft.Win32.SafeHandles.SafeFileHandle, Byte[], Int32, Int32, System.Threading.NativeOverlapped*, Int32 ByRef)$##600184B+0x86

01 000000e2`16b0ee50 00007ffb`c5923aa7 mscorlib_ni!System.IO.FileStream.WriteCore(Byte[], Int32, Int32)$##600183D+0x62

02 000000e2`16b0eec0 00007ffb`c5923a34 mscorlib_ni!System.IO.FileStream.FlushInternalBuffer()$##600182F+0x57

03 000000e2`16b0ef00 00007ffb`c58d4f4c mscorlib_ni!System.IO.FileStream.Flush(Boolean)$##600182E+0x24

04 000000e2`16b0ef40 00007ffb`69ccac7b mscorlib_ni!System.IO.StreamWriter.Flush(Boolean, Boolean)$##60019BE+0x8c

05 000000e2`16b0efa0 00007ffb`69ccaaae 0x00007ffb`69ccac7b

06 000000e2`16b0efe0 00007ffb`69ca103e 0x00007ffb`69ccaaae

07 000000e2`16b0f030 00007ffb`c58eca72 0x00007ffb`69ca103e

08 000000e2`16b0f070 00007ffb`c58ec904 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003A95+0x162

09 000000e2`16b0f140 00007ffb`c58ec8c2 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003A94+0x14

0a 000000e2`16b0f170 00007ffb`c5926472 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)$##6003A93+0x52

0b 000000e2`16b0f1c0 00007ffb`c8fd6793 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()$##6003B8E+0x52

0c 000000e2`16b0f200 00007ffb`c8fd6665 clr!CallDescrWorkerInternal+0x83

0d 000000e2`16b0f240 00007ffb`c8fd736d clr!CallDescrWorkerWithHandler+0x4e

0e 000000e2`16b0f280 00007ffb`c90bbf59 clr!MethodDescCallSite::CallTargetWorker+0xf8

0f 000000e2`16b0f380 00007ffb`c8fd7ce5 clr!ThreadNative::KickOffThread_Worker+0x109

10 000000e2`16b0f5e0 00007ffb`c8fd7c60 clr!Frame::Push+0x59

11 000000e2`16b0f620 00007ffb`c8fd7b9e clr!FillInRegTypeMap+0x198

12 000000e2`16b0f720 00007ffb`c8fd7d1f clr!FillInRegTypeMap+0xc1

13 000000e2`16b0f7b0 00007ffb`c90bbe3b clr!FillInRegTypeMap+0x47

14 000000e2`16b0f810 00007ffb`c919159f clr!ThreadNative::KickOffThread+0xdb

15 000000e2`16b0f8e0 00007ffb`d90d13d2 clr!Thread::intermediateThreadProc+0x86

16 000000e2`16b0fa20 00007ffb`d92254f4 kernel32!BaseThreadInitThunk+0x22

17 000000e2`16b0fa50 00000000`00000000 ntdll!RtlUserThreadStart+0x34

0:037> ub rip

mscorlib_ni!System.IO.FileStream.WriteFileNative(Microsoft.Win32.SafeHandles.SafeFileHandle, Byte[], Int32, Int32, System.Threading.NativeOverlapped*, Int32 ByRef)$##600184B+0x6c:

00007ffb`c5923d5c c9 leave

00007ffb`c5923d5d 4903d1 add rdx,r9

00007ffb`c5923d60 4533c9 xor r9d,r9d

00007ffb`c5923d63 4c894c2420 mov qword ptr [rsp+20h],r9

00007ffb`c5923d68 4c8d4de8 lea r9,[rbp-18h]

00007ffb`c5923d6c 448bc0 mov r8d,eax

00007ffb`c5923d6f e8e40bf2ff call mscorlib_ni!System.Runtime.Remoting.Activation.ActivationServices.GetActivator()$##6005B45 (mscorlib_ni+0x434958) (00007ffb`c5844958)

00007ffb`c5923d74 33c9 xor ecx,ecx

2) Thread 39

0:042> ~39s

mscorlib_ni!System.Text.UTF8Encoding.GetBytes(Char*, Int32, Byte*, Int32, System.Text.EncoderNLS)$##600675A+0x1df:

00007ffb`c5963cef f7c280ff80ff test edx,0FF80FF80h

0:039> kL

# Child-SP RetAddr Call Site

00 000000e2`16d0ee40 00007ffb`c58d50be mscorlib_ni!System.Text.UTF8Encoding.GetBytes(Char*, Int32, Byte*, Int32, System.Text.EncoderNLS)$##600675A+0x1df

01 000000e2`16d0eed0 00007ffb`c58d4f17 mscorlib_ni!System.Text.EncoderNLS.GetBytes(Char[], Int32, Int32, Byte[], Int32, Boolean)$##6006608+0x11e

02 000000e2`16d0ef60 00007ffb`69ccac7b mscorlib_ni!System.IO.StreamWriter.Flush(Boolean, Boolean)$##60019BE+0x57

03 000000e2`16d0efc0 00007ffb`69ccaaae 0x00007ffb`69ccac7b

04 000000e2`16d0f000 00007ffb`69ca103e 0x00007ffb`69ccaaae

05 000000e2`16d0f050 00007ffb`c58eca72 0x00007ffb`69ca103e

06 000000e2`16d0f090 00007ffb`c58ec904 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003A95+0x162

07 000000e2`16d0f160 00007ffb`c58ec8c2 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)$##6003A94+0x