Sometimes it’s easy to see why your .NET server application is using so much memory, but other times it makes no sense at all. I was at Microsoft earlier this week and someone who’d taken my debugging class stopped me and asked an excellent question. The scenario they had was their service memory would just grow at a steady rate without ever going down. The team found a fix for the memory leak through savvy internet searching but found it frustrating they could not see the answer through SOS and SOSEX. The person gave me the dumps as they wanted to know how they could have found the issue quicker. The research was pretty interesting so I thought I’d share the results.
The first command you always run after loading SOS is !dumpheap –stat so you can get a picture of the overall memory usage. On the dumps the team gave me, the result showed something very similar to the following at the end of the output:
In the real mini dump those WeakReferences were taking up over 270MB! The weak can kill you in the .NET world.
Whenever I see a WeakReference, you’re looking at some form of cache because it’s a special class that you use to reference an object, but allow that object to be garbage collected. So we know someone’s caching something, but who is doing the caching?
Running !dumpheap –type System.Weak yields the following output:
Yep, that List<WeakReference>, is probably the issue. So it’s time to look who created it by doing a !gcroot on it’s address.
<p>Life just got miserable. .NET stores static fields in an Object Array for each app domain. The static array is pinned in memory so that’s the clue. Sadly, with .NET 4 SOS the only way to see which object has the List<WeakReference> as a field without manually dumping each object in the heap. Back in the .NET 1.1 days there was a way to pretty easily figure out the holding class, but Microsoft changed the implementation so it no longer works.</p> <p>Fortunately, there is a way to figure out those static fields. All it takes is a little investment in the Professional edition on the amazing <a href="http://memprofiler.com/">.NET Memory Profiler</a>. Always purchase the Professional edition because that’s the version with the advanced feature to open mini dumps. Opening large mini dumps can take a long time as .NET Memory Profiler has to build up the reference chains and other data. However, I’m more than happy to let .NET Memory Profiler take it’s time because to do all of that work manually in SOS would consume months and make me give up technology.</p> <p>After opening the mini dump of my sample program, which took about 60 seconds, I typed List into the Overview tab to narrow down to the List<WeakReference></p> <p><a href="http://www.wintellect.com/devcenter/media/Default/Blogs/Images/jrobbins/image_68f31063.png"><img style="border-bottom:0px;border-left:0px;padding-left:0px;padding-right:0px;border-top:0px;border-right:0px;padding-top:0px" title="image" border="0" alt="image" src="http://www.wintellect.com/devcenter/media/Default/Blogs/Images/jrobbins/image_thumb_798b0b51.png" width="668" height="202"></a></p> <p>Double clicking on the on List<WeakRefefence> takes you to the Type details tab and shows you exactly who owns that pesky static.</p> <p><a href="http://www.wintellect.com/devcenter/media/Default/Blogs/Images/jrobbins/image_4c95a876.png"><img style="border-bottom:0px;border-left:0px;padding-left:0px;padding-right:0px;border-top:0px;border-right:0px;padding-top:0px" title="image" border="0" alt="image" src="http://www.wintellect.com/devcenter/media/Default/Blogs/Images/jrobbins/image_thumb_0b1af61d.png" width="678" height="194"></a></p> <p>It’s a <a href="http://msdn.microsoft.com/en-us/library/system.diagnostics.tracesource.aspx">TraceSource</a> so we have the culprit! In fact, looking at the type instance graph shows the whole reference chain.</p> <p><a href="http://www.wintellect.com/devcenter/media/Default/Blogs/Images/jrobbins/image_0d084526.png"><img style="border-bottom:0px;border-left:0px;padding-left:0px;padding-right:0px;border-top:0px;border-right:0px;padding-top:0px" title="image" border="0" alt="image" src="http://www.wintellect.com/devcenter/media/Default/Blogs/Images/jrobbins/image_thumb_32fdf57c.png" width="211" height="285"></a></p> <p>Looking at the TraceSource constructor in Reflector shows exactly where the WeakReference is created.</p>