A few weeks ago, I got a nice mail from Steve Johnson saying he was working on a WinDBG extension to make it easier to set breakpoints in your .NET code. We swapped some mails bouncing additional features back and forth. Steve completely came through and his free SOSEX extension brings some hugely needed functionality to your hard core SOS debugging. This is especially important to .NET 2.0 because Microsoft dropped huge amounts of functionality in the SOS that ships with .NET 2.0 so digging through those production mini dumps was a complete carpel tunnel inducing exercise. Even better, Steve offers both 32 and 64-bit versions of SOSEX.
Steve supports the traditional !help command in SOSEX, but I thought I'd show you some tricks to show the super powers SOSEX puts at your fingertips. I really appreciate that Steve let me test and use SOSEX before releasing it. SOSEX is rock solid so you can start using it today.
(Note that for the rest of this entry, I'm assuming you know how to use WinDBG and SOS. If you don't, the links in the first paragraph will get you started.)
Setting Breakpoints
While SOS has the !bpmd (Breakpoint on Method Descriptor) command, it's limited because you can only set a breakpoint at the first instruction of the method. What would be better is setting a breakpoint on a source line. While WinDBG has a little CLR love now, that love is merely a vague attraction, not much interest. The SOSEX !bpsc (Breakpoint on Source Code), as the name implies, lets you set the breakpoint on the source, line, and optional column if you have multiple expressions on the line, such as a for statement. The help for the command shows the appropriate values you need to set (note that all values are in decimal):
0:007> !help bpsc
<strSourceFile> <decLineNum> [decColNum]
Sets a breakpoint at the specified source file, line and, if specified, column.
Typing in the command is fine, but you can use WinDBG to help you out. First you'll want to open your .NET source code in WinDBG with the File, Open Source File… command. Move the cursor to the line where you want to set the breakpoint, and switch to the command window. The WinDBG status bar will retain the line and column information. From that, you can easily set the breakpoint without having to guess. The following shot shows where I executed !bpsc SorterControl.cs 348.
What's really interesting to me is that WinDBG shows the breakpoint in the source code as it was native code. Once you execute the breakpoint you've set with !bpsc, you'll see the usual WinDBG purple highlight indicating you stopped on the line. The super extremely cool thing is that once stopped, if you start pressing F10, the step key, you single step your .NET source code! It's not perfect as you can't step into calls, but being able to single step both the .NET source side and the native source side is very exciting. While most of you are saying, "What's so exciting about that? I'll just use Visual Studio." Just try to do that mixed debugging on x64 some time. It doesn't work. The screen shot above, that's running on x64! For those of us fully supporting x64 now we have a way to finally really do mixed mode debugging thanks to Steve and SOSEX.
If you don't have source code, SOSEX offers up the !bpmo (Breakpoint on Method Offset) command. With it, you can set a breakpoint on a type method and IL offset. For those using Reflection.Emit, you just found the answer to how you can debug your generated code.
Seeing Arguments and Locals
One area where SOS really breaks down is trying see the locals and arguments for a particular method. While the !clrstack (Stack Walk) command has the –a option that's supposed to show them, but as anyone who's wrestled with that option, you know it doesn't work. About the only way to see what your locals and arguments are is to use the !dso (Dump Stack Objects) command with the starting and ending stack pointers for the method. While not perfect, it at least gets you close. What would be much better is a command that would just show us the locals and arguments and that's what the SOSEX !vars (Show Argument and Local Variables) command is all about. Using the screen shot above, I single stepped a few lines down and ran !vars:
0:000> !vars
Arguments:
[0]:this:0x14fd8 (Bugslayer.SortDisplayControl2.SorterControl2)
Locals:
[0]:elemCount:0x96 (System.Int32)
[1]:list:0x802bfdf8 (System.Int32[])
[2]:sl:0x802c0068 (System.Collections.SortedList)
[3]:doneIndex:0x0 (System.Int32)
[4]:next:0x82 (System.Int32)
[5]:i:0x0 (System.Int32)
[6]:CS$1$0000:null (System.Int32[])
[7]:CS$4$0001:0x0 (System.Boolean)
Solving Deadlocks
Sync blocks are the synchronization object of choice in .NET because they are blindingly fast. When you use the lock keyword, you're using a sync block. However, if you deadlock on a sync block, the SOS !syncblk (Show Sync Blocks) command shows you what each thread owns, but not what they are deadlocking on. In the output below, you can see that thread 4 owns the System.Object, that's about it. To find the deadlock you have to manually look at call stacks and objects on the stack. It's doable, but tedious.
0:016> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
28 00000000024af1c0 21 1 00000000024f0700 1650 6 00000000105cfde8 System.Byte[]
35 00000000024af3b8 3 1 00000000024d5d50 1404 4 00000000105cfcb8 System.Object
-----------------------------
Total 35
CCW 0
RCW 0
ComClassFactory 0
Free 0
If you think you've got a sync block deadlock, SOSEX's !dlk (Deadlock) command comes to the rescue. Do you think you can see the deadlock?
0:016> !dlk
Deadlock detected:
CLR thread 3 holds sync block 00000000024af3b8 OBJ:00000000105cfcb8[System.Object]
waits sync block 00000000024af1c0 OBJ:00000000105cfde8[System.Byte[]]
CLR thread 5 holds sync block 00000000024af1c0 OBJ:00000000105cfde8[System.Byte[]]
waits sync block 00000000024af3b8 OBJ:00000000105cfcb8[System.Object]
CLR Thread 3 is waiting at ThreadsDemo.MonitorBad.WriterFunc()+0x37(IL)
[c:\Junk\cruft\ThreadsDemo\MonitorBad.cs, line 63]
CLR Thread 5 is waiting at ThreadsDemo.MonitorBad.ReaderFunc()+0x15(IL)
[c:\Junk\cruft\ThreadsDemo\MonitorBad.cs, line 42]
1 deadlock detected.
Note that the thread numbers reported by !dlk are the managed thread IDs reported by the SOS !threads (Show Managed Threads) command, not the native debugger thread IDs. The confusion is that the !syncblk command shows the native thread IDs. I love how !dlk even tells you the source and line of the deadlock. With SOSEX it's now so easy to figure out a deadlock even a manager could do it!
Memory, Memory, and More Memory!
Where all the pain in production is figuring out what generation an object belongs and who's referencing an object. In .NET 1.1's SOS, there were nice options to dump each generational heap. Sadly, with .NET 2.0, those options went away so the only way to figure out what's in a particular generation is to manually dump out the ranges. While that's nice for simple applications, the instant you're using a server Garbage Collector (one heap per CPU) it's a huge and utter pain to piece all the output together.
For those of us with crippled fingers attempting to look at the generational heaps, you are so going to love the SOSEX !dumpgen command! To see the statistics for the gen 2 heap, simply use the !dumpgen 2 –stat command. This is a huge help! Basically, the !dumpgen command is like the !dumpheap command, but dedicated to looking at particular heaps. To see your objects in a particular generation, add the –type switch. I can't stress enough how useful it is to have the !dumgen command to look at those production mini dumps.
Because Steve is doing all the heavy memory analysis for us, he also added the !gcgen command which takes an object address on the command line and tells you what generation it's in. As you're fighting through a production mini dump, being able to do the following is wonderful:
0:000> !gcgen 0000064278807ee8
Gen 2
While seeing what's in the generational heaps is huge, my absolute favorite command in SOSEX is !refs (Show References). When you're looking at a .NET object, especially one which is in generation 2, you're often wondering why exactly that objects still in memory. It's there because another object has a reference to it. With the !refs command, it's trivial to see parent objects, those holding onto the object in question, and the child objects, those the object is holding onto:
0:004> !refs 0000000002696918
Objects referenced by 0000000002696918 (System.Diagnostics.DefaultTraceListener):
00000000026941c0 40 System.String STRVAL=Default
0000000002696b78 24 System.Collections.Specialized.StringDictionary
00000642787c78c8 26 System.String
Objects referencing 0000000002696918 (System.Diagnostics.DefaultTraceListener):
00000000026941e8 160 System.Diagnostics.ListenerElement
0000000002695468 24 System.Object
0000000002697150 48 System.Collections.ArrayList+ArrayListEnumeratorSimple
Thread 4
stack:000000001c1be7c8
stack:000000001c1be880
stack:000000001c1be8b0
stack:000000001c1be8e0
stack:000000001c1be8f0
stack:000000001c1be920
stack:000000001c1be930
Thanks Steve!
Go get SOSEX so you'll have an easier time in your next production debugging challenge. I really appreciate that Steve took the time to write SOSEX and release it for the rest of us.