See the I/O You Caused by Getting Your Diagnostic Tracing into Process Monitor

35 Comments April 13, 2010

Recently I was working on a performance bug for a client where our hypothesis was that there was too much file I/O going on in the process, which was causing unnecessary page faults. As this application is a kiosk application running on limited hardware with limited memory we needed to know exactly what the I/O patterns were especially since there were third party numerous components in the product we didn’t control. There’s only one tool for watching I/O and that’s the amazing Process Monitor from Sysinternals.

To help isolate down which actions caused what I/O we added tracing to the application to report when operations started and ended so we could see what operations were causing the most I/O. To get both the I/O stream and your tracing you have to run both Process Monitor and another Sysinternals tool, Debug View. We were able to narrow down the problem and recode some operations to eliminate the unnecessary I/O. However, having I/O in one program and tracing in another, it was very tedious to sit there and match up traces with the start of I/O in Process Monitor.

What I really wanted was for my trace statements to be part of the Process Monitor viewing so that way it would be trivial mapping the I/O activity to operations in my code. Fortunately, I have a personal developer at my disposal that is keen to tackle these kinds of challenges. He’s a very nice guy named Mark Russinovich who happens to be the author of Process Monitor. Mark is always eager to hear feature requests for his tools and I think he’s implemented at least 30 features in Sysinternals tools over the years that I thought would be great to have. Don’t hesitate to email Mark with feature ideas so he can be your personal developer as well.

Mark and I bounced a couple of emails back and forth discussing how the trace statements should get into Process Monitor. Originally, I thought merging the functionality of Debug View and Process Monitor would be the way to go but Mark thought adding the OutputDebugString capturing to Process Monitor would add lots of extraneous lines of output that would get in most people’s way. We settled on the idea that getting your tracing into Process Monitor should be something a developer specifically opts into by calling an API.

You can download the API I put together for both native and managed code to get your trace statements into Process Monitor here. For native code, you’ll need to include the header file ProcMonDebugOutput.h and link against ProcMonDebugOutputx86.lib or ProcMonDebugOutputx64.lib as appropriate. The API you’ll call is, appropriately named, ProcMonDebugOutput which takes a single parameter of a UNICODE string. Obviously, you’ll need to add ProcMonDebugOutputx86.DLL or ProcMonDebugOutputx64.DLL as part of your distribution.

For managed code, the API is wrapped up in a TraceListener derived class, ProcessMonitorTraceListener, in Sysinternals.Debug.DLL. That means you can add ProcessMonitorTraceListener through configuration files like any TraceListener you’ve ever used. With your application you’ll need to include Sysinternals.Debug.DLL as well as both ProcMonDebugOutputx86.DLL and ProcMonDebugOutputx64.DLL. The ProcessMonitorTraceListener works with both 32-bit and 64-bit code and calls the appropriate native DLL as necessary.

Once you’ve added calls to push your tracing to Process Monitor, you’ll see the trace output as part of the Profiling Events so make sure that the red circled button below is enabled to see them. In the screen shot, any line where the operation is “Debug Output Profiling” is one of your trace statements going through the API.

clip_image002

The new tracing API in Process Monitor will be a huge boon to developers trying to get a real picture of registry, file, and network I/O going on with individual operations. I’ve had the Process Monitor build that supports the API for a couple of weeks and it’s been fascinating to watch what really happens under the hood when certain operations happen in your code. I’m now much more cognizant of my I/O operations than ever before. Half the battle debugging is just seeing what’s going on and how you caused it. Process Monitor’s new API will make your debugging considerably easier!


35 Comments

  • Gravatar Image
    alik April 13, 2010 4:11 PM

    Amazing stuff. Thank you for sharing. I really love both tools [and others from sysinternals]. Combining their capabilities boosts perf consultant's productivity [personal perf?] in identifying perf issues - just what you outlined here.
    Appreciate you've shared it here.

    P.S. and I loved your book too :)

  • Gravatar Image
    Mike Diack April 13, 2010 5:33 PM

    Hi John

    Now that is useful. Any idea when the new process monitor will be released by sysinternals? I'm a huge of their stuff and this would be SERIOUSLY useful.

    Mike

  • Gravatar Image
    jrobbins April 13, 2010 9:44 PM

    Alik & Mike,

    I have no idea what happened with the Process Monitor release. Mark and I coordinated our blog postings so they were supposed to go out around the same time. Sadly, I guess his hasn't hit the web yet. It should be today or tomorrow at the latest.

    - John Robbins

  • Gravatar Image
    Josh Einstein April 14, 2010 3:30 AM

    Nice! I think you just obsoleted my TerminalTraceListener.

  • Gravatar Image
    Mike Diack April 15, 2010 2:11 AM

    It's out now (version 2.9) at Sysinternals

  • Gravatar Image
    Oliver Giesen April 15, 2010 4:21 AM

    Hmm, looks useful in principle but could you elaborate on why the two of you decided against simply displaying the OutputDebugString messages in ProcessMonitor? I don't quite see the benefit of the proprietary API as, just like ODS, it also doesn't allow to do anything beyond sending simple strings. Also, given PM's excellent filtering capabilities I can't quite buy into the "adding lots of extraneous lines of output that would get in most people’s way". I'm already only viewing 5% or less of total PM output at the best of times...

  • Gravatar Image
    Oliver Giesen April 15, 2010 6:37 AM

    FWIW: A Delphi translation of your API can now be found on my blog (hope the link makes it through):

    http://ogware.wordpress.com/2010/04/15/interface-unit-for-sending-debug-messages-to-sysinternals-processmonitor/

  • Gravatar Image
    jrobbins April 15, 2010 11:24 AM

    Oliver,

    Thanks for the port! Keep in mind it was more than just the OutputDebugString flood we were avoiding. First, the majority of Process Monitor users are *definitely not* software developers, but network administrators. Adding more stuff to the log that they probably don't care about is not helpful to them. Second it was much easier to add the custom interface than to essentially add all the Debug View code to Process Monitor. Like all development there are trade offs. :)

    Hope that clarifies!

    John Robbins

  • Gravatar Image
    alik April 15, 2010 6:04 PM

    Joh.
    I am unable to make it work. Receiving error 0x7d1 when running NativeTest.exe seems like the the API cannot connect to procmon.
    Compiled using VS2010 RC on win7 64.
    What am I doing wrong? What I should be doing to make it work?
    thank you!

  • Gravatar Image
    jrobbins April 15, 2010 6:34 PM

    Alik,

    I bet you have an older Process Monitor driver loaded because you ran an older version of Process Monitor. To ensure you have the latest driver, reboot your computer and make sure you load the 2.9 version of Process Monitor.

    - John Robbins

  • Gravatar Image
    alik April 16, 2010 1:38 AM

    John,
    HEAVEN! It did the trick.
    It scares me how powerful this tool is.

  • Gravatar Image
    Igor Okulist April 16, 2010 2:54 PM

    John,

    why not have a selection in process monitor itself (say checkbox-listbox) say from which process to capture OutputDebugString?

    -Igor

  • Gravatar Image
    jrobbins April 16, 2010 3:01 PM

    Alik,

    Glad we tracked that one down!

    Igor,

    See my previous response: http://www.wintellect.com/CS/blogs/jrobbins/archive/2010/04/13/see-the-i-o-you-caused-by-getting-your-diagnostic-tracing-into-process-monitor.aspx#13346

    John

  • Gravatar Image
    Igor Okulist April 16, 2010 4:22 PM

    John,

    (Your previous post does not completely covers the topic :()

    and I am thinking with "network admin" hat on and mindful to avoid flooding the Process Explorer.

    So it would be an option that is *OFF* by default (just like current performance monitoring feature), but user of Process Monitor can from the Process Monitor itself opt-in a *specific* application into it.

    -Igor

  • Gravatar Image
    VinceG April 19, 2010 5:47 AM

    John,

    my knowledge of C is just about zilch. How do I get Visual Studio to build the x86 version of the dll? Configuration Manager doesnt want to let me change to the x86 platform.

  • Gravatar Image
    jrobbins April 19, 2010 1:42 PM

    Vince,

    The easiest thing to do is go into the Build menu, select Batch Build, and in the Batch Build dialog, click "Select All" followed by the Build button. That'll build everything you need.

    - John Robbins

  • Gravatar Image
    VinceG April 20, 2010 11:24 AM

    Thanks John, that worked a treat, and the ManagedTest worked great with Process Monitor.

    Are the checkboxes on the Filter window one of your requests? I can never be bothered to save and re-load filters so that will be a useful addition!

  • Gravatar Image
    Andrey Kraynov May 4, 2010 5:39 AM

    John, I think this is the way it should look like in ProcMonDebugOutput.cpp - case DLL_PROCESS_ATTACH:case DLL_THREAD_ATTACH:case DLL_THREAD_DETACH: break;case DLL_PROCESS_DETACH: // Close the handle to the driver. CloseProcessMonitorLogger ( ) ; break;And a question - maybe it would be better to include OutputDebugString() call in ProcMonDebugOutput() so that the programmer is not forced to choose where he wants to see the data output?

  • Gravatar Image
    Paul Williams May 7, 2010 12:35 PM

    Intersting. As a developer, I would have much preferred including OutputDebugString() in Process Monitor. Yes, this will obsolete DbgView.exe in favor of Process Monitor, but that is completely acceptable to me.

  • Gravatar Image
    Chuck May 10, 2010 10:47 AM

    I have a problem running an intranet Web site that is only resolved by recycling the Application Pool. How can I use Process Monitor to find out what is going wrong? I filtered on aspnet_wp.exe, but I think I need to refine my filtering. I would like to see what threads are being created. I saw a Thread Exit operation, but I do not see where the thread was created.

  • Gravatar Image
    jrobbins May 10, 2010 1:18 PM

    For all of you wanting Process Monitor to handle OutputDebugString,

    Mark has heard you. When Mark and I were discussing this feature that's what I wanted as well but there were two choices. 1) redesign a major portion of Process Explorer or 2) extend an existing API to get the feature done quickly. Shipping is a feature. :)

    Chuck,

    If you've got a hung application pool, a debugger would be more appropriate than Process Monitor.

    Hope it helps!
    - John Robbins

  • Gravatar Image
    Chuck Sabol May 10, 2010 4:22 PM

    Thanks for Responding.

    Our App Pool deadlock condition only occurs on our production servers, not on our development boxes.

    Does your answer mean I do not have to attend your on line .NET Performance Tuning this Wednesday? Just kidding :)

  • Gravatar Image
    Pascal Renaudon May 12, 2010 11:42 AM

    I am using windows 7 x64; I recompiled the DLL and native test. I downloaded version 2.9. I set filter in procmon to nativetest.exe and click on show profiling events. Then I run NativeTest.exe and see "process profiling" in procmon. But NativeTest fails in CreateFile with a last error code of 2 meaning file not found. I really feel stupid to ask since so many others are using your DLL and it works fine but I dont see what i am missing

    pascal

  • Gravatar Image
    jrobbins May 12, 2010 8:21 PM

    Pascal,

    It's probably because you have an older version of the Process Monitor driver loaded. Reboot your machine and I bet everything works.

    - John Robbins

  • Gravatar Image
    pascal renaudon May 13, 2010 7:08 AM

    Thank you for your help. Now that I have rebooted everyting works fine. Your work will serve as basis for debugging deployment problems in our web app that calls into a native C++ DLL. We have had numerous issues related to file, security, DLL load. Now that we can have our traces AND procmon output finding these pb will be so much easier. Thank you again.

    Pascal

  • Gravatar Image
    Jonathan Edmonds May 26, 2010 11:22 AM

    This is pretty fantastic. I am finding value in this already as someone who already often used ProcMon. I wonder if, as a major benefit to folks using this for tracking down some performance issues, if there would be a way to easily show time deltas between two events. For example, if you highlighted two rows, if something in the status bar could show the time delta between the two rows automatically? Seems pretty trivial, and would save me a bunch of calc.exe entries!

    -Jonathan

  • Gravatar Image
    Sysinternals Site Discussion June 23, 2010 7:35 PM

    Process Monitor v2.9: This update to Process Monitor adds translations for more error codes, the ability

  • Gravatar Image
    Benjamin Kim August 8, 2010 9:52 PM

    Does it work on Windows XP?

  • Gravatar Image
    Manish February 22, 2011 8:31 AM

    I can make modifications to the shared code and distribute? Does it comes under some open source licensing?

  • Gravatar Image
    Dhaval October 4, 2012 10:58 AM

    John Robbins,
    We are using batch file to terminate procmon and generate csv file of procmon events.
    But How can I populate my ListBox with datasource as procmon?
    Or
    What is effective method to transfer data from Procmon to our application without terminating Procmon Process?
    Thanks

  • Gravatar Image
    Dwayne January 5, 2013 12:29 PM

    I found this gem last night. However I'm unable to build it. The post build process is failing. Can you provide the necessary managed dll for X86 and x64. I'm a novice and not familiar with the post build process macros.

  • Gravatar Image
    Dwayne January 5, 2013 12:31 PM

    What is the post build doing? Is it renaming and moving the dll's. Can I manully accomplish this task

  • Gravatar Image
    Vic Klien November 26, 2013 9:11 PM

    Great idea. Unfortunately, the link to the source code is currently 404-ing:
    http://www.wintellect.com/downloads/ProcMonDebugOutput.zip

    I see the help file for Process Monitor shows some raw C source, but it would be nice to have your ProcMonDebugOutput.zip.

    Thanks

  • Gravatar Image
    Shane Blank January 23, 2014 3:48 PM

    Great idea. Unfortunately, the link to the source code is currently 404-ing:
    http://www.wintellect.com/downloads/ProcMonDebugOutput.zip

    I would like to implement this but cannot find a download for it, can you help?

    Thanks

Have a Comment?

Archives