John Robbins' Blog

See the I/O You Caused by Getting Your Diagnostic Tracing into Process Monitor

Recently I was working on a performance bug for a client where our hypothesis was that there was too much file I/O going on in the process, which was causing unnecessary page faults. As this application is a kiosk application running on limited hardware with limited memory we needed to know exactly what the I/O patterns were especially since there were third party numerous components in the product we didn’t control. There’s only one tool for watching I/O and that’s the amazing Process Monitor from Sysinternals.

To help isolate down which actions caused what I/O we added tracing to the application to report when operations started and ended so we could see what operations were causing the most I/O. To get both the I/O stream and your tracing you have to run both Process Monitor and another Sysinternals tool, Debug View. We were able to narrow down the problem and recode some operations to eliminate the unnecessary I/O. However, having I/O in one program and tracing in another, it was very tedious to sit there and match up traces with the start of I/O in Process Monitor.

What I really wanted was for my trace statements to be part of the Process Monitor viewing so that way it would be trivial mapping the I/O activity to operations in my code. Fortunately, I have a personal developer at my disposal that is keen to tackle these kinds of challenges. He’s a very nice guy named Mark Russinovich who happens to be the author of Process Monitor. Mark is always eager to hear feature requests for his tools and I think he’s implemented at least 30 features in Sysinternals tools over the years that I thought would be great to have. Don’t hesitate to email Mark with feature ideas so he can be your personal developer as well.

Mark and I bounced a couple of emails back and forth discussing how the trace statements should get into Process Monitor. Originally, I thought merging the functionality of Debug View and Process Monitor would be the way to go but Mark thought adding the OutputDebugString capturing to Process Monitor would add lots of extraneous lines of output that would get in most people’s way. We settled on the idea that getting your tracing into Process Monitor should be something a developer specifically opts into by calling an API.

You can download the API I put together for both native and managed code to get your trace statements into Process Monitor here. For native code, you’ll need to include the header file ProcMonDebugOutput.h and link against ProcMonDebugOutputx86.lib or ProcMonDebugOutputx64.lib as appropriate. The API you’ll call is, appropriately named, ProcMonDebugOutput which takes a single parameter of a UNICODE string. Obviously, you’ll need to add ProcMonDebugOutputx86.DLL or ProcMonDebugOutputx64.DLL as part of your distribution.

For managed code, the API is wrapped up in a TraceListener derived class, ProcessMonitorTraceListener, in Sysinternals.Debug.DLL. That means you can add ProcessMonitorTraceListener through configuration files like any TraceListener you’ve ever used. With your application you’ll need to include Sysinternals.Debug.DLL as well as both ProcMonDebugOutputx86.DLL and ProcMonDebugOutputx64.DLL. The ProcessMonitorTraceListener works with both 32-bit and 64-bit code and calls the appropriate native DLL as necessary.

Once you’ve added calls to push your tracing to Process Monitor, you’ll see the trace output as part of the Profiling Events so make sure that the red circled button below is enabled to see them. In the screen shot, any line where the operation is “Debug Output Profiling” is one of your trace statements going through the API.


The new tracing API in Process Monitor will be a huge boon to developers trying to get a real picture of registry, file, and network I/O going on with individual operations. I’ve had the Process Monitor build that supports the API for a couple of weeks and it’s been fascinating to watch what really happens under the hood when certain operations happen in your code. I’m now much more cognizant of my I/O operations than ever before. Half the battle debugging is just seeing what’s going on and how you caused it. Process Monitor’s new API will make your debugging considerably easier!

On Apr 13 2010 11:00 AMBy jrobbins With 35 Comments

Comments (35)

  1. Amazing stuff. Thank you for sharing. I really love both tools [and others from sysinternals]. Combining their capabilities boosts perf consultant's productivity [personal perf?] in identifying perf issues - just what you outlined here.
    Appreciate you've shared it here.

    P.S. and I loved your book too :)

  2. Hi John

    Now that is useful. Any idea when the new process monitor will be released by sysinternals? I'm a huge of their stuff and this would be SERIOUSLY useful.


  3. Alik & Mike,

    I have no idea what happened with the Process Monitor release. Mark and I coordinated our blog postings so they were supposed to go out around the same time. Sadly, I guess his hasn't hit the web yet. It should be today or tomorrow at the latest.

    - John Robbins

  4. Hmm, looks useful in principle but could you elaborate on why the two of you decided against simply displaying the OutputDebugString messages in ProcessMonitor? I don't quite see the benefit of the proprietary API as, just like ODS, it also doesn't allow to do anything beyond sending simple strings. Also, given PM's excellent filtering capabilities I can't quite buy into the "adding lots of extraneous lines of output that would get in most people’s way". I'm already only viewing 5% or less of total PM output at the best of times...

  5. FWIW: A Delphi translation of your API can now be found on my blog (hope the link makes it through):

  6. Oliver,

    Thanks for the port! Keep in mind it was more than just the OutputDebugString flood we were avoiding. First, the majority of Process Monitor users are *definitely not* software developers, but network administrators. Adding more stuff to the log that they probably don't care about is not helpful to them. Second it was much easier to add the custom interface than to essentially add all the Debug View code to Process Monitor. Like all development there are trade offs. :)

    Hope that clarifies!

    John Robbins

  7. Joh.
    I am unable to make it work. Receiving error 0x7d1 when running NativeTest.exe seems like the the API cannot connect to procmon.
    Compiled using VS2010 RC on win7 64.
    What am I doing wrong? What I should be doing to make it work?
    thank you!

  8. Alik,

    I bet you have an older Process Monitor driver loaded because you ran an older version of Process Monitor. To ensure you have the latest driver, reboot your computer and make sure you load the 2.9 version of Process Monitor.

    - John Robbins

  9. John,

    why not have a selection in process monitor itself (say checkbox-listbox) say from which process to capture OutputDebugString?


  10. Alik,

    Glad we tracked that one down!


    See my previous response:


  11. John,

    (Your previous post does not completely covers the topic :()

    and I am thinking with "network admin" hat on and mindful to avoid flooding the Process Explorer.

    So it would be an option that is *OFF* by default (just like current performance monitoring feature), but user of Process Monitor can from the Process Monitor itself opt-in a *specific* application into it.


  12. John,

    my knowledge of C is just about zilch. How do I get Visual Studio to build the x86 version of the dll? Configuration Manager doesnt want to let me change to the x86 platform.

  13. Vince,

    The easiest thing to do is go into the Build menu, select Batch Build, and in the Batch Build dialog, click "Select All" followed by the Build button. That'll build everything you need.

    - John Robbins

  14. Thanks John, that worked a treat, and the ManagedTest worked great with Process Monitor.

    Are the checkboxes on the Filter window one of your requests? I can never be bothered to save and re-load filters so that will be a useful addition!

  15. Andrey Kraynov

    John, I think this is the way it should look like in ProcMonDebugOutput.cpp - case DLL_PROCESS_ATTACH:case DLL_THREAD_ATTACH:case DLL_THREAD_DETACH: break;case DLL_PROCESS_DETACH: // Close the handle to the driver. CloseProcessMonitorLogger ( ) ; break;And a question - maybe it would be better to include OutputDebugString() call in ProcMonDebugOutput() so that the programmer is not forced to choose where he wants to see the data output?

  16. Paul Williams

    Intersting. As a developer, I would have much preferred including OutputDebugString() in Process Monitor. Yes, this will obsolete DbgView.exe in favor of Process Monitor, but that is completely acceptable to me.

  17. I have a problem running an intranet Web site that is only resolved by recycling the Application Pool. How can I use Process Monitor to find out what is going wrong? I filtered on aspnet_wp.exe, but I think I need to refine my filtering. I would like to see what threads are being created. I saw a Thread Exit operation, but I do not see where the thread was created.

  18. For all of you wanting Process Monitor to handle OutputDebugString,

    Mark has heard you. When Mark and I were discussing this feature that's what I wanted as well but there were two choices. 1) redesign a major portion of Process Explorer or 2) extend an existing API to get the feature done quickly. Shipping is a feature. :)


    If you've got a hung application pool, a debugger would be more appropriate than Process Monitor.

    Hope it helps!
    - John Robbins

  19. Thanks for Responding.

    Our App Pool deadlock condition only occurs on our production servers, not on our development boxes.

    Does your answer mean I do not have to attend your on line .NET Performance Tuning this Wednesday? Just kidding :)

  20. Pascal Renaudon

    I am using windows 7 x64; I recompiled the DLL and native test. I downloaded version 2.9. I set filter in procmon to nativetest.exe and click on show profiling events. Then I run NativeTest.exe and see "process profiling" in procmon. But NativeTest fails in CreateFile with a last error code of 2 meaning file not found. I really feel stupid to ask since so many others are using your DLL and it works fine but I dont see what i am missing


  21. Pascal,

    It's probably because you have an older version of the Process Monitor driver loaded. Reboot your machine and I bet everything works.

    - John Robbins

  22. pascal renaudon

    Thank you for your help. Now that I have rebooted everyting works fine. Your work will serve as basis for debugging deployment problems in our web app that calls into a native C++ DLL. We have had numerous issues related to file, security, DLL load. Now that we can have our traces AND procmon output finding these pb will be so much easier. Thank you again.


  23. Jonathan Edmonds

    This is pretty fantastic. I am finding value in this already as someone who already often used ProcMon. I wonder if, as a major benefit to folks using this for tracking down some performance issues, if there would be a way to easily show time deltas between two events. For example, if you highlighted two rows, if something in the status bar could show the time delta between the two rows automatically? Seems pretty trivial, and would save me a bunch of calc.exe entries!


  24. I can make modifications to the shared code and distribute? Does it comes under some open source licensing?

  25. John Robbins,
    We are using batch file to terminate procmon and generate csv file of procmon events.
    But How can I populate my ListBox with datasource as procmon?
    What is effective method to transfer data from Procmon to our application without terminating Procmon Process?

  26. I found this gem last night. However I'm unable to build it. The post build process is failing. Can you provide the necessary managed dll for X86 and x64. I'm a novice and not familiar with the post build process macros.

  27. What is the post build doing? Is it renaming and moving the dll's. Can I manully accomplish this task

  28. Great idea. Unfortunately, the link to the source code is currently 404-ing:

    I see the help file for Process Monitor shows some raw C source, but it would be nice to have your


  29. Great idea. Unfortunately, the link to the source code is currently 404-ing:

    I would like to implement this but cannot find a download for it, can you help?


  30. Everyone,

    I moved the code to GitHub: Hope it helps!

    - John Robbins

Leave a Comment