PDB Files: What Every Developer Must Know

80 Comments May 11, 2009


Thanks for visiting one of the most popular pieces I’ve ever written! Based on questions I’ve written a few follow up articles you might find useful to extend your PDB knowledge after reading this one.

Keeping Specific PDB Files from Loading in the Debugger
Do PDB Files Affect Performance?
Correctly Creating Native C++ Release Build PDBs
How Many Secrets do .NET PDB Files Really Contain?

If you’d love to know more about debugging, check out my Mastering .NET Debugging or Mastering Windows Debugging (for native C++) courses over at our great on-demand learning site WintellectNOW. Because you’re obviously a great developer for reading this, use the code JOHNR-2013 and get your first two weeks free! Are you more interested in instructor lead training? We can do that to so check out our course catalog.

If you have any questions about PDB files ask a question in the comments, email me (john AT wintellect DOT com) or ask me on Twitter @JohnWintellect.

Most developers realize that PDB files are something that help you debug, but that's about it. Don't feel bad if you don't know what's going on with PDB files because while there is documentation out there, it's scattered around and much of it is for compiler and debugger writers. While it's extremely cool and interesting to write compilers and debuggers, that's probably not your job.

What I want to do here is to put in one place what everyone doing development on a Microsoft operating system has to know when it comes to PDB files. This information also applies to both native and managed developers, though I will mention a trick specific to managed developers. I'll start by talking about PDB file storage as well as the contents. Since the debugger uses the PDB files, I'll discuss exactly how the debugger finds the right PDB file for your binary. Finally, I'll talk about how the debugger looks for the source files when debugging and show you a favorite trick related to how the debugger finds source code.

Before we jump in, I need to define two important terms. A build you do on your development machine is a private build. A build done on a build machine is a public build. This is an important distinction because debugging binaries you build locally is easy, it is always the public builds that cause problems.

The most important thing all developers need to know: PDB files are as important as source code! Yes, that's red and bold on purpose. I've been to countless companies to help them debug those bugs costing hundreds of thousands of dollars and nobody can find the PDB files for the build running on a production server. Without the matching PDB files you just made your debugging challenge nearly impossible. With a huge amount of effort, my fellow Wintellectuals and I can find the problems without the right PDB files, but it will save you a lot of money if you have the right PDB files in the first place.

As John Cunningham, the development manager for all things diagnostics on Visual Studio, said at the 2008 PDC, "Love, hold, and protect your PDBs." At a minimum, every development shop must set up a Symbol Server. I've written about Symbol Servers in MSDN Magazine and more extensively in my book, Debugging .NET 2.0 Applications. You can also read the Symbol Server documentation itself in the Debugging Tools for Windows help file. Look at those resources to learn more about the details. Briefly, a Symbol Server stores the PDBs and binaries for all your public builds. That way no matter what build someone reports a crash or problem, you have the exact matching PDB file for that public build the debugger can access. Both Visual Studio and WinDBG know how to access Symbol Servers and if the binary is from a public build, the debugger will get the matching PDB file automatically.

Most of you reading this will also need to do one preparatory step before putting your PDB files in the Symbol Server. That step is to run the Source Server tools across your public PDB files, which is called source indexing. The indexing embeds the version control commands to pull the exact source file used in that particular public build. Thus, when you are debugging that public build you never have to worry about finding the source file for that build. If you're a one or two person team, you can sometimes live without the Source Server step. For the rest of you, read my article in MSDN Magazine on Source Server to learn how to use it.

The rest of this entry will assume you have set up Symbol Server and Source Server indexing. One good piece of news for those of you who will be using TFS 2010, out of the box the Build server will have the build task for Source Indexing and Symbol Server copying as part of your build.

One complaint I've heard against setting up a Symbol Server from some teams is that their software is too big and complex. I have to admit that when I hear people say that it translates to me as "My team is dysfunctional." There's no way your software is bigger and more complex than everything Microsoft does. They source index and store every single build of all products they ship into a Symbol Server. That means everything from Windows, to Office, to SQL, to Games and everything in between is stored in one central location. My guess is that Building 34 in Redmond is nothing but SAN drives to hold all of those files and everyone in that building is there to support those SANs. It's so amazing to be able to debug anything inside Microsoft and you never have to worry about symbols or source (provided you have appropriate rights to that source tree).

With the key infrastructure discussion out of the way, let me turn to what's in a PDB and how the debugger finds them. The actual file format of a PDB file is a closely guarded secret but Microsoft provides APIs to return the data for debuggers. A native C++ PDB file contains quite a bit of information:

  • Public, private, and static function addresses
  • Global variable names and addresses
  • Parameter and local variable names and offsets where to find them on the stack
  • Type data consisting of class, structure, and data definitions
  • Frame Pointer Omission (FPO) data, which is the key to native stack walking on x86
  • Source file names and their lines

A .NET PDB only contains two pieces of information, the source file names and their lines and the local variable names. All the other information is already in the .NET metadata so there is no need to duplicate the same information in a PDB file.

When you load a module into the process address space, the debugger uses two pieces of information to find the matching PDB file. The first is obviously the name of the file. If you load ZZZ.DLL, the debugger looks for ZZZ.PDB. The extremely important part is how the debugger knows this is the exact matching PDB file for this binary. That's done through a GUID that's embedded in both the PDB file and the binary. If the GUID does not match, you certainly won't debug the module at the source code level.

The .NET compiler, and for native the linker, puts this GUID into the binary and PDB. Since the act of compiling creates this GUID, stop and think about this for a moment. If you have yesterday's build and did not save the PDB file will you ever be able to debug the binary again? No! This is why it is so critical to save your PDB files for every build. Because I know you're thinking it, I'll go ahead and answer the question already forming in your mind: no, there's no way to change the GUID.

However, you can look at the GUID value in your binary. Using a command line tool that comes with Visual Studio, DUMPBIN, you can list all the pieces of your Portable Executable (PE) files. To run DUMPBIN, open the Visual Studio 2008 Command Prompt from the Program's menu, as you will need the PATH environment variable set in order to find the DUMPBIN EXE. By the way, if you're interested in more about the information that DUMPBIN shows you, I highly recommend the definitive articles on the PE file by Matt Pietrek in the February 2002 and March 2002 issues of MSDN Magazine.

There are numerous command line options to DUMPBIN, but the one that shows us the build GUID is /HEADERS. The Pietrek articles will explain the output, but the important piece to us is the Debug Directories output:

Debug Directories
Time Type Size RVA Pointer
-------- ------ -------- -------- --------
4A03CA66 cv 4A 000025C4 7C4 Format: RSDS,
  {4B46C704-B6DE-44B2-B8F5-A200A7E541B0}, 1,
C:\junk\stuff\HelloWorld\obj\Debug\HelloWorld.pdb

With the knowledge of how the debugger determines the correctly matching PDB file, I want to talk about where the debugger looks for the PDB files. You can see all of this order loading yourself by looking at the Visual Studio Modules window, Symbol File column when debugging. The first place searched is the directory where the binary was loaded. If the PDB file is not there, the second place the debugger looks is the hard coded build directory embedded in the Debug Directories in the PE file. If you look at the above output, you see the full path C:\JUNK\STUFF\HELLOWORLD\OBJ\DEBUG\HELLOWORD.PDB. (The MSBUILD tasks for building .NET applications actually build to the OBJ\<CONFIG> directory and copy the output to DEBUG or RELEASE directory only on a successful build.) If the PDB file is not in the first two locations, and a Symbol Server is set up for the on the machine, the debugger looks in the Symbol Server cache directory. Finally, if the debugger does not find the PDB file in the Symbol Server cache directory, it looks in the Symbol Server itself. This search order is why your local builds and public build parts never conflict.

How the debugger searches for PDB files works just fine for nearly all the applications you'll develop. Where PDB file loading gets a little more interesting are those .NET applications that require you to put assemblies in the Global Assembly Cache (GAC). I'm specifically looking at you SharePoint and the cruelty you inflict on web parts, but there are others. For private builds on your local machine, life is easy because the debugger will find the PDB file in the build directory as I described above. The pain starts when you need to debug or test a private build on another machine.

On the other machine, what I've seen numerous developers do after using GACUTIL to put the assembly into the GAC is to open up a command window and dig around in C:\WINDOWS\ASSEMBLY\ to look for the physical location of the assembly on disk. While it is subject to change in the future, an assembly compiled for Any CPU is actually in a directory like the following:

C:\Windows\assembly\GAC_MSIL\Example\1.0.0.0__682bc775ff82796a

Example is the name of the assembly, 1.0.0.0 is the version number, and 682bc775ff82796a is the public key token value. Once you've deduced the actual directory, you can copy the PDB file to that directory and the debugger will load it.

If you're feeling a little queasy right now about digging through the GAC like this, you should, as it is unsupported and fragile. There's a better way that seems like almost no one knows about, DEVPATH. The idea is that you can set a couple of settings in .NET and it will add a directory you specify to the GAC so you just need to toss the assembly and it's PDB file into that directory so debugging is far easier. Only set up DEVPATH on development machines because any files stored in the specified directory are not version checked as they are in the real GAC.

By the way, if you search for DEVPATH in any internet search engine one of the top entries is an out of date blog entry by Suzanne Cook saying Microsoft was getting rid of DEVPATH. That is no longer true. As with any blog entry, look at the date on Suzanne's blog: 2003. That's the equivalent of 1670 in internet years.

To use DEVPATH, you will first create a directory that has read access rights for all accounts and at least write access for your development account. This directory can be anywhere on the machine. The second step is to set a system wide environment variable, DEVPATH whose value is the directory you created. The documentation on DEVPATH doesn't make this clear, but set the DEVPATH environment variable before you do the next step.

To tell the .NET runtime that you have DEVPATH set up requires you to add the following to your APP.CONFIG, WEB.CONFIG, or MACHINE.CONFIG as appropriate for your application:

<configuration>
   <runtime>
      <developmentMode developerInstallation="true"/>
   </runtime>
</configuration>

Once you turn on development mode, you'll know there's a problem with either the DEVPATH environment variable missing for the process or the path you set does not exist if your application dies at startup with a COMException with the error message saying the completely non-intuitive: "Invalid value for registry." Also, be extremely vigilant if you do want to use DEVPATH in MACHINE.CONFIG because every process on the machine is affected. Causing all .NET applications to fail on a machine won't win you many friends around the office.

The final item every developer needs to know about PDB files is how the source file information is stored in a PDB file. For public builds that have had source indexing tools run on them, the storage is the version control command to get that source file into the source cache you set. For private builds, what's stored is the full path to the source files that compiler used to make the binary. In other words, if you use a source file MYCODE.CPP in C:\FOO, what's embedded in the PDB file is C:\FOO\MYCODE.CPP. This is probably what you already suspected, but I just wanted to make it clear.

Ideally, all public builds are automatically being source indexed immediately and stored in your Symbol Server so if you don't have to even think any more about where the source code is. However, some teams don't do the source indexing across the PDB files until they have done smoke tests or other blessings to see if the build is good enough for others to use. That's a perfectly reasonable approach, but if you do have to debug the build before its source indexed, you had better pull that source code to the exact same drive and directory structure the build machine used or you may have some trouble debugging at the source code level. While both the Visual Studio debugger and WinDBG have options for setting the source search directories, I've found it hard to get right.

For smaller projects, it's no problem because there's always plenty of room for your source code. Where life is more difficult is on bigger projects. What are you going to do if you have 30 MB of source code and you have only 20 MB of disk space left on your C: drive? Wouldn't it be nice to have a way to control the path stored in the PDB file?

While we can't edit the PDB files, there's an easy trick to controlling the paths put inside the PDB files: SUBST.EXE. What SUBST does is associate a path with a drive letter. If you pull your source code down to C:\DEV and you execute "SUBST R: C:\DEV" the R: drive will now show at its top level the same files and directories if you typed "DIR C:\DEV." You'll also see the R: drive in Explorer as a new drive. You can also achieve the drive to path affect by mapping a drive to a shared directory in Explorer. I personally prefer the SUBST approach because it doesn't require any shares on the machine. While some of you are thinking that you can share through <DRIVE>$, some organizations disable that functionality.

What you'll do on the build machine is set a startup item that executes your particular SUBST command. When the build system account logs in, it will have the new drive letter available and that's where you'll do your builds. With complete control over the drive and root embedded in the PDB file, all you need to do to set up the source code on a test machine is to pull it down wherever you want and do a SUBST execution using the same drive letter the build machine used. Now there's no more thinking about source matching again in the debugger.

While not all of this information about PDB files I've discussed in this entry is entirely new, I didn't see it in one place before. I hope by getting it all together that you'll find it easier to deal with what's going on and debug your applications faster. Debugging faster means shipping faster so that's always high on the good things scale. Please ask any questions you may have on PDB files in comments, and I'll be happy to dig up the answers for you.


80 Comments

  • Gravatar Image
    Steve Munyan May 11, 2009 1:19 PM

    Thanks for a great blog post. I've had to relate similar information to customers on hundreds of occasions.

    I'm adding this link to my resources for new technical leads and engineering managers.

    Test early, test often, and follow engineering best practices.

    Steve Munyan

  • Gravatar Image
    PJ Gray May 11, 2009 1:22 PM

    Nice article!

    You probably left the gory details out on purpose, however, given that many companies (including the one I work for), didn't have correct and matching PDBs, I bet others would be interested:

    I asked this question at StackOverflow.com:
    http://stackoverflow.com/questions/744870/how-can-you-change-an-age-mismatched-pdb-to-match-properly

    And got some interesting responses. Including the mention of this online book:
    http://undocumented.rawol.com/

    That book has a great section on PDBs. All undocumented of course. But, sometimes you _KNOW_ the pdb is right, but the debugger refuses to let them match.

    Finally, I wrote a quick blog post a while back about my experiences with pdb matching here:
    http://www.saygoodnight.com/2009/04/fixing-mismatched-pdbs-to-match.html

    Like I said...this is a good article about the basics, however, there is plenty to dig into if you find yourself in a jam.

  • Gravatar Image
    Eric Hill May 11, 2009 3:15 PM

    Hey, John,

    Thanks for the deep dive on PDB files. You're exactly right, all the information you have provided probably exists somewhere, but not all together.

    I suspect the information you've provided explains a problem I had recently doing remote debugging. I copied my private build to another machine, with PDB file right there with it, but when I remote-debugged, VS could not find any of my breakpoints. Just on a hunch, I moved the executable files on the remote machine such the path was the same as where the files were on the build machine, and that fixed the problem. But that doesn't seem like a very reasonable requirement. What approach would you take to remove this restriction for remote debugging?

    Thanks!

    Eric

  • Gravatar Image
    DotNetShoutout May 12, 2009 7:14 PM

    Thank you for submitting this cool story - Trackback from DotNetShoutout

  • Gravatar Image
    Aneef May 13, 2009 2:10 AM

    Excellent Article :) thanks alot.

    Regards,
    Aneef

  • Gravatar Image
    Patrick Smacchia May 13, 2009 8:29 AM

    Thanks a lot John for these precious insider info.

    I have 2 related remarks:

    >The actual file format of a PDB file is a closely guarded secret but Microsoft provides APIs to return the data for debuggers.

    I know this is not your fault John, but frankly, do the world needs such an important format to be closed? And hopefully, this is no secret anymore since the OSS CCI project comes with a managed PDB reader code:
    http://ccimetadata.codeplex.com/
    Which immediately has been added to Mono.Cecil:
    http://www.mono-project.com/Cecil

    From my own usage, this managed reader is much much more faster than any native PDB API reader.


    Second remark: the tool NDepend analyzes .NET assemblies + related PDB + related C# code. NDepend reports many sync/versioning problem, included:

    -PDB files and code source files not in-sync.

    -PDB files and assemblies are not in-sync.

    -Missing PDB files (missing from the list of directories provided in the NDepend project).





  • Gravatar Image
    Andrew Smith May 13, 2009 9:02 AM

    What is the relation between source server, pdbs and the source code debugging provided within VS? Since there is a lag between the release of the source and the release of the assemblies and the fact that the source code debugging doesn't work if you obtain a QFE (e.g. through Windows Update) there must be some special handling going on. I'm asking partially out of curiousity but primarily because we are interested in using Symbol Server & Source Server but we supply source code to some customers and that source code is not exactly as it is in our source control (TFS) since we need to inject copyright info, etc. and we would like them to be able to use the source code with the assemblies we ship. Currently this works because we pre-process the source before we build so the source and pdbs match.

  • Gravatar Image
    Jonathan May 14, 2009 12:22 PM

    Rather than using SUBST (which our team almost did), we decided to go one step further and use "source indexing" which embeds the source control URL into the PDB, that way the debugger can pull the actual source code from from your repository (CVS, SVN, SourceSafe) as you step through your code.

    The other thing that we do is commit all binaries and associated PDBs to version control as well.

  • Gravatar Image
    jrobbins May 14, 2009 7:01 PM

    All,

    Sorry for the delay in answering questions, but I'm working like crazy in China.

    PJ Gray:
    Honestly, I would recommend fixing your build instead of going down the path of editing the PDB files. That would be much more sustainable in the end.

    Eric Hill:
    When I get home, I’m going to do a blog post on remote debugging and where the PDB files go to answer your question. In a nutshell, .NET PDB files are loaded on the remote machine and native PDB files are loaded on the local machine. I’ll go through all the iterations in my blog post to show where you have to put the different types of files.

    Patrick Smacchia:
    Doh! I forgot that the Common Compiler Infrastructure was released. You also have the DIA SDK (http://msdn.microsoft.com/en-us/library/x93ctkx8.aspx) you can use to read the PDB files. I should have been clearer, but I was trying to say that hacking the PDB file, as PJ Gray is doing. :), isn’t a good idea.

    Andrew Smith:
    You’re right, Microsoft is doing something interesting with the .NET Reference Source code in order to update the source *after* they’ve done the build. The Reference Source blog has mentioned that they have some internal tools that manipulate the PDBs. As for the lag time, I don’t know why that is, but I’m sure there are some requirements from Microsoft lawyers that they have to do before letting you and I access the code. What I would recommend is that you just keep doing what you’re doing because it’s working.

    Jonathan:
    I agree with you that the Source Server approach is *much* better and that’s why I recommended that you do it in the beginning of the blog entry. As for the SUBST approach, I just wanted to offer an option for those tiny teams or teams where their management won’t let them set up a Source Server.

    Great question and comments everyone, thanks!!
    - John Robbins

  • Gravatar Image
    Ajith May 15, 2009 1:38 PM

    This is a great post.Thank you for sharing this with us.

  • Gravatar Image
    ComponentGear.com Feed May 16, 2009 1:57 PM

    This week on Channel 9 at TechEd 2009, Brian is joined by Jeff Hadfield and Greg Duncan to discuss this

  • Gravatar Image
    gOODiDEA.NET May 17, 2009 9:47 AM

    Web The Evolution of a Website Design Twitter from ASP.NET IIS 7 Tip # 10 You can generate machine keys

  • Gravatar Image
    gOODiDEA May 17, 2009 9:48 AM

    WebTheEvolutionofaWebsiteDesignTwitterfromASP.NETIIS7Tip#10Youcangenera...

  • Gravatar Image
    CoqBlog May 17, 2009 10:26 AM

    Juste un relai d'un post de John Robbins (en anglais donc) à propos de ces fameux fichiers PDB, des Symbol

  • Gravatar Image
    EugeneZ blog May 19, 2009 12:12 AM

    Usually I try not to repost the links, but in this case just could not; I must recommend absolutely awesome

  • Gravatar Image
    Konstantinos May 22, 2009 9:33 PM

    Thanks for the information John,
    i have created a blog entry that links here.

  • Gravatar Image
    Sriram's WebLog Microsoft solution specialist on web platform, Columbus Ohio May 26, 2009 7:27 AM

    I did not know about the importance of PDB files until I read this article. John Robbins wrote about

  • Gravatar Image
    JC May 26, 2009 11:36 AM

    I've always questioned the need for these files. Thanks for the great article.

  • Gravatar Image
    Tom May 26, 2009 12:44 PM

    For nearly two years I've been searching the Internet for an answer to this question: is it possible to get the DBGHELP.DLL to /not/ load a PDB for a specific DLL? I have applications that have lots of third-party DLLs for which I have no PDB files (read: "drivers for a database company that shall remain nameless") and I'd like to tell the DBGHELP.DLL to avoid attempting to search for those PDBs. In fact, I have been investigating the possibility of making stub PDBs with no information and stuffing them in a private symbol server just to make symbol loading faster.

    Any thoughts?

  • Gravatar Image
    John Robbins' Blog May 26, 2009 8:23 PM

    Thanks for all the comments and emails about my blog entry, PDB Files: What Every Developer Must Know

  • Gravatar Image
    Notes from a dark corner May 27, 2009 8:58 AM

    I don’t think I’ve ever come across anything written by John Robbins that is not worth reading. But in

  • Gravatar Image
    John Robbins' Blog May 29, 2009 1:31 AM

    As I've discussed , PDB files are wondrous bundles of binary joy. However, loading missing PDB files

  • Gravatar Image
    alik levin June 3, 2009 6:30 AM

    Thank you!
    Very thoughtful and practical post

  • Gravatar Image
    onDevelopment =1; June 4, 2009 5:12 PM

    &lt;p&gt;&amp;#160;&lt;/p&gt; ...

  • Gravatar Image
    onDevelopment =1; June 4, 2009 5:18 PM

    &lt;p&gt;&amp;#160;First things first: If you think .pdb files are only used when debugging your application locally and are not needed when deploying production binaries, please read the following ar ...

  • Gravatar Image
    Rick Byers June 9, 2009 1:59 PM

    I'm just finishing up my Masters in Computer Science, and was surprised when I recently got an assignment

  • Gravatar Image
    marc ochsenmeier June 22, 2009 4:20 AM

    Hi John,

    your excellent blogs regarding PDB made me very curious...

    I decided to test what kind of information DIA is providing...and published a part of my work at code project (
    http://www.codeproject.com/KB/bugs/PdbParser.aspx)

    Regards,
    Marc Ochsenmeier
    www.breakpoints.de

  • Gravatar Image
    jrobbins June 22, 2009 12:26 PM

    Marc,

    Cool! Thanks for the link and work wrapping DIA. I've looked hard at DIA before, but couldn't really stomache how ugly that API is to use. Your wrapper was much needed!

    -John Robbins

  • Gravatar Image
    Kuenga June 24, 2009 12:37 AM

    Nice Article...I often loose track of storing the PDBs..
    Is there any way to debug a crash dump if we donot have matching PDB, since VS2005 complains as No Matching Symbols found..

    I find hard to maintain PDB and keep track for various hotfix binaries and there matching PDB..

    If we receive a crash dump from customer and we donot have matching PDB, is there a way to get the stack trace of the crash ?

  • Gravatar Image
    jrobbins June 24, 2009 1:49 AM

    Kuenga,

    You need to set up a Symbol Server. That's the only way you will be able to track and match your PDB files. See the links in the article for how to set one up.

    Without a Symbol Server you are screwed and will never get a call stack from your crash dumps.

    - John Robbins

  • Gravatar Image
    Warren Tang June 24, 2009 3:52 AM

    .PDBfiles(ProgramDebuggingDatabase) PDBFiles:WhatEveryDeveloperMustKnowhttp://www.win...

  • Gravatar Image
    marc ochsenmeier July 6, 2009 10:54 AM

    Hi John,

    just posted another article at www.codeproject.com/KB/DLL/Symbols_File_Locator.aspx
    about DIA, this time focused on the executable and locating the associated PDB file.

    Best regards,
    Marc Ochsenmeier

  • Gravatar Image
    Web Design and Development India July 11, 2009 2:15 AM

    Thanks for this nice post...

  • Gravatar Image
    Nachiket07 August 6, 2009 6:54 PM

    Nice article John. Do you know why I cannot see source file and line numbers when I debug managed crash dump is Windbg. I am using SOS.dll extension and pointing to correct debug symbols. All I see is modules and function names.

  • Gravatar Image
    David K. August 13, 2009 10:21 AM

    Sorry for the length of this post :-(

    I wish you had written this superb Blog earlier and that I had read it earlier. I have now read your other article on the Source Server and the documentation included in Microsoft’s latest Debugging Tools for Windows.

    However, I would be grateful for your view on our Project environment, which must be mirrored by so many others and how we can take advantage of the technology you have described.

    I am part of a small group of developers working on a particular Open Source application. It is a native C++ (MFC) dialog application built and maintained using Visual Studio 2005. For all of us, it is a hobby and not our ‘day job’. Our project is held on SourceForge and the source within a Subversion repository on that site. Whilst we can call commit changes to the source in that repository, only the project Administrator creates a ‘formal’ release and uploads the installer/executable/zip files (including the source at that revision, so that interested users may download it without having to check it out of the repository) to the SourceForge web site for users to download.

    The Administrator, like most of us, does the development (private) builds and also the release (public) builds on his home PC used for this and just normal general home PC usage. None of us (I think) have the luxury of having separate machines for private and public builds. The only thing defining a public build is the source revision number and the release configuration within Visual Studio. He does save the PDB files associated with every public build but doesn’t (yet) have a Source or Symbol Server and doesn’t use source indexing.

    Even if he set up a Source and Symbol Server, we are all geographically dispersed around the globe. I don’t think there are even two in the same country. So, even if he did set these up, the other developers would not be able to access it. We are therefore in a position that only he can effectively use a user’s minidump to analyse a problem! As you can imagine, he is not over enamoured by this!

    Can you suggest a way that all developers can use a Source and Symbol Server so that the load can be shared? He can send us the PDB files associated with any formal release and we can get the corresponding executable from the project web site but is there a type of export/import function so that we can have an effective copy of his Source and Symbol servers so that any developer can work on any problem?

    Lastly, as simple home users, we all probably have just the Standard Edition of Visual Studio 2005 and your earlier article in May 2006 entitled “Source Server Helps You Kill Bugs Dead In Visual Studio 2005” says that “only the Professional and Team editions know about Source Server”. Does this mean I should give up trying to use this approach completely and leave it to the Administrator to debug all minidumps?

    Any other thoughts on this would be gratefully received.

    PS. A link to an idiot’s guide to setting up both Source Servers and Symbol Servers for non-professional developers in our situation using Subversion would be gratefully appreciated.

  • Gravatar Image
    John Robbins' Blog August 22, 2009 8:34 PM

    Now that Devscovery is over, it's time to get back to answering questions about one of my favorite subjects:

  • Gravatar Image
    Emilio August 23, 2009 5:02 PM

    Quick note on PDB's and Outlook 2007 AddIn development w/ VS 2008.

    For those attempting the C++ add-in shim that calls the C# managed code addin and wish to debug the managed code on a target production install (not the dev machine), here's a pointer:

    Create a setup.exe project that installs the debug complied version of your addin. Install it on the target and copy the PDB file into the directory where the managed code DLL resides. Install VS 2008 on the target machine and put the C# file you wish to step through in the EXACT same directory structure found on the dev machine.

    Last, but not least (took me 3 days to figure out), when you run Outlook and "Attach to Process" through VS, make sure to change the field labeled "Attach to:". It defaults to "Automatically determine the type of code to debug". Change it to "Managed".

    All of a sudden, the addin that would never stop no matter how many breakpoints you set, starts behaving as desired!

    Thanks and good nite!

  • Gravatar Image
    jrobbins August 24, 2009 12:16 AM

    Emilio,

    Thanks for sharing the steps! That will help a lot of people.

    - John Robbins

  • Gravatar Image
    jrobbins August 28, 2009 5:37 PM

    David K,

    I'm very sorry for the delay in getting back to you, but I've been working like crazy.

    Because of the nature of your open source project, I think getting a Symbol and Source server set up would be a pain. You could set up a server using IIS and use a HTTP Symbol Store (http://msdn.microsoft.com/en-us/library/cc901414.aspx), but that would take money and as I pointed out in several articles, at least everyone using Visual Studio Professional Edition or higher.

    Having only your project administrator debugging minidumps certainly won't scale. Now if you could convince him to give up his day job and do that full time, that's an acceptable solution. :) Here's what I think would work for your project:

    1. Continue to build all public builds with full PDB files.
    2. For the nightly builds, include that build's PDB files in the package. As those are more likely to be used by fellow contributors, they would only have to build the parts they need and wouldn't have to worry about building the whole package.
    3. For milestone releases, which are mostly used by non developers, add that build's PDB files as a separate download for the release. That way if anyone reports a minidump, and developer on the team can grab the PDBs for that release and debug the minidump.

    Hope it helps!
    -John Robbins

  • Gravatar Image
    John Robbins' Blog August 31, 2009 6:55 PM

    In my ongoing discussion of PDB files and debugging, Sa Li had a great question : Should the final release

  • Gravatar Image
    Akhilesh September 3, 2009 9:24 AM

    Nice Post and is very helpful.

  • Gravatar Image
    i lold September 15, 2009 6:12 AM

    "What are you going to do if you have 30 MB of source code and you have only 20 MB of disk space left on your C: drive?"

    i lold

  • Gravatar Image
    Carl Appellof October 8, 2009 11:26 AM

    Let's say I'm debugging some code in a module that doesn't have a pdb file. I've laboriously figured out the addresses of some functions like strlen, strcat, etc. within the code. Right now I'm using a pencil and paper to record that info.

    Is there any way I can generate a .pdb file that will at least give me these symbolic names when I debug? Perhaps compiling the right sort of C++ program?

  • Gravatar Image
    EricJ October 14, 2009 12:53 PM

    Regarding symbol server for DavidK's project, an approach I often use is to create a symbol store with pointers. Remember a symbol store is just a directory structure. By storing pointers, you can periodically save space by deleting old builds without having to clean up the symbol store.

    In one former project the drop directories would move around and the private dev team daily build would get deleted for space reasons. I often had to look at minidumps from builds spanning several years, so I had a script to wake up at 2am and scan big directories and insert a compressed copy into my local symbol store (a directory tree).

    if not exist \SymStoreForDailyBuilds md \SymStoreForDailyBuilds
    c:\Debuggers\symstore.exe add /r /l /p /s \SymStoreForDailyBuildsSymStoreForDailyBuilds /t . /f c:\dailydrop\091012\*.pdb

    /r = recurse
    /p = store pointers rather than the files
    /l = allow local paths when storing pointers (references)
    /s = symbol store location
    /t = product name (required)
    /f = file

  • Gravatar Image
    jrobbins October 14, 2009 4:07 PM

    Eric,

    Interesting idea. Yes, in an environment like yours where space is a premium or you just have gigantic numbers of symbols, ptr files really help.

    In David's case it still might not help because of the distributed manner of the project. However, anything could help!

    Thanks!
    - John Robbins

  • Gravatar Image
    John Robbins' Blog October 19, 2009 12:24 PM

    I love all things diagnostics. After installing VS 2010 Beta 2, the first thing I had to look at my favorite

  • Gravatar Image
    ricko January 20, 2010 6:23 PM

    John, very nice article. I've been to devscovery a couple of times and enjoyed one of the talks about debugging, in which the speaker (you or one of your cohorts...can't remember exactly who it was), layed out the exact process that you guys use when you get into a client site where you are not allowed access to the src code, but, nonetheless, are expected to debug the code. The whole process was pretty straight forward. In fact, it seemed so straight forward that I did not take detailed notes...silly me.

    With that in mind, I got a kick out of your statement, "I've been to countless companies to help them debug those bugs costing hundreds of thousands of dollars and nobody can find the PDB files for the build running on a production server."

    So, John, how about laying out the process of which tools and procedures to use to get access to the source code when we are faced with a similar situation. I remember that it involved (heavily) the Reflector tool, but it also involved one of the Microsoft debugging tools.

    Does this ring any bells for you?...ricko

  • Gravatar Image
    Martin January 28, 2010 2:16 PM

    John,
    Thanks for this article. Whew! I am pulling my hair out over these pdb files. I am building dll's. In VS2008, which I am using, there are two GUI fields for pdb files. One for compiling and one for linking. According to dumpbin /headers the one under linking is being used as it is in the final dll. Should the other pdb be renamed and installed along side of my dll for debugging programs that need to test the dll? I just can't find any explanation as to why there are two pdb specs.

    Thanks

  • Gravatar Image
    Dhananjay Goyani February 3, 2010 7:04 AM

    This may be out of track however it would be great if you comment on this... Part that says debugging through reflected/decompiled code base are now possible through Reflector Pro.

    If that is possible, do we still require debug symbols? I haven't gone through exact details and not used the pro version yet but liked the idea very much. It seems to be revolutionary. What do you say?

    http://www.infoq.com/news/2010/02/ReflectorPro

    http://www.simple-talk.com/community/blogs/alex/archive/2010/01/20/87639.aspx

  • Gravatar Image
    jrobbins March 11, 2010 12:11 AM

    Ricko,

    When we don't get access to the source, it's pure reverse engineering. With .NET that's not to bad because we have Reflector, and now the amazing Reflector Pro. For native code, it's translating assembly language. No magic on that one, just huge amounts of brute force effort. :)

    Martin,

    The important field is the linker field and the name of the PDB file that's the one that matters. Ignore the compiler setting (leave it to the default). The compiler one is a temporary PDB file normally called something like VC90.PDB and can be ignored.

    Dhananjay,

    Reflector Pro is GREAT! However, it does not eliminate the need for having the source and PDB because it's always easier to figure out what's going on when you have the source and COMMENTS as to why someone designed the code the way they did.

    - John Robbins

  • Gravatar Image
    Sandeep Aparajit March 16, 2010 11:51 AM

    Excellent article!

  • Gravatar Image
    tmb June 1, 2010 7:24 PM

    Hey John - thanks for the great article!

    Just to clarify, it seems like your article mentions a pdb file per module. If you have an assembly made up of multiple modules, would you expect multiple pdb files?

    thanks!

  • Gravatar Image
    marc ochsenmeier September 30, 2010 7:16 AM

    Here a link (www.winitor.net/en/pestudio.html) to a tool that retrieves the GUID, pdb path and pdb file age (when available) for any (windows) application. Using this information, you can compare it with the GUID of the PDB and test whether they match.Thanks for this blog!

  • Gravatar Image
    Harry October 11, 2010 8:34 PM

    That was a helpful post, nice to see all this stuff in one place. Now here's a question, we have a special application server, that can load and execute user written code modules. These are created using a special tool we have created, the tool is able to compile C# code and "publish" the .DLL and .PDB and .CS data to a persistent medium (not the file system).

    Network requests to the server, cause it to locate and load the DLL and run methods within, results being sent back over the network.

    This is working well, but we need to support the ability for customers to debug their code modules. So we want to load the DLL and PDB together (using the existing Assembly.Load() overload).

    However, although that will probably load OK, I think the source code must always be a physical file on disk (the DLL and PDB are (in effect) loaded from MemoryStreams).

    So assuming the app server has loaded the DLL and PDB succesfully, what is the best way to make the source available to the debugger?

    We can copy the src stream to a temp disk file, but I am unsure of where to put it and what options we have for controlling this apsect of things.

    Any info much appreciated!

    Harry.

  • Gravatar Image
    Some random reader October 20, 2010 11:38 AM

    Hi there,

    While googling around, I just found out that someone ripped your work:

    http://memetic-thoughts.blogspot.com/2009/07/pdb-files.html

  • Gravatar Image
    Dave Black January 6, 2011 5:26 PM

    Hi John,

    I feel like I missed something....I know how to find the GUID in the binary, but how can you find the GUID in the .pdb?

  • Gravatar Image
    ram February 11, 2011 9:10 AM

    Hi John,

    i am working in our product , basically which is a MFC application,
    i don't have the pdb file for some MFC DLL which is loaded by my application exe. i have the following question. but i have all the source code with me.

    1. is it possible to break in to a specific place , ( like i am setting breakpoint in a source line number )

    2. can i see the vector element values in watch window ..


  • Gravatar Image
    Noury June 14, 2011 4:03 PM

    Hi,

    Great post !

    Does the logic explained above apply to mixed .dll as it apply to native and managed .dll ?

    I tried to debug a crash .dmp file (I created) with mixed debugging since the assembly in question is a mixed one but Visual Studio 2010 does not find the source files (.pdb/.dll were source server indexed --> source code stored in subversion).

    If i debug the crash .dmp in native mode only, the source file get fetched ok from source control (subversion) as expected. I get prompted if i want to accept the command execution that will fetch the source file.

    For some reason this does not work while debugging in mixed mode.

    Thanks for any feedback


    Does debugging a .dmp work created from a mixed .dll work with .pdb and source server ?

  • Gravatar Image
    jrobbins June 14, 2011 6:31 PM

    Dave Black,

    You can find the GUID in the PDB, but if you try to edit it the debugger will crash 80% of the time. You need to treat PDB files as a super black box.

    Ram,

    Nope. If you don't have the PDB for the DLL, you won't be able to break nor look at locals. See the whole article as to why.

    Noury,

    You should be able to find source in a symbol server with mixed debugging. I bet you don't have either _NT_SYMBOL_PATH set or you are missing the settings in VS.

    - John Robbins

  • Gravatar Image
    Steve June 15, 2011 1:27 AM

    Just wondering how remote debugging on an ASP.NET Web Application varies with the instructions/descriptions you've provided above.

    I'm using VS2010 to "Attach to process..." on my remote server to the w3wp.exe process. VS2010 then shows all my breakpoints up as hollow red circles, saying that the symbols aren't loaded :-(

    When I look at the Module debug window, it lists a range of DLLs that are being loaded by my Web App, but not the actual DLL of the Web App itself.

    So, my questions are, should I expect to see the DLL of the Web App in this list, and if so, where should the PDB file be located to make sure the symbols get loaded.

    Thanks. This is by far the best PDB summary I've seen on the web.

  • Gravatar Image
    Noury June 15, 2011 8:31 AM

    Thanks for your prompt response. I have set the symbols server path in VS (Options->Debugging -> Symbols -> Symbol files (.pdb):--> my networkPath\mySymoblesStore).

    The symbols are found and source files are downloaded ok when I debug using 'Debug with Native only'. But if debug the same .dmp using 'Debug with Mixed' the source files are not downloaded even if the symbols are copied from the symbols server to the local symbols cache. I get ask to locate the source files. Its odd.

    Through the 'Output" Window I can see that symbols are copied and loaded:

    'MAP9B79.tmp.dmp': Loaded 'C:\Users\myUserName\AppData\Local\Temp\SymbolCache\mymixedassembly.dll\4df7abbe758000\mymixedassembly.dll', Symbols loaded.

    Then:
    'MAP9B79.tmp.dmp' (Managed): Loaded 'D:\MyOriginalPath\release\mymixedassembly.dll', Symbols loaded.

    Then near the end of the output window:
    SRCSRV: The module 'D:\MyOriginalPath\release\mymixedassembly.dll' does not contain source server information.

    Note: 'D:\MyOriginalPath\release\' does not exist on my machine its the original path of the mixed assembly when it was built on a different machine.

    There is probably some thing different about debugging in mixed mode and source server...

    Thanks

  • Gravatar Image
    jrobbins June 15, 2011 3:04 PM

    Steve,

    Remote debugging of .NET apps means the PDB files are loaded on the REMOTE MACHINE. See my blog article here for what's going on: http://www.wintellect.com/CS/blogs/jrobbins/archive/2009/05/26/visual-studio-remote-debugging-and-pdb-files.aspx

    Noury,

    When you see that SRCSRV warning, the debugger is telling you that it can't find the PDBSRC section in the PDB file. Are you sure that PDB file has been source indexed? Use the WinDBG PDBSTR.EXE program to dump out the pdbsrc section. If you don't see the text stuff for the indexing, that's your issue.

    Hope it helps!
    - John Robbins

  • Gravatar Image
    John Robbins' Blog June 28, 2011 12:14 AM

    While you should always have your PDB files in symbol servers, there are times when you’re doing private

  • Gravatar Image
    mgrzeg.net - Admin on Rails :) July 25, 2011 2:36 PM

    Każdy, kto choć raz skorzystał z WinDbg do analizy zrzutu po BSOD, albo po prostu debuggował coś, co

  • Gravatar Image
    mgrzeg.net - Admin on Rails :) July 25, 2011 2:56 PM

    Każdy, kto choć raz skorzystał z WinDbg do analizy zrzutu po BSOD, albo po prostu debuggował coś, co

  • Gravatar Image
    Jiho Han November 14, 2011 1:43 PM

    Regarding Source Server setup, it seems like with source indexing, you're tied down to the current version control system as the version control command is what's embedded in the PDB. Should you change your version control system in the future, you'd have to keep the old VCS running in parallel in order for the PDB's that were generated previously to find the source.

    Is that correct?

  • Gravatar Image
    Jiho Han November 14, 2011 1:46 PM

    Do you have any recommendation regarding whether or not to keep the .NET PDBs on production web servers? Is there a security risk? I am not particularly worried about source stealing aspect of it, however.

  • Gravatar Image
    xavierdecoster.com November 16, 2011 11:09 AM

    Why everyone should be using a symbol server

  • Gravatar Image
    Vaibhav December 6, 2011 7:26 AM

    Hi, This was a nice read, can you tell me how to aviod full path information in c# dlls? Is there a way to do it?

  • Gravatar Image
    Venkatesh December 12, 2011 11:04 AM

    I have the same question as Jiho Han.

    John, as you said in your article, the managed pdb has much less informaiton than an unmanaged pdb. Besides, you can get to the source code of a .Net binary using a reflector. So does it really pose a security risk to deploy release-build pdb files on the production server?

  • Gravatar Image
    Noury May 14, 2012 8:40 AM

    Just a followup on debugging managed minidump as I did have an issue (see above) with source server not fetching the source code. When I debug as native only i get the source code but not when debugging in mixed mode.

    I turns out that this was indeed a 'limitation' in Visual Studio.
    One work around is described here at:

    http://www.wintellect.com/CS/blogs/jrobbins/archive/2012/01/06/want-source-server-to-work-with-managed-minidumps-in-visual-studio-here-s-how.aspx

    or in the original post at:

    http://smilingbaldguy.wordpress.com/2012/01/06/managed-minidump-debugging-now-with-source-server-support/

    Noury

  • Gravatar Image
    ruju July 4, 2012 10:08 AM

    gr8 article!! very useful.

  • Gravatar Image
    KS August 27, 2012 9:07 PM

    Thanks for the explanation. Takes some of the mystery out of using PDB files and VC++ debugging.

  • Gravatar Image
    John Robbins' Blog September 28, 2012 2:19 AM

    Conservation is always good so I’m going to recycle some keystrokes. Today I got an email where a company

  • Gravatar Image
    Marian Ionita November 27, 2012 10:03 AM

    The links for Matt Pietrek's articles in the MSDN Magazine are dead. Could you please fix them? Thanks!

  • Gravatar Image
    jrobbins November 27, 2012 2:09 PM

    Marian,

    Thanks. Fixed!

    - John Robbins

  • Gravatar Image
    Thomas February 20, 2013 9:08 AM

    Would you mind elaborating a bit on using the subst trick with the PDBs? Thanks.

  • Gravatar Image
    Jones sabo amigo internet microscopic lense is ide April 22, 2013 11:49 AM

    I think this site contains some real good info for everyone. "Je veux que les paysans mettent la poule au pot tous les dimanches." by King Henry IV of France.

  • Gravatar Image
    replica louis vuitton April 24, 2013 8:35 AM

    Lots of people

  • Gravatar Image
    rm May 13, 2014 2:23 PM

    Great, really helpful and concrete. Thanks.

Have a Comment?

Archives

Blogs