John Robbins' Blog

Download All the .NET Reference Source Code at Once with Net Mass Downloader

Microsoft has really helped out the development community by making the .NET Reference Source Code available to the community. Being able to step in the debugger from your code to the Framework code truly helps with you understanding and use of all parts of .NET. The only drawback I've seen with what Microsoft is offering is that the source code can only be downloaded while you're debugging. This means that unless you write a sample program that calls literally every class and method in the .NET Framework, you won't get all the source. As Scott Hanselman has so excellently pointed out, reading code is as important, if not more, than writing it. What I really wanted was a way to download all the .NET Reference Source so I could just start reading everything. Additionally, I wanted to fill my download cache with all the source so I had it all even when I didn't have a network connection.

It turns out I wasn't the only one who had the same wishes. Kerem Kusmezer contacted me and said that he'd been hacking around with Fiddler and figured out how to download the PDBs and code from Microsoft. He asked if I I'd be interested in helping him develop the utility and I don't think I ever wrote a response faster in my life! Kerem had already started a discussion with Shawn Burke about the legality of producing a source downloader and once we got back a very encouraging "go for it" we got to work. All the credit for the tool belongs to Kerem. He did all the really hard work of figuring out the HTTP downloads ,the PE file processing, and PDB Stream Extraction. I did the console driver, a lot of testing, and sat around amazed at how much work Kerem could accomplish in very short periods of time. I really appreciate that Kerem let me work on the project with him and being his code monkey.

The tool is called Net Mass Downloader and we made the source code available at CodePlex at http://www.codeplex.com/NetMassDownloader so you can see how it works. It's a command line tool and running the tool without parameters produces the following output:

.Net Mass Downloader 1.0.0.0 - (c) 2008 by Kerem Kusmezer, John Robbins

Batch download the Microsoft .NET Reference Source code.

Usage: NetMassDownloader [-file <file>]
                         [-directory <directory]
                         [-output <directory>]
                         [-vsver <version>]
                         [-force] [-nologo] [-verbose] [-?]

    -file      - Download an individual file's PDB and source code. You can
                 specify multiple file parameters. (Short -f).
    -directory - Download all the found PDB and source code for all files in
                 the specified directory. You can specify multiple
                 directory parameters (Short -d).
    -output    - The output directory for PDB and source files. The default
                 directory is the cache directory set in Visual Studio 2008.
                 By using the cache directory, you'll have the PDB and source
                 files available to Visual Studio 2008. However, to use the
                 .NET Reference Source Code with VS 2005, use the -output
                 switch and in the Options dialog, Debugging, Symbols property
                 page, add the specified output directory to the "Symbol file
                 (.pdb) locations." Also, add the directory to the Solution
                 Properties, Common Properties, Debug Source Files, Directories
                 containing source code location. The Visual Studio 2005
                 debugger will automatically load the source code. (Short -o)
    -vsver     - The Visual Studio version number to use for finding the cache
                 directory. The default is Visual Studio 2008,
                 but if you want to use the cache directory for Visual Studio
                 2005, you would pass '-vsver 8.0' (without quotes) (Short -vs)
    -force     - If specified, forces the downloading the PDB files into the
                 symbol server. When downloading to a symbol server if the PDB
                 exists, it's not downloaded. Using the -output switch will
                 always download and process the PDB. (Short -fo)
    -nologo    - Don't show the logo information. (Short -n)
    -verbose   - Do verbose output. May be worth turning on as the downloading
                 source code can take a long time. (Short -v)
     -?        - This help message..

The only required arguments are –file or –directory, both of which can be specified as many times as you'd like. When you specify a directory, only the .DLL and .EXE files from that directory will be processed. If you wanted to download all the source code from binaries in the .NET 2.0 32-bit and 64-bit directories, the command line you'd pass is: -d C:\Windows\Microsoft.NET\Framework\v2.0.50727 –d C:\Windows\Microsoft.NET\Framework64\v2.0.50727.

The main purpose of Net Mass Downloader is to populate the source code download cache for debugging, the default download location is the cache you specified to Visual Studio 2008. We put in the –vsver switch to account for future Visual Studio versions so Mass Downloader could work with future CTPs and versions.

While it's great to see the .NET Reference Source Code in Visual Studio 2008, there are a lot of developers out there who can't upgrade yet. Initially, Kerem and I went down the path of seeing if we could patch the PDB files so Visual Studio 2005 users could have access to the .NET Reference Source Code. After a lot of hassle, we realized that there was a much easier way. If you specify the –output switch, we will put the PDB and the source code into that directory using the build paths instead of the source server paths. Thus, if your command line to NetMassDownloader.exe is –d C:\Windows\Microsoft.NET\Framework\v2.0.50727 –output c:\ReferenceSource, the PDB files are put in C:\ReferenceSource as well as the source files.

To use the .NET Reference Source with Visual Studio 2005, you would need to do the following three additional steps. The first is in the Options dialog for Visual Studio, Debugging, Symbols property page. In the "Symbol file (.pdb) locations" list box, you would add "c:\ReferenceSource" like the following.

The second setting is in Options dialog, Debugging, General property page, uncheck Require source files to exactly match the original version.

The final step you'll have to do in each solution where you want to access the downloaded .NET Reference Source Code, you'll go into the solution property pages, Common Properties, Debugging Source Files, and add C:\ReferenceSources to the top of the "Directories containing source code" list box as follows:

Visual Studio 2005 will automatically do the right thing and replace the first directory of every source file it finds in a PDB file with c:\ReferenceSource. With the automation model in Visual Studio 2005, I'm sure you could wipe up a macro to automatically set that path in the solution.

When you first run Net Mass Downloader, you will be prompted with the current EULA for accessing the source code. If you don't agree with the Microsoft EULA, clicking the Decline button will not download the source code. While you might be tempted to remove the EULA check, I would strongly recommend that you not do so as Microsoft has said that they may change the EULA at any time. The last thing you want to do is get caught with a license change that could impact your life.

One group we need to thank is the Developer Division at Microsoft. First they released the .NET Reference Source Code, and second for allowing a couple of developers to have some fun and provide a utility for the community.

As always let us know on CodePlex if you encounter any bugs or have feature requests.

On Feb 5 2008 4:55 PMBy jrobbins With 71 Comments

Comments (71)

  1. Hi,

    I'm having some problems - the SymbolCacheDir registry value is empty what exactly should be in that location :).

    I put a directory in there and it didn't complain anymore. But I still get a
    "File not found on symbol server". Any tips ?

  2. Another drawback is that only a few files are available and the debugger feature is only available in non-Express edition of Visual Studio 2008.

    Compared to X11 license that covers Novell Mono, MS-RL is not a convenient license for end users. Microsoft .NET should go true open source some day in order to promote .NET to a higher level like SUN GPLed Java.

  3. Sirrococo - See Shawn's blog entry on which symbol files are actually downloadable. Only a portion of the .NET Framework code is available.

    Lex Y. Li - It's Microsoft's code. They can do whatever they want with the code and license it however they see fit.

    - John Robbins

  4. When I run the downloader with the -d option and point it to various framework directories, I get a 'No matching DB file on symbol server for xxx' message for every .dll in the directory, and nothing downloads. I am able to successfully download source and PDB in VS2008 while debugging. Has anyone gotten this to work? Thx in advance.

  5. I'm having the same problem as Joe Modica, every file reports it can't be found on the symbol server. I'm kind of wondering if the app is not making a connection to the server (server slammed by people running this tool perhaps?) but I haven't downloaded the source to debug this yet.

  6. I should have mentioned, thanks to both of you! Once I get this working it will be a tremendous boon... we'll be using VS2005 for a while yet, and I've had several debug sessions that would have been minutes rather than hours if I'd had the .net source!

  7. Joe Modica & Squidbot,

    Have you installed the .NET 3.5 Framework? (http://www.microsoft.com/downloads/details.aspx?FamilyID=333325fd-ae52-4e35-b531-508d977d32a6&displaylang=en). You probably don't have the matching .NET Framework version on your machines. We shoud have made that requirement clearer. You can install the .NET 3.5 Framework and VS 2005 will work just fine.

    Let us know how it goes!

    -John Robbins

  8. Hello,

    I'm having a problem: all the .cs files are the same size and contain apparently the same contents (some kind of EULA)!?!

    What am I doing wrong?

    Thanks.

  9. Florin,

    Weird! Did you install the .NET Framework 3.5? Any more you can say about your machine/configuration? We saw lots of strange things hacking the download, but never that problem!

    -John Robbins

  10. Kerem Kusmezer

    Hi Florin, it seems to be a problem in your computer that causes our framework not to detect the eula requests made for your code.
    Just get the version again, and try again, if problem persists enter as a bug with your computers regional settings, computer ram etc. in our issue tracker.

  11. I used a batch file to get everything like this:

    NetMassDownloader.exe -output "C:\Dev\Microsoft.Net\v2.0" -d "C:\Windows\Microsoft.Net\Framework\v2.0.50727"
    NetMassDownloader.exe -output "C:\Dev\Microsoft.Net\v3.0" -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5"
    NetMassDownloader.exe -output "C:\Dev\Microsoft.Net\v3.5" -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5"

  12. It is about 210 mb with pdb files included.

    The NetMassDownloader also accepts multiple -d in single line no need for batch :)

    NetMassDownloader.exe -output "C:\Dev\Microsoft.Net\v2.0"
    -d "C:\Windows\Microsoft.Net\Framework\v2.0.50727"
    -d "C:\Dev\Microsoft.Net\v3.0" -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5"
    -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5"

    This is also legal.

  13. Hello John,

    I have .NET Framework 3.5 installed by VS 2008. My regional settings are set on Romanian, but I've tried also when set on English(United States). The same results. I use a proxy server to access the symbols server (that might be the issue?!?).

    Thanks for your help.

  14. Florin,

    The proxy server is probably the issue. Neither Kerem or I tested against a proxy server. You can grab the code on CodePlex and see if you can make it work in your environment. If so, please let us know how it goes!

    - John Robbins

  15. Happy news! :D

    Apparently when using proxy, the status description for the EULA response is not "End User License Agreement", but "Unknown". Changed that in PDBWebClient and I finally got the EULA accepting dialog and then the source code.

    Florin Sabau.

  16. .NET Mass Downloader 从软件的名称就可以看出它的特点了:如果把Mass看成一个单词,则软件的名称可以翻译为“团块下载器

  17. 更新:程序员的思维模式都差不多,KeremKusmezer和JohnRobbin发布了一个类似的工具.NETMassDownloader。既然有的用我就不重新发明轮子了,下面那个项目停止...

  18. is there anyway we can get a md5 or sha256 of all the files given the command line everyone is running:

    netmassdownloader -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5"
    -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0"
    -d "c:\Windows\Microsoft.net\framework\v2.0.50727" -output c:\dev\NetFXSource


    every once in a while in the command window output i get a line:
    "The remote server returned an error: (404) Not Found."

  19. The final output when "complete" after running:
    netmassdownloader -d "c:\Windows\Microsoft.net\framework\v2.0.50727" -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0" -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5" -output c:\dev\NetFXSource

    Processed binaries/PDBs : 21
    Files not processed : 132
    Downloaded source files : 7174


    how about everyone else?

  20. i can't download file

    message:

    File ont on symbol server : C:\Windows\Microsoft.Net\Framework\v2.0.50727\*.dll

    please help me ,very thank you

    my E-mail:eibo521@hotmail.com

  21. Hello,
    I am trying to use the tool by running the following:
    NetMassDownloader -directory C:\Windows\Microsoft.NET\Framework\v2.0.50727

    I am getting this error:
    There is no value set for the Visual Studio debugger symbol cache.

    I have .NET 3.5 installed and VS 2008.

    Any solution?

  22. I have the same problem: on normal VS installation MassDownloader gives error "There is no value set ...blah...". What about to up your ass and put NORMAL INSTRUCTION into readme??

  23. First I got -d option not recognized. So I installed .NET 3.5
    - I only have VS 2005

    When I try with just directory I get:
    =========================
    Debugger key. The default is to look for Visual Studio 2008. If you specified
    a version with a -vsver command line option, make sure to specify the
    complete version. For example, Visual Studio 2005 is 8.0 and Visual Studio
    2008 is 9.0.

    When I try with -vs (OR -vsver) 8.0 - To say that I have VS 2005
    =============================================
    I get following error:

    Error: Only one -vsver switch can be specified on the command line.

    Even though I specified only one -VS option -it complains about specifing this twice [as seen above]

    I am just finding how frustrating it can be to use this tool. Can some one please help?


  24. I am getting "407: ProxyAuthenticationRequired" all downloads.

    Please let me know what needs to be done.

  25. If I try to do this for .Net v2.0 i get as output a whole list of pdb files with behind it and dll files with behind it. At the end the message

    Processed binaries/PDBs : 0
    Files not processed : 204
    Downloaded source files : 0

    appears?

    I have VS 2005 with addons for .net V3.5

  26. Nils,

    I'm willing to bet you don't have the latest framework installed. Microsoft only posts the source for the absolute latest and greatest version of .NET. That would be something like .NET 3.5 SP1 with all hotfixes.

    John Robbins

  27. I saw a number of people are getting the error:

    "Error on the -d switch.
    Unknown command line argument."

    when copy\pasting the example command lines,
    I got that as well, it appears the '-' is being copied as Unicode character 0x2013 which isn't supported by the arg parser:

    protected ArgParser ( String [] flagSymbols , String [] dataSymbols )
    : this ( flagSymbols ,
    dataSymbols ,
    false ,
    new string [] { "/" , "-" } ) // only regular minus, not en-dash
    {}

    also the error reporting in OnUsage then makes it look like it did process it as a switch when it didn't, ie it drops to:

    Console.WriteLine ( Constants.ErrorSwitch , errorInfo );

    producing "Error on the -d switch."

  28. If I use the current version i get source files under:
    C:\symbols\src\source\dotnetfx35SP1_3053\1.1

    However in visual studio it then doesn't find these files and instead downloads new copies to:

    C:\symbols\src\source\dotnetfx35SP1_3053\1.3

    This appears to be caused by the code hardcoing 1.1 on the assumption that .NET 3.5 SP1 maps to v1.1, this no longer seems to be the case. The code comments indicate this is a known issue but no patch to the code or issue tracking seems to exist.

    A quick hack to honor the value from the Http_Alias exacted from the pdb is to add the following code to the BuildSrcSrvDownloadFromFile method:

    string[] segments = new Uri(Http_Alias).Segments;
    targetPath = Path.GetFullPath(Path.Combine(Path.Combine(targetPath, ".."), segments[segments.Length-1]));

  29. Glen,

    Thanks! I'll get both those issues fixed. Thanks for the code patch too.

    -John Robbins

  30. Ramachandra,

    And I'd like a pony. :) Hahaha! Looks like an upcoming version of Reflector will do what you want.

    - John Robbins

  31. Hi:
    I can debug in the windows project, but it is fail in the wince. I can download the source code in windows project. why?

  32. Mr. Chen,

    WinCE, as far as I know, isn't supported by the .NET Reference Source Code. Sorry.

    - John Robbins

  33. Andrey Polovodov

    John, do you know why downloader is not able to download all sources - I've got error 400: Bad Request on most of .cs files?
    As result I've got sources from WPF, WCF, but there are no System.Net, System, System.Runtime.Remoting, etc.
    May be something was changed?

  34. Kerem Kusmezer

    The Massdownloader has been updated to support VS2010.
    Please Download The Latest Version

Leave a Comment

Archives

Tags