Download All the .NET Reference Source Code at Once with Net Mass Downloader

71 Comments February 5, 2008

Microsoft has really helped out the development community by making the .NET Reference Source Code available to the community. Being able to step in the debugger from your code to the Framework code truly helps with you understanding and use of all parts of .NET. The only drawback I've seen with what Microsoft is offering is that the source code can only be downloaded while you're debugging. This means that unless you write a sample program that calls literally every class and method in the .NET Framework, you won't get all the source. As Scott Hanselman has so excellently pointed out, reading code is as important, if not more, than writing it. What I really wanted was a way to download all the .NET Reference Source so I could just start reading everything. Additionally, I wanted to fill my download cache with all the source so I had it all even when I didn't have a network connection.

It turns out I wasn't the only one who had the same wishes. Kerem Kusmezer contacted me and said that he'd been hacking around with Fiddler and figured out how to download the PDBs and code from Microsoft. He asked if I I'd be interested in helping him develop the utility and I don't think I ever wrote a response faster in my life! Kerem had already started a discussion with Shawn Burke about the legality of producing a source downloader and once we got back a very encouraging "go for it" we got to work. All the credit for the tool belongs to Kerem. He did all the really hard work of figuring out the HTTP downloads ,the PE file processing, and PDB Stream Extraction. I did the console driver, a lot of testing, and sat around amazed at how much work Kerem could accomplish in very short periods of time. I really appreciate that Kerem let me work on the project with him and being his code monkey.

The tool is called Net Mass Downloader and we made the source code available at CodePlex at http://www.codeplex.com/NetMassDownloader so you can see how it works. It's a command line tool and running the tool without parameters produces the following output:

.Net Mass Downloader 1.0.0.0 - (c) 2008 by Kerem Kusmezer, John Robbins

Batch download the Microsoft .NET Reference Source code.

Usage: NetMassDownloader [-file <file>]
                         [-directory <directory]
                         [-output <directory>]
                         [-vsver <version>]
                         [-force] [-nologo] [-verbose] [-?]

    -file      - Download an individual file's PDB and source code. You can
                 specify multiple file parameters. (Short -f).
    -directory - Download all the found PDB and source code for all files in
                 the specified directory. You can specify multiple
                 directory parameters (Short -d).
    -output    - The output directory for PDB and source files. The default
                 directory is the cache directory set in Visual Studio 2008.
                 By using the cache directory, you'll have the PDB and source
                 files available to Visual Studio 2008. However, to use the
                 .NET Reference Source Code with VS 2005, use the -output
                 switch and in the Options dialog, Debugging, Symbols property
                 page, add the specified output directory to the "Symbol file
                 (.pdb) locations." Also, add the directory to the Solution
                 Properties, Common Properties, Debug Source Files, Directories
                 containing source code location. The Visual Studio 2005
                 debugger will automatically load the source code. (Short -o)
    -vsver     - The Visual Studio version number to use for finding the cache
                 directory. The default is Visual Studio 2008,
                 but if you want to use the cache directory for Visual Studio
                 2005, you would pass '-vsver 8.0' (without quotes) (Short -vs)
    -force     - If specified, forces the downloading the PDB files into the
                 symbol server. When downloading to a symbol server if the PDB
                 exists, it's not downloaded. Using the -output switch will
                 always download and process the PDB. (Short -fo)
    -nologo    - Don't show the logo information. (Short -n)
    -verbose   - Do verbose output. May be worth turning on as the downloading
                 source code can take a long time. (Short -v)
     -?        - This help message..

The only required arguments are –file or –directory, both of which can be specified as many times as you'd like. When you specify a directory, only the .DLL and .EXE files from that directory will be processed. If you wanted to download all the source code from binaries in the .NET 2.0 32-bit and 64-bit directories, the command line you'd pass is: -d C:\Windows\Microsoft.NET\Framework\v2.0.50727 –d C:\Windows\Microsoft.NET\Framework64\v2.0.50727.

The main purpose of Net Mass Downloader is to populate the source code download cache for debugging, the default download location is the cache you specified to Visual Studio 2008. We put in the –vsver switch to account for future Visual Studio versions so Mass Downloader could work with future CTPs and versions.

While it's great to see the .NET Reference Source Code in Visual Studio 2008, there are a lot of developers out there who can't upgrade yet. Initially, Kerem and I went down the path of seeing if we could patch the PDB files so Visual Studio 2005 users could have access to the .NET Reference Source Code. After a lot of hassle, we realized that there was a much easier way. If you specify the –output switch, we will put the PDB and the source code into that directory using the build paths instead of the source server paths. Thus, if your command line to NetMassDownloader.exe is –d C:\Windows\Microsoft.NET\Framework\v2.0.50727 –output c:\ReferenceSource, the PDB files are put in C:\ReferenceSource as well as the source files.

To use the .NET Reference Source with Visual Studio 2005, you would need to do the following three additional steps. The first is in the Options dialog for Visual Studio, Debugging, Symbols property page. In the "Symbol file (.pdb) locations" list box, you would add "c:\ReferenceSource" like the following.

The second setting is in Options dialog, Debugging, General property page, uncheck Require source files to exactly match the original version.

The final step you'll have to do in each solution where you want to access the downloaded .NET Reference Source Code, you'll go into the solution property pages, Common Properties, Debugging Source Files, and add C:\ReferenceSources to the top of the "Directories containing source code" list box as follows:

Visual Studio 2005 will automatically do the right thing and replace the first directory of every source file it finds in a PDB file with c:\ReferenceSource. With the automation model in Visual Studio 2005, I'm sure you could wipe up a macro to automatically set that path in the solution.

When you first run Net Mass Downloader, you will be prompted with the current EULA for accessing the source code. If you don't agree with the Microsoft EULA, clicking the Decline button will not download the source code. While you might be tempted to remove the EULA check, I would strongly recommend that you not do so as Microsoft has said that they may change the EULA at any time. The last thing you want to do is get caught with a license change that could impact your life.

One group we need to thank is the Developer Division at Microsoft. First they released the .NET Reference Source Code, and second for allowing a couple of developers to have some fun and provide a utility for the community.

As always let us know on CodePlex if you encounter any bugs or have feature requests.


71 Comments

  • Gravatar Image
    DotNetKicks.com February 5, 2008 11:31 PM

    You've been kicked (a good thing) - Trackback from DotNetKicks.com

  • Gravatar Image
    sirrocco February 6, 2008 12:08 AM

    Hi,

    I'm having some problems - the SymbolCacheDir registry value is empty what exactly should be in that location :).

    I put a directory in there and it didn't complain anymore. But I still get a
    "File not found on symbol server". Any tips ?

  • Gravatar Image
    Lex Y. Li February 6, 2008 12:29 AM

    Another drawback is that only a few files are available and the debugger feature is only available in non-Express edition of Visual Studio 2008.

    Compared to X11 license that covers Novell Mono, MS-RL is not a convenient license for end users. Microsoft .NET should go true open source some day in order to promote .NET to a higher level like SUN GPLed Java.

  • Gravatar Image
    jrobbins February 6, 2008 12:54 AM

    Sirrococo - See Shawn's blog entry on which symbol files are actually downloadable. Only a portion of the .NET Framework code is available.

    Lex Y. Li - It's Microsoft's code. They can do whatever they want with the code and license it however they see fit.

    - John Robbins

  • Gravatar Image
    Guy Burstein's Blog February 6, 2008 2:11 AM

    Download the .Net Reference Source Code Developers have always wanted to be able to download and browse

  • Gravatar Image
    Jason Haley February 6, 2008 12:25 PM

  • Gravatar Image
    PhilloPuntoIt February 6, 2008 1:33 PM

    Sorgenti di .Net anche con VS2005? SI... PUO'... FARE...

  • Gravatar Image
    Joe Modica February 6, 2008 3:21 PM

    When I run the downloader with the -d option and point it to various framework directories, I get a 'No matching DB file on symbol server for xxx' message for every .dll in the directory, and nothing downloads. I am able to successfully download source and PDB in VS2008 while debugging. Has anyone gotten this to work? Thx in advance.

  • Gravatar Image
    squidbot February 6, 2008 4:17 PM

    I'm having the same problem as Joe Modica, every file reports it can't be found on the symbol server. I'm kind of wondering if the app is not making a connection to the server (server slammed by people running this tool perhaps?) but I haven't downloaded the source to debug this yet.

  • Gravatar Image
    squidbot February 6, 2008 4:24 PM

    I should have mentioned, thanks to both of you! Once I get this working it will be a tremendous boon... we'll be using VS2005 for a while yet, and I've had several debug sessions that would have been minutes rather than hours if I'd had the .net source!

  • Gravatar Image
    Ben February 6, 2008 4:29 PM

    Joe - sounds like you have a firewall opened...

  • Gravatar Image
    jrobbins February 6, 2008 4:30 PM

    Joe Modica & Squidbot,

    Have you installed the .NET 3.5 Framework? (http://www.microsoft.com/downloads/details.aspx?FamilyID=333325fd-ae52-4e35-b531-508d977d32a6&displaylang=en). You probably don't have the matching .NET Framework version on your machines. We shoud have made that requirement clearer. You can install the .NET 3.5 Framework and VS 2005 will work just fine.

    Let us know how it goes!

    -John Robbins

  • Gravatar Image
    Florin Sabau February 6, 2008 5:08 PM

    Hello,

    I'm having a problem: all the .cs files are the same size and contain apparently the same contents (some kind of EULA)!?!

    What am I doing wrong?

    Thanks.

  • Gravatar Image
    squidbot February 6, 2008 5:56 PM

    Beautiful, thanks, that was it. Installed .net 3.5 and downloading occurred!

  • Gravatar Image
    jrobbins February 6, 2008 7:05 PM

    Florin,

    Weird! Did you install the .NET Framework 3.5? Any more you can say about your machine/configuration? We saw lots of strange things hacking the download, but never that problem!

    -John Robbins

  • Gravatar Image
    Kerem Kusmezer February 6, 2008 7:19 PM

    Hi Florin, it seems to be a problem in your computer that causes our framework not to detect the eula requests made for your code.
    Just get the version again, and try again, if problem persists enter as a bug with your computers regional settings, computer ram etc. in our issue tracker.

  • Gravatar Image
    Kerem Kusmezer February 6, 2008 7:27 PM

    Florin Sabau
    Florin just mail me keremskusmezer@gmail.com

  • Gravatar Image
    Marc Brooks February 6, 2008 8:50 PM

    I used a batch file to get everything like this:

    NetMassDownloader.exe -output "C:\Dev\Microsoft.Net\v2.0" -d "C:\Windows\Microsoft.Net\Framework\v2.0.50727"
    NetMassDownloader.exe -output "C:\Dev\Microsoft.Net\v3.0" -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5"
    NetMassDownloader.exe -output "C:\Dev\Microsoft.Net\v3.5" -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5"

  • Gravatar Image
    anonymous February 7, 2008 3:19 AM

    Can you give the estimated disk space required? :)

  • Gravatar Image
    Kerem Kusmezer February 7, 2008 3:43 AM

    It is about 210 mb with pdb files included.

    The NetMassDownloader also accepts multiple -d in single line no need for batch :)

    NetMassDownloader.exe -output "C:\Dev\Microsoft.Net\v2.0"
    -d "C:\Windows\Microsoft.Net\Framework\v2.0.50727"
    -d "C:\Dev\Microsoft.Net\v3.0" -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5"
    -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5"

    This is also legal.

  • Gravatar Image
    Eralper's Blog on Software Development February 7, 2008 5:14 AM

    Hello all debuggers You know Microsoft has released the .NET Framework source codes. You can now...

  • Gravatar Image
    Florin Sabau February 7, 2008 2:03 PM

    Hello Kerem,

    I've sent you an email with a sample .cs file.

    Thanks.

  • Gravatar Image
    Florin Sabau February 7, 2008 3:05 PM

    Hello John,

    I have .NET Framework 3.5 installed by VS 2008. My regional settings are set on Romanian, but I've tried also when set on English(United States). The same results. I use a proxy server to access the symbols server (that might be the issue?!?).

    Thanks for your help.

  • Gravatar Image
    jrobbins February 7, 2008 4:06 PM

    Florin,

    The proxy server is probably the issue. Neither Kerem or I tested against a proxy server. You can grab the code on CodePlex and see if you can make it work in your environment. If so, please let us know how it goes!

    - John Robbins

  • Gravatar Image
    Florin Sabau February 7, 2008 4:44 PM

    Happy news! :D

    Apparently when using proxy, the status description for the EULA response is not "End User License Agreement", but "Unknown". Changed that in PDBWebClient and I finally got the EULA accepting dialog and then the source code.

    Florin Sabau.

  • Gravatar Image
    John Robbins' Blog February 7, 2008 7:04 PM

    The feedback and downloads for .Net Mass Downloader have been quite pleasing! Kerem and I just went through

  • Gravatar Image
    Unknown Tales From Kerem Kusmezer February 7, 2008 7:14 PM

    The feedback and downloads for .Net Mass Downloader have been quite pleasing! Kerem and I just went through...

  • Gravatar Image
    Kerem Kusmezer February 7, 2008 7:37 PM

    We have a 1.1 Version now, whole the above mentioned issues fixed.
    Check the site :)
    http://www.codeplex.com/NetMassDownloader

  • Gravatar Image
    OPC Diary February 12, 2008 5:40 AM

    John Robbins' Blog : Download All the .N...

  • Gravatar Image
    Adrian H. February 12, 2008 8:58 AM

    原文:DownloadAllthe.NETReferenceSourceCodeatOncewithNetMassDownloader 这篇文章介绍了如何使用NetMas...

  • Gravatar Image
    ひろえむの日々是勉強 February 14, 2008 4:55 PM

    VS2005でも.NETのソースコードが参照できるツールがCodePlexで公開されたそうな。

  • Gravatar Image
    自由、创新、研究、探索…… February 15, 2008 7:16 AM

    .NETMassDownloader从软件的名称就可以看出它的特点了:如果把Mass看成一个单词,则软件的名称可以翻译为“团块下载器

  • Gravatar Image
    shanyou February 15, 2008 7:16 AM

    .NET Mass Downloader 从软件的名称就可以看出它的特点了:如果把Mass看成一个单词,则软件的名称可以翻译为“团块下载器

  • Gravatar Image
    自由、创新、研究、探索…… February 15, 2008 7:30 AM

    通过.NET Mass Downloader完成.NET整体源码的下载,而且可以通过配置VS 2005,以非常类似的方法在VS 2005中使用这些源码。

  • Gravatar Image
    Flier Lu February 15, 2008 9:59 PM

    更新:程序员的思维模式都差不多,KeremKusmezer和JohnRobbin发布了一个类似的工具.NETMassDownloader。既然有的用我就不重新发明轮子了,下面那个项目停止...

  • Gravatar Image
    vincent February 17, 2008 12:29 AM

    .NET Mass Downloader 如果把Mass看成一个单词,则软件的名称可以翻译为

  • Gravatar Image
    Leblanc Meneses February 18, 2008 3:48 AM

    is there anyway we can get a md5 or sha256 of all the files given the command line everyone is running:

    netmassdownloader -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5"
    -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0"
    -d "c:\Windows\Microsoft.net\framework\v2.0.50727" -output c:\dev\NetFXSource


    every once in a while in the command window output i get a line:
    "The remote server returned an error: (404) Not Found."

  • Gravatar Image
    Leblanc Meneses February 18, 2008 11:47 AM

    The final output when "complete" after running:
    netmassdownloader -d "c:\Windows\Microsoft.net\framework\v2.0.50727" -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0" -d "C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5" -output c:\dev\NetFXSource

    Processed binaries/PDBs : 21
    Files not processed : 132
    Downloaded source files : 7174


    how about everyone else?

  • Gravatar Image
    CoqBlog February 24, 2008 9:45 AM

    Comme vous le savez sans doute, l'information a été abordée sur les blogs CodeS-SourceS ( entre autres

  • Gravatar Image
    eibo February 25, 2008 6:41 AM

    i can't download file

    message:

    File ont on symbol server : C:\Windows\Microsoft.Net\Framework\v2.0.50727\*.dll

    please help me ,very thank you

    my E-mail:eibo521@hotmail.com

  • Gravatar Image
    AdvDbg System Section February 28, 2008 6:57 AM

    It's a long time that I didn't visit John's Blog. Today when...

  • Gravatar Image
    Jan Toček March 2, 2008 4:03 PM

    Good job...

  • Gravatar Image
    gOODiDEA March 3, 2008 8:44 PM

    Web: jQuery1.2CheatSheet FasterJavaScriptTrim Self-curryingJavaScriptfunctions Five...

  • Gravatar Image
    Javan March 7, 2008 2:19 AM

    I am need to this pakage.

  • Gravatar Image
    Pablo March 8, 2008 3:14 PM

    Yea! Now i have all .net source on my disc xD

  • Gravatar Image
    Bilal Haidar [MVP] March 12, 2008 3:15 PM

    Hello,
    I am trying to use the tool by running the following:
    NetMassDownloader -directory C:\Windows\Microsoft.NET\Framework\v2.0.50727

    I am getting this error:
    There is no value set for the Visual Studio debugger symbol cache.

    I have .NET 3.5 installed and VS 2008.

    Any solution?

  • Gravatar Image
    jrobbins March 13, 2008 4:12 PM

    Bilal,

    Set up VS in accordance with the instructions in Shawn Burke's blog.

    - John Robbins

  • Gravatar Image
    Thorn March 25, 2008 2:03 PM

    I have the same problem: on normal VS installation MassDownloader gives error "There is no value set ...blah...". What about to up your ass and put NORMAL INSTRUCTION into readme??

  • Gravatar Image
    Kerem Kusmezer March 31, 2008 9:22 AM

    Check the codeproject article for details.

  • Gravatar Image
    Jonathan Mezach May 23, 2008 12:25 PM

    Today I was working on an issue in our application that was brought to our attention by one of the users.

  • Gravatar Image
    Craig Shoemaker July 9, 2008 9:38 AM

    Listen to the Show Debugging guru John Robbins brings to life the saying, “you cannot debug what you

  • Gravatar Image
    John Robbins' Blog August 12, 2008 1:23 AM

    As the world knows, Visual Studio 2008 SP1 is out so start your download engines. I've installed it on

  • Gravatar Image
    Notes from a dark corner August 28, 2008 7:16 PM

    In mid January the first release of the .NET Framework 3.5 reference sources occurred. In mid February

  • Gravatar Image
    ismail September 25, 2008 5:40 PM

    thanks

  • Gravatar Image
    ismail September 25, 2008 5:41 PM

    thanks

  • Gravatar Image
    JT October 10, 2008 3:42 PM

    First I got -d option not recognized. So I installed .NET 3.5
    - I only have VS 2005

    When I try with just directory I get:
    =========================
    Debugger key. The default is to look for Visual Studio 2008. If you specified
    a version with a -vsver command line option, make sure to specify the
    complete version. For example, Visual Studio 2005 is 8.0 and Visual Studio
    2008 is 9.0.

    When I try with -vs (OR -vsver) 8.0 - To say that I have VS 2005
    =============================================
    I get following error:

    Error: Only one -vsver switch can be specified on the command line.

    Even though I specified only one -VS option -it complains about specifing this twice [as seen above]

    I am just finding how frustrating it can be to use this tool. Can some one please help?


  • Gravatar Image
    JTBS October 13, 2008 5:33 PM

    I am getting "407: ProxyAuthenticationRequired" all downloads.

    Please let me know what needs to be done.

  • Gravatar Image
    John Robbins' Blog April 2, 2009 9:11 PM

    In the last two weeks I've gotten many questions about how to automate and program the exception handling

  • Gravatar Image
    Nils Berghs April 29, 2009 10:16 AM

    If I try to do this for .Net v2.0 i get as output a whole list of pdb files with behind it and dll files with behind it. At the end the message

    Processed binaries/PDBs : 0
    Files not processed : 204
    Downloaded source files : 0

    appears?

    I have VS 2005 with addons for .net V3.5

  • Gravatar Image
    jrobbins April 29, 2009 11:09 AM

    Nils,

    I'm willing to bet you don't have the latest framework installed. Microsoft only posts the source for the absolute latest and greatest version of .NET. That would be something like .NET 3.5 SP1 with all hotfixes.

    John Robbins

  • Gravatar Image
    Glen Summers May 17, 2009 8:20 AM

    I saw a number of people are getting the error:

    "Error on the -d switch.
    Unknown command line argument."

    when copy\pasting the example command lines,
    I got that as well, it appears the '-' is being copied as Unicode character 0x2013 which isn't supported by the arg parser:

    protected ArgParser ( String [] flagSymbols , String [] dataSymbols )
    : this ( flagSymbols ,
    dataSymbols ,
    false ,
    new string [] { "/" , "-" } ) // only regular minus, not en-dash
    {}

    also the error reporting in OnUsage then makes it look like it did process it as a switch when it didn't, ie it drops to:

    Console.WriteLine ( Constants.ErrorSwitch , errorInfo );

    producing "Error on the -d switch."

  • Gravatar Image
    Glen Summers May 17, 2009 9:13 AM

    If I use the current version i get source files under:
    C:\symbols\src\source\dotnetfx35SP1_3053\1.1

    However in visual studio it then doesn't find these files and instead downloads new copies to:

    C:\symbols\src\source\dotnetfx35SP1_3053\1.3

    This appears to be caused by the code hardcoing 1.1 on the assumption that .NET 3.5 SP1 maps to v1.1, this no longer seems to be the case. The code comments indicate this is a known issue but no patch to the code or issue tracking seems to exist.

    A quick hack to honor the value from the Http_Alias exacted from the pdb is to add the following code to the BuildSrcSrvDownloadFromFile method:

    string[] segments = new Uri(Http_Alias).Segments;
    targetPath = Path.GetFullPath(Path.Combine(Path.Combine(targetPath, ".."), segments[segments.Length-1]));

  • Gravatar Image
    jrobbins May 17, 2009 12:28 PM

    Glen,

    Thanks! I'll get both those issues fixed. Thanks for the code patch too.

    -John Robbins

  • Gravatar Image
    ramachandra November 4, 2009 5:48 AM

    good.but i want direct source code for applications

  • Gravatar Image
    jrobbins November 4, 2009 10:58 AM

    Ramachandra,

    And I'd like a pony. :) Hahaha! Looks like an upcoming version of Reflector will do what you want.

    - John Robbins

  • Gravatar Image
    Mr.Chen November 28, 2009 2:00 AM

    Hi:
    I can debug in the windows project, but it is fail in the wince. I can download the source code in windows project. why?

  • Gravatar Image
    jrobbins November 28, 2009 2:29 AM

    Mr. Chen,

    WinCE, as far as I know, isn't supported by the .NET Reference Source Code. Sorry.

    - John Robbins

  • Gravatar Image
    amjath February 25, 2010 4:31 AM

    source code

  • Gravatar Image
    Andrey Polovodov April 9, 2010 6:34 AM

    John, do you know why downloader is not able to download all sources - I've got error 400: Bad Request on most of .cs files?
    As result I've got sources from WPF, WCF, but there are no System.Net, System, System.Runtime.Remoting, etc.
    May be something was changed?

  • Gravatar Image
    Kerem Kusmezer April 16, 2010 8:52 AM

    The Massdownloader has been updated to support VS2010.
    Please Download The Latest Version

  • Gravatar Image
    Richard Dingwall April 19, 2010 8:24 AM

    Nice tool, taking forever to download however. Why aren't there prebuilt zips available with all the source in it?

Have a Comment?

Archives

Blogs