Wintellect Blogs

Download All the .NET Reference Source Code at Once with Net Mass Downloader

5 Feb , 2008  

Microsoft has really helped out the development community by making the .NET Reference Source Code available to the community. Being able to step in the debugger from your code to the Framework code truly helps with you understanding and use of all parts of .NET. The only drawback I’ve seen with what Microsoft is offering is that the source code can only be downloaded while you’re debugging. This means that unless you write a sample program that calls literally every class and method in the .NET Framework, you won’t get all the source. As Scott Hanselman has so excellently pointed out, reading code is as important, if not more, than writing it. What I really wanted was a way to download all the .NET Reference Source so I could just start reading everything. Additionally, I wanted to fill my download cache with all the source so I had it all even when I didn’t have a network connection.

It turns out I wasn’t the only one who had the same wishes. Kerem Kusmezer contacted me and said that he’d been hacking around with Fiddler and figured out how to download the PDBs and code from Microsoft. He asked if I I’d be interested in helping him develop the utility and I don’t think I ever wrote a response faster in my life! Kerem had already started a discussion with Shawn Burke about the legality of producing a source downloader and once we got back a very encouraging “go for it” we got to work. All the credit for the tool belongs to Kerem. He did all the really hard work of figuring out the HTTP downloads ,the PE file processing, and PDB Stream Extraction. I did the console driver, a lot of testing, and sat around amazed at how much work Kerem could accomplish in very short periods of time. I really appreciate that Kerem let me work on the project with him and being his code monkey.

The tool is called Net Mass Downloader and we made the source code available at CodePlex at http://www.codeplex.com/NetMassDownloader so you can see how it works. It’s a command line tool and running the tool without parameters produces the following output:

.Net Mass Downloader 1.0.0.0 – (c) 2008 by Kerem Kusmezer, John Robbins

Batch download the Microsoft .NET Reference Source code.

Usage: NetMassDownloader [-file <file>]
                         [-directory <directory]
                         [-output <directory>]
                         [-vsver <version>]
                         [-force] [-nologo] [-verbose] [-?]

    -file      – Download an individual file’s PDB and source code. You can
                 specify multiple file parameters. (Short -f).
    -directory – Download all the found PDB and source code for all files in
                 the specified directory. You can specify multiple
                 directory parameters (Short -d).
    -output    – The output directory for PDB and source files. The default
                 directory is the cache directory set in Visual Studio 2008.
                 By using the cache directory, you’ll have the PDB and source
                 files available to Visual Studio 2008. However, to use the
                 .NET Reference Source Code with VS 2005, use the -output
                 switch and in the Options dialog, Debugging, Symbols property
                 page, add the specified output directory to the “Symbol file
                 (.pdb) locations.” Also, add the directory to the Solution
                 Properties, Common Properties, Debug Source Files, Directories
                 containing source code location. The Visual Studio 2005
                 debugger will automatically load the source code. (Short -o)
    -vsver     – The Visual Studio version number to use for finding the cache
                 directory. The default is Visual Studio 2008,
                 but if you want to use the cache directory for Visual Studio
                 2005, you would pass ‘-vsver 8.0′ (without quotes) (Short -vs)
    -force     – If specified, forces the downloading the PDB files into the
                 symbol server. When downloading to a symbol server if the PDB
                 exists, it’s not downloaded. Using the -output switch will
                 always download and process the PDB. (Short -fo)
    -nologo    – Don’t show the logo information. (Short -n)
    -verbose   – Do verbose output. May be worth turning on as the downloading
                 source code can take a long time. (Short -v)
     -?        – This help message..

The only required arguments are –file or –directory, both of which can be specified as many times as you’d like. When you specify a directory, only the .DLL and .EXE files from that directory will be processed. If you wanted to download all the source code from binaries in the .NET 2.0 32-bit and 64-bit directories, the command line you’d pass is: -d C:\Windows\Microsoft.NET\Framework\v2.0.50727 –d C:\Windows\Microsoft.NET\Framework64\v2.0.50727.

The main purpose of Net Mass Downloader is to populate the source code download cache for debugging, the default download location is the cache you specified to Visual Studio 2008. We put in the –vsver switch to account for future Visual Studio versions so Mass Downloader could work with future CTPs and versions.

While it’s great to see the .NET Reference Source Code in Visual Studio 2008, there are a lot of developers out there who can’t upgrade yet. Initially, Kerem and I went down the path of seeing if we could patch the PDB files so Visual Studio 2005 users could have access to the .NET Reference Source Code. After a lot of hassle, we realized that there was a much easier way. If you specify the –output switch, we will put the PDB and the source code into that directory using the build paths instead of the source server paths. Thus, if your command line to NetMassDownloader.exe is –d C:\Windows\Microsoft.NET\Framework\v2.0.50727 –output c:\ReferenceSource, the PDB files are put in C:\ReferenceSource as well as the source files.

To use the .NET Reference Source with Visual Studio 2005, you would need to do the following three additional steps. The first is in the Options dialog for Visual Studio, Debugging, Symbols property page. In the “Symbol file (.pdb) locations” list box, you would add “c:\ReferenceSource” like the following.

The second setting is in Options dialog, Debugging, General property page, uncheck Require source files to exactly match the original version.

The final step you’ll have to do in each solution where you want to access the downloaded .NET Reference Source Code, you’ll go into the solution property pages, Common Properties, Debugging Source Files, and add C:\ReferenceSources to the top of the “Directories containing source code” list box as follows:

Visual Studio 2005 will automatically do the right thing and replace the first directory of every source file it finds in a PDB file with c:\ReferenceSource. With the automation model in Visual Studio 2005, I’m sure you could wipe up a macro to automatically set that path in the solution.

When you first run Net Mass Downloader, you will be prompted with the current EULA for accessing the source code. If you don’t agree with the Microsoft EULA, clicking the Decline button will not download the source code. While you might be tempted to remove the EULA check, I would strongly recommend that you not do so as Microsoft has said that they may change the EULA at any time. The last thing you want to do is get caught with a license change that could impact your life.

One group we need to thank is the Developer Division at Microsoft. First they released the .NET Reference Source Code, and second for allowing a couple of developers to have some fun and provide a utility for the community.

As always let us know on CodePlex if you encounter any bugs or have feature requests.


  • http://www.dotnetkicks.com/tipsandtricks/Download_all_the_NET_Reference_Source_at_once Anonymous

    You’ve been kicked (a good thing) – Trackback from DotNetKicks.com

  • Anonymous

    Hi,

    I’m having some problems – the SymbolCacheDir registry value is empty what exactly should be in that location :).

    I put a directory in there and it didn’t complain anymore. But I still get a
    “File not found on symbol server”. Any tips ?

  • http://lextm.blogspot.com Anonymous

    Another drawback is that only a few files are available and the debugger feature is only available in non-Express edition of Visual Studio 2008.

    Compared to X11 license that covers Novell Mono, MS-RL is not a convenient license for end users. Microsoft .NET should go true open source some day in order to promote .NET to a higher level like SUN GPLed Java.

  • jrobbins

    Sirrococo – See Shawn’s blog entry on which symbol files are actually downloadable. Only a portion of the .NET Framework code is available.

    Lex Y. Li – It’s Microsoft’s code. They can do whatever they want with the code and license it however they see fit.

    – John Robbins

  • http://blogs.microsoft.co.il/blogs/bursteg/archive/2008/02/06/download-the-net-reference-source-code.aspx Anonymous

    Download the .Net Reference Source Code Developers have always wanted to be able to download and browse

  • http://jasonhaley.com/blog/archive/2008/02/06/141168.aspx Anonymous
  • http://blogs.ugidotnet.org/PhilloPuntoIt/archive/2008/02/06/sorgenti-di-.net-anche-con-vs2005-si.-puo.-fare.aspx Anonymous

    Sorgenti di .Net anche con VS2005? SI… PUO’… FARE…

  • Anonymous

    When I run the downloader with the -d option and point it to various framework directories, I get a ‘No matching DB file on symbol server for xxx’ message for every .dll in the directory, and nothing downloads. I am able to successfully download source and PDB in VS2008 while debugging. Has anyone gotten this to work? Thx in advance.

  • Anonymous

    I’m having the same problem as Joe Modica, every file reports it can’t be found on the symbol server. I’m kind of wondering if the app is not making a connection to the server (server slammed by people running this tool perhaps?) but I haven’t downloaded the source to debug this yet.

  • Anonymous

    I should have mentioned, thanks to both of you! Once I get this working it will be a tremendous boon… we’ll be using VS2005 for a while yet, and I’ve had several debug sessions that would have been minutes rather than hours if I’d had the .net source!

  • Anonymous

    Joe – sounds like you have a firewall opened…

  • jrobbins

    Joe Modica & Squidbot,

    Have you installed the .NET 3.5 Framework? (http://www.microsoft.com/downloads/details.aspx?FamilyID=333325fd-ae52-4e35-b531-508d977d32a6&displaylang=en). You probably don’t have the matching .NET Framework version on your machines. We shoud have made that requirement clearer. You can install the .NET 3.5 Framework and VS 2005 will work just fine.

    Let us know how it goes!

    -John Robbins

  • Anonymous

    Hello,

    I’m having a problem: all the .cs files are the same size and contain apparently the same contents (some kind of EULA)!?!

    What am I doing wrong?

    Thanks.

  • Anonymous

    Beautiful, thanks, that was it. Installed .net 3.5 and downloading occurred!

  • jrobbins

    Florin,

    Weird! Did you install the .NET Framework 3.5? Any more you can say about your machine/configuration? We saw lots of strange things hacking the download, but never that problem!

    -John Robbins

  • Anonymous

    Hi Florin, it seems to be a problem in your computer that causes our framework not to detect the eula requests made for your code.
    Just get the version again, and try again, if problem persists enter as a bug with your computers regional settings, computer ram etc. in our issue tracker.

  • Anonymous

    Florin Sabau
    Florin just mail me keremskusmezer@gmail.com

  • http://musingmarc.blogspot.com Anonymous

    I used a batch file to get everything like this:

    NetMassDownloader.exe -output “C:\Dev\Microsoft.Net\v2.0″ -d “C:\Windows\Microsoft.Net\Framework\v2.0.50727″
    NetMassDownloader.exe -output “C:\Dev\Microsoft.Net\v3.0″ -d “C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5″
    NetMassDownloader.exe -output “C:\Dev\Microsoft.Net\v3.5″ -d “C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5″

  • Anonymous

    Can you give the estimated disk space required? :)

  • http://www.codeplex.com/NetMassDownloader Anonymous

    It is about 210 mb with pdb files included.

    The NetMassDownloader also accepts multiple -d in single line no need for batch :)

    NetMassDownloader.exe -output “C:\Dev\Microsoft.Net\v2.0″
    -d “C:\Windows\Microsoft.Net\Framework\v2.0.50727″
    -d “C:\Dev\Microsoft.Net\v3.0″ -d “C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5″
    -d “C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5″

    This is also legal.

  • http://www.kodyaz.com/blogs/software_development_blog/archive/2008/02/07/819.aspx Anonymous

    Hello all debuggers You know Microsoft has released the .NET Framework source codes. You can now…

  • Anonymous

    Hello Kerem,

    I’ve sent you an email with a sample .cs file.

    Thanks.

  • Anonymous

    Hello John,

    I have .NET Framework 3.5 installed by VS 2008. My regional settings are set on Romanian, but I’ve tried also when set on English(United States). The same results. I use a proxy server to access the symbols server (that might be the issue?!?).

    Thanks for your help.

  • jrobbins

    Florin,

    The proxy server is probably the issue. Neither Kerem or I tested against a proxy server. You can grab the code on CodePlex and see if you can make it work in your environment. If so, please let us know how it goes!

    – John Robbins

  • Anonymous

    Happy news! :D

    Apparently when using proxy, the status description for the EULA response is not “End User License Agreement”, but “Unknown”. Changed that in PDBWebClient and I finally got the EULA accepting dialog and then the source code.

    Florin Sabau.

  • http://www.wintellect.com/cs/blogs/jrobbins/archive/2008/02/07/net-mass-downloader-1-1-released.aspx Anonymous

    The feedback and downloads for .Net Mass Downloader have been quite pleasing! Kerem and I just went through

  • http://www.kodyaz.com/blogs/unknown_tales_from_kerem/archive/2008/02/08/821.aspx Anonymous

    The feedback and downloads for .Net Mass Downloader have been quite pleasing! Kerem and I just went through…

  • http://www.codeplex.com/NetMassDownloader Anonymous

    We have a 1.1 Version now, whole the above mentioned issues fixed.
    Check the site :)
    http://www.codeplex.com/NetMassDownloader

  • http://www.isisaka.com/blog/archives/2008/02/visual_studio_2_43.html Anonymous

    John Robbins’ Blog : Download All the .N…

  • http://www.cnblogs.com/Dah/archive/2008/02/12/1067485.html Anonymous

    原文:DownloadAllthe.NETReferenceSourceCodeatOncewithNetMassDownloader
    这篇文章介绍了如何使用NetMas…

  • http://blogs.wankuma.com/hirom/archive/2008/02/15/123209.aspx Anonymous

    VS2005でも.NETのソースコードが参照できるツールがCodePlexで公開されたそうな。

  • http://www.cnblogs.com/shanyou/archive/2008/02/15/1070151.html Anonymous

    .NETMassDownloader从软件的名称就可以看出它的特点了:如果把Mass看成一个单词,则软件的名称可以翻译为“团块下载器

  • http://blog.csdn.net/shanyou/archive/2008/02/15/2097612.aspx Anonymous

    .NET Mass Downloader 从软件的名称就可以看出它的特点了:如果把Mass看成一个单词,则软件的名称可以翻译为“团块下载器

  • http://www.cnblogs.com/shanyou/archive/2008/02/15/netmassdownloader.html Anonymous

    通过.NET Mass Downloader完成.NET整体源码的下载,而且可以通过配置VS 2005,以非常类似的方法在VS 2005中使用这些源码。

  • http://www.cnblogs.com/flier/archive/2008/01/23/1049400.html Anonymous

    更新:程序员的思维模式都差不多,KeremKusmezer和JohnRobbin发布了一个类似的工具.NETMassDownloader。既然有的用我就不重新发明轮子了,下面那个项目停止…

  • http://blog.csdn.net/vincent37528/archive/2008/02/17/2100493.aspx Anonymous

    .NET Mass Downloader 如果把Mass看成一个单词,则软件的名称可以翻译为

  • http://leblanc.mixhacks.com Anonymous

    is there anyway we can get a md5 or sha256 of all the files given the command line everyone is running:

    netmassdownloader -d “C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5″
    -d “C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0″
    -d “c:\Windows\Microsoft.net\framework\v2.0.50727″ -output c:\dev\NetFXSource

    every once in a while in the command window output i get a line:
    “The remote server returned an error: (404) Not Found.”

  • http://leblanc.mixhacks.com Anonymous

    The final output when “complete” after running:
    netmassdownloader -d “c:\Windows\Microsoft.net\framework\v2.0.50727″ -d “C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0″ -d “C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5″ -output c:\dev\NetFXSource

    Processed binaries/PDBs : 21
    Files not processed : 132
    Downloaded source files : 7174

    how about everyone else?

  • http://blogs.codes-sources.com/coq/archive/2008/02/24/net-reference-source-code-outil-netmassdownloader.aspx Anonymous

    Comme vous le savez sans doute, l’information a été abordée sur les blogs CodeS-SourceS ( entre autres

  • http://none Anonymous

    i can’t download file

    message:

    File ont on symbol server : C:\Windows\Microsoft.Net\Framework\v2.0.50727\*.dll

    please help me ,very thank you

    my E-mail:eibo521@hotmail.com

  • http://advdbg.org/blogs/advdbg_system/articles/503.aspx Anonymous

    It’s a long time that I didn’t visit John’s Blog. Today when…

  • http://www.jantucek.cz Anonymous

    Good job…

  • http://www.cnblogs.com/gOODiDEA/archive/2008/02/14/1089672.html Anonymous

    Web: jQuery1.2CheatSheet
    FasterJavaScriptTrim
    Self-curryingJavaScriptfunctions
    Five…

  • Anonymous

    I am need to this pakage.

  • Anonymous

    Yea! Now i have all .net source on my disc xD

  • http://bhaidar.net Anonymous

    Hello,
    I am trying to use the tool by running the following:
    NetMassDownloader -directory C:\Windows\Microsoft.NET\Framework\v2.0.50727

    I am getting this error:
    There is no value set for the Visual Studio debugger symbol cache.

    I have .NET 3.5 installed and VS 2008.

    Any solution?

  • jrobbins

    Bilal,

    Set up VS in accordance with the instructions in Shawn Burke’s blog.

    – John Robbins

  • Anonymous

    I have the same problem: on normal VS installation MassDownloader gives error “There is no value set …blah…”. What about to up your ass and put NORMAL INSTRUCTION into readme??

  • http://www.codeproject.com/KB/dotnet/netmassdownloader.aspx Anonymous

    Check the codeproject article for details.

  • http://blogs.infosupport.com/blogs/jonathan/archive/2008/05/23/Cool-tools_3A00_-NetMassDownloader.aspx Anonymous

    Today I was working on an issue in our application that was brought to our attention by one of the users.

  • http://weblogs.asp.net/craigshoemaker/archive/2008/07/09/full-disclosure-debugging.aspx Anonymous

    Listen to the Show Debugging guru John Robbins brings to life the saying, “you cannot debug what you

  • http://www.wintellect.com/CS/blogs/jrobbins/archive/2008/08/12/random-thoughts-on-visual-studio-2008-sp1.aspx Anonymous

    As the world knows, Visual Studio 2008 SP1 is out so start your download engines. I’ve installed it on

  • http://blogs.msdn.com/dougste/archive/2008/08/29/net-3-5-framework-sp1-reference-sources-available.aspx Anonymous

    In mid January the first release of the .NET Framework 3.5 reference sources occurred. In mid February

  • Anonymous

    thanks

  • Anonymous

    thanks

  • Anonymous

    First I got -d option not recognized. So I installed .NET 3.5
    – I only have VS 2005

    When I try with just directory I get:
    =========================
    Debugger key. The default is to look for Visual Studio 2008. If you specified
    a version with a -vsver command line option, make sure to specify the
    complete version. For example, Visual Studio 2005 is 8.0 and Visual Studio
    2008 is 9.0.

    When I try with -vs (OR -vsver) 8.0 – To say that I have VS 2005
    =============================================
    I get following error:

    Error: Only one -vsver switch can be specified on the command line.

    Even though I specified only one -VS option -it complains about specifing this twice [as seen above]

    I am just finding how frustrating it can be to use this tool. Can some one please help?

  • Anonymous

    I am getting “407: ProxyAuthenticationRequired” all downloads.

    Please let me know what needs to be done.

  • http://www.wintellect.com/CS/blogs/jrobbins/archive/2009/04/03/customizing-exception-handling-in-the-vs-debugger.aspx Anonymous

    In the last two weeks I’ve gotten many questions about how to automate and program the exception handling

  • Anonymous

    If I try to do this for .Net v2.0 i get as output a whole list of pdb files with behind it and dll files with behind it. At the end the message

    Processed binaries/PDBs : 0
    Files not processed : 204
    Downloaded source files : 0

    appears?

    I have VS 2005 with addons for .net V3.5

  • jrobbins

    Nils,

    I’m willing to bet you don’t have the latest framework installed. Microsoft only posts the source for the absolute latest and greatest version of .NET. That would be something like .NET 3.5 SP1 with all hotfixes.

    John Robbins

  • Anonymous

    I saw a number of people are getting the error:

    “Error on the -d switch.
    Unknown command line argument.”

    when copy\pasting the example command lines,
    I got that as well, it appears the ‘-‘ is being copied as Unicode character 0x2013 which isn’t supported by the arg parser:

    protected ArgParser ( String [] flagSymbols , String [] dataSymbols )
    : this ( flagSymbols ,
    dataSymbols ,
    false ,
    new string [] { “/” , “-” } ) // only regular minus, not en-dash
    {}

    also the error reporting in OnUsage then makes it look like it did process it as a switch when it didn’t, ie it drops to:

    Console.WriteLine ( Constants.ErrorSwitch , errorInfo );

    producing “Error on the -d switch.”

  • Anonymous

    If I use the current version i get source files under:
    C:\symbols\src\source\dotnetfx35SP1_3053\1.1

    However in visual studio it then doesn’t find these files and instead downloads new copies to:

    C:\symbols\src\source\dotnetfx35SP1_3053\1.3

    This appears to be caused by the code hardcoing 1.1 on the assumption that .NET 3.5 SP1 maps to v1.1, this no longer seems to be the case. The code comments indicate this is a known issue but no patch to the code or issue tracking seems to exist.

    A quick hack to honor the value from the Http_Alias exacted from the pdb is to add the following code to the BuildSrcSrvDownloadFromFile method:

    string[] segments = new Uri(Http_Alias).Segments;
    targetPath = Path.GetFullPath(Path.Combine(Path.Combine(targetPath, “..”), segments[segments.Length-1]));

  • jrobbins

    Glen,

    Thanks! I’ll get both those issues fixed. Thanks for the code patch too.

    -John Robbins

  • Anonymous

    good.but i want direct source code for applications

  • jrobbins

    Ramachandra,

    And I’d like a pony. :) Hahaha! Looks like an upcoming version of Reflector will do what you want.

    – John Robbins

  • Anonymous

    Hi:
    I can debug in the windows project, but it is fail in the wince. I can download the source code in windows project. why?

  • jrobbins

    Mr. Chen,

    WinCE, as far as I know, isn’t supported by the .NET Reference Source Code. Sorry.

    – John Robbins

  • Anonymous

    source code

  • Anonymous

    John, do you know why downloader is not able to download all sources – I’ve got error 400: Bad Request on most of .cs files?
    As result I’ve got sources from WPF, WCF, but there are no System.Net, System, System.Runtime.Remoting, etc.
    May be something was changed?

  • Anonymous

    The Massdownloader has been updated to support VS2010.
    Please Download The Latest Version

  • http://richarddingwall.name Anonymous

    Nice tool, taking forever to download however. Why aren’t there prebuilt zips available with all the source in it?