Paul Mehner's Blog

How To Get Diagnostic Data From Services Running In The Windows Azure Cloud

2010-03-10T19:48:19Z

Each instance of Windows Azure Service Role runs its own monitor to gather its own instance specific diagnostic data. The problem that immediately presents itself is knowing what exactly is being collected, where the data is being saved, and how to retrieve it for inspection. The purpose of this blog post is to illuminate these areas a little bit better.

So lets start at the beginning… When you create a new Windows Azure Web Role, Visual Studio will automatically add a boilerplate WebRole.cs file to your project. By default, the OnStart() method of the WebRole is overridden with an implementation that starts the Windows Azure Diagnostic Monitor. By default, Windows Azure will log its own diagnostics, IIS 7.0 logs, plus Windows Diagnostics.

The argument to the static Start method of the DiagnosticMonitor class is the Windows Azure Data Storage connection string located in the ServiceConfiguration.cscfg file.

When the value of the connection string is “UseDevelopmentStorage=true” then the Developer Fabric will use the local Development Storage to simulate storage in the cloud. Of course in staging or production, this string would point to the RESTful data storage endpoint and would contain your Windows Azure Data Storage AccountName and AccountKey.

We can inspect the “wad-control-container” of Blob storage to find the collected diagnostic information. run your favorite Windows Azure Storage exploration tool. In my example, I am using the Windows Azure Storage Explorer from the CodePlex site. You can use this tool to download the container and its contents to your local file-system for further analysis.

We can also augment the diagnostic data collected to include other data sources as well.

Let’s say you’re also interested in capturing failed IIS and ASP.NET requests. You can augment the data that Windows Azure is already capturing by adding a <traceFailedRequest> element to the <system.webServer/tracing> section. Of course you can control the paths of the page(s) to be tracked, and you can set the verbosity to an appropriate tracing level for your circumstance, including filtering the general areas of coverage such as Authentication, Security, etc. An example might look like this:

We can also collect Windows Event Logs by simply adding an XPath expression of the event sources to be captured of the WindowsEventLog.DataSources property located on the configuration object.

It is possible that a hardware or software defect might be causing mysterious or intermittent operating system failures. Fortunately, we can also configure our instances to collect full or partial crash dumps by calling the static EnableCollection method of the Microsoft.WindowsAzure.Diagnostics.CrashDumps type. Passing true to this method will capture complete crash dumps, passing false will collect partial dumps.

Although the path may be slightly more illuminated now, there are still many dark areas beyond our present location. In my opinion, there is still much work to be done in tooling and making this data useable in “real world” scenarios. It is trivial to sift through a dozen or so entries from a single service instance, but it is nearly impossible to imagine the difficulty of finding what you are looking for in the potentially massive data collected by multiple simultaneous service instances running a busy high-volume application. There are several parties working to provide solutions in this space, but no clear leaders at this time.

Idempotency for Windows Azure Message Queues

2010-03-01T03:58:41Z

Idempotency is the mathematical term used to describe a system that produces the same result when a formula or procedure is applied numerous times against the same target. In software systems, this translates to an ability to perform an operation more than one time with knowledge that the resulting state of the system will be consistent. Idempotency does not dictate the mechanism by which this consistency is to be achieved, only the fact that it must.

Queues are useful in Windows Azure for delivering work requests to worker roles. It is the primary architectural means by which web roles signal worker roles to begin asynchronous performance of work. When a worker role accepts a message from a queue, the queue hides that message from other workers for 30 seconds to reduce the probability that a message will be operated on by multiple simultaneous workers. This approach does much to greatly reduce the probability that redundant work will be performed by the system, but it does not prevent it!

If a message takes longer to process than is allowed by Windows Azure, then the message is made visible again for other workers to pick up and process. It is therefore possible for more than one worker to be working on the same work at the same time… the original recipient of the message, plus the new worker who picks it up when it becomes visible in the queue again. In addition, the typical pattern for failed or corrupted message receipt in a fault tolerant system is to retry message delivery. This can also lead to redundant work being performed.

The fact that multiple workers may work on the same message makes it essential for us to design our software for use in the cloud with idempotence in mind. An argument that idempotence will only matter once in hundreds of thousands of transactions is still very problematic if your system may be processing millions of transactions, or where the integrity of your data may be mission critical.

There seems to be a lot of blog posts and forum entries on the importance of writing idempotent services, but very little in the way of constructive feedback that I was able to find on how developers should go about achieving the objective of idempotency, and thus the purpose of this blog post.

One suggested technique for achieving idempotency that I read on several blogs and saw being discussed in forums while grokking material on this topic was to avoid the problem altogether. Many people suggested creating a table of message IDs and then forcing the workers verify the state of a message by consulting the table before processing an incoming message. Even one book author of SOA architectures put this idea forward. To my way of thinking, avoidance of idempotency does not make your software idempotent; such schemes are merely a pattern to avoid the problem rather than to design for it. This isn’t necessarily a bad way to go for some software systems, but be aware that pattern itself may contain its own set of flaws because an error could keep the table from being updated, and there is a time windows where the database table itself might hold inaccurate state information thereby allowing the two workers to still execute simultaneously. The old two-phase commit solution starts to raise its ugly head. Since such schemes could have problems, a better question to ask yourself is this… what is the sate of your data will be after the execution of a message received multiple times. Is your data consistent or inconsistent?

For a system to be truly idempotent we must be capable of processing the same message twice and after processing that message we must still be in a consistent state.

Let’s say that we want to update a customer’s address. Our service receives a message from some application with the new street address of our customer. We process the message and the address is changed in our database. If we receive this message again the work will be performed twice. No matter how inefficient or unsavory this may be, the resulting state of the customer’s address will be identical. In other words, our overly-simplified address change operation would be considered idempotent. If two messages for the same customer arrive carrying two separate addresses, the first one would succeed and so would the second one. Again, we would still be idempotent in the sense that our data was consistent; however, we have set ourselves up for a “last-in-wins” model. This is not necessarily a bad thing but we should be aware of it in our design.

Many businesses extend credit to their customers. No reasonable business would extend such credit without placing limits on it. Instead of the customer address example, let imagine that our messages are for new orders from our customers. If such a message were to be processed twice without any concern for idempotency, our customer might receive twice as much product as they ordered, and they may find themselves prematurely exceeding their credit limit on subsequent orders. This would clearly not be idempotent. So how do we get to where we want to go?

If the message contains the invoice number, then we might construct our business and database operations to perform the add operation in such a manner as to ensure that the data is never inserted into the table twice. We could perform the insertion into the invoice table as part of a transaction where the invoice number was not already present in the table. This would result in the insertion of one row into the table for the first receipt, but zero rows into the table on subsequent attempts. In other words our add operation would leave the data in a consistent state no matter how many times we replayed the message.

If the message was to perform an update of an existing invoice, then things get a little more sophisticated, but still very manageable. By using and comparing a timestamp column for equality with the value contained in the message we can perform the update where the row’s timestamp column is equal to the value contained in the message. If the incoming message caries an equivalent timestamp of the data at the time that it was issued to the sender, then this timestamp can be checked against the one in the database as parameter to the WHERE clause

UPDATE Invoice Set Amount=@Amount WHERE InvoiceNumber=12345 and tstamp = @tstamp

If the data has not been updated by another worker since it was issued, then the update operation will modify the matching invoice number row, but if the message is duplicate, then the tstamp column will have a new value which will result in zero rows being updated (as no rows will satisfy the timestamp equality constraint). We can now process an infinite number of updates while remaining idempotent. Of course this technique would be a better approach for the simpler address change example that I provided above.

Clearly there is much more that can be said in this space, but that is all I have time for in this blog post. Look for additional advice and commentary in future posts.

Windows Communication Foundation Data Services (Astoria) – The Stuff They Should Have Told You Before You Started

2010-02-10T21:13:05Z

Entity Framework derived types support inheritance and relationships, just as you’d expect from any Object Relational Mapper tool (ORM)

Windows Communication Foundation Data Services (Astoria) throws an exception if your derived entities have relationships. That means if a Contact, a Lead, and a Doctor all inherit from a Person… and derived entities have different needs to link to external data… too bad!… Suddenly we have Sales Leads having properties for writing medical prescriptions! Good grief! That’s not going to be pretty… so ugly in fact… we probably don’t even want to go there…. The official prescription for this from the MS forums and various blogs is to move those relationships into the base Person class and then refactor your database storage as necessary…. perhaps having separate and distinct tables for Doctor, Lead, and Contact. Wait-a-sec you say… you picked an ORM because you didn’t want your Object Model to be dictated by your Data Model… well… that’s true… but here you find yourself anyway…

This bit me in the butt big-time on a project I was working on. There was nothing in the documentation and or the services behavior to lead me to believe that this constraint would be present… (well… okay… other than the initials CTP </grin> but still…) Knowledge of the constraint happens way too far along on the development cycle. We did not realize the presence of this constraint until after my team had invested heavily in development of our ORM and database models. These models worked well so long as we were only unit testing them. It wasn’t until we tried to expose them through Astoria that the big bomb dropped in the room.

Having the service tier force a rigid database implementation upon a team is the tail wagging the dog. Also... developer teams often do not have control over what a DBA may require in the database, and WCF Data Services should not be so brittle and constrictive as to disallow common use-case scenarios like practical and real-world use of inheritance. There is nothing complex about this business use-case… in fact I would say it is quite representative of the norm.

My friend Julie Lerman (author of Programming Entity Framework on O’Reilly Press) created a suggestion for this basic “feature” on the Microsoft Connect site. If you agree with its importance… please take a second to click the link and vote on it:

https://connect.microsoft.com/data/feedback/details/532592/derived-entites-should-be-allowed-to-have-relationships-in-wcf-data-services

Another item to be aware of regarding inheritance with Entity Framework: two derived types cannot share the same primary key. That means that the a Sales Lead cannot also be a Contact… thus forcing duplication of the Person and all the data related to that person (duplicate addresses, duplicate phone numbers, duplicate emails, etc.). There are no warnings in the designer of this snake in the grass… you’ll know the first time you attempt to retrieve data in the database where these conditions occur (a Person existing in more than one of the derived database tables). You can save the data this way… you can just never retrieve it after the fact! Again… this appears to be a very naïve constraint.

Migrating Polling Duplex Binding Element from Silverlight v2.0 to v3.0 for WCF Services In The Cloud Code Samples

2009-10-25T04:08:00Z

When running the WCF / Windows Azure samples (see my previous blog entry) you may get an error indicating that the Polling Duplex Binding Element cannot be loaded if you have Silverlight v3.0 installed (The type 'System.ServiceModel.Configuration.PollingDuplexElement, System.ServiceModel.PollingDuplex' registered for extension 'pollingDuplex' could not be loaded).

Replace the reference in the WcfSamples project from the v2.0 Silverlight Duplex Polling assembly C:\Program Files (x86)\Microsoft SDKs\Silverlight\v2.0\Libraries\Server\System.ServiceModel.PollingDuplex.dll with the v3.0 version: C:\Program Files (x86)\Microsoft SDKs\Silverlight\v3.0\Libraries\Server\System.ServiceModel.PollingDuplex.dll

Set the Copy Local property to true.

Code Samples for Hosting WCF Services In The Cloud

2009-10-22T22:51:26Z

Code samples demonstrating how to host WCF Services under Windows Azure can be found on the MSDN site at:

http://code.msdn.microsoft.com/wcfazure

HOWTO: Obtain and Configure A Free Certificate For Digitally Signing Your Outlook 2007 Email

2009-10-10T21:08:02Z

The protocols for authenticating the sender of an email have always been very weak. For the most part its an “honor” system, which is why spammers have such an easy time sending email with forged “from” addresses. We’ve needed effective email security for two decades now and one of the impediments to this has been the availability of digital certificates for email at an affordable price (e.g. free). In addition to providing us with the ability to authenticate a message’s sender, certificates also afford us confidentiality and integrity because they can be used to encrypt our message’s content and ensure that those same contents are not altered during transport.

Recently StartSSL began offering free trusted certificates which can be used for email and other purposes. The following set of instructions will show you how to get one of these certificates and configure Outlook 2007 to digitally sign your outgoing email.

NOTE: At this time, only Windows 7 will trust these certificates out-of-the-box; however, recipients who are running on Vista or XP can install the update for Windows Root Certificates (http://support.microsoft.com/kb/931125) to gain the same trust capability.

Use a web browser other than IE8… (I used Mozilla Firefox v3.5). IE8 on Windows 7 wouldn’t allow me to create the certificate.
Go to the StartSSL website at http://www.startssl.com/?app=1 (this is not an advertisement for these folks… you can get your certificate from any certificate provider, but this firm is currently offering them for free).
Click on the Sign-Up button
Accurately provide your name, address, country, phone number and email. StartSSL may invalidate your certificate if you don’t answer all the questions accurately. This is for everyone’s protection!
Click on continue
Receive a verification code via email; copy and past it into the verification form.
Select a “High Grade” certificate.
After the certificate has been generated, press “Install”.
From the Mozilla Tools menu select Options to get this dialog:
From the Mozilla Options dialog, select the Advanced Toolbar ribbon item.
From the Advanced Toolbar ribbon item, select the Encryption tab.
Click on “View Certificates” button to get the Certificate Manager dialog:
Find and highlight the SmartCom Free Certificate Member under SmartCom Ltd.
Click on the “Backup…” button.
Provide a password to protect your certificate file with. IMPORTANT: You will need to remember this password as you will not be able to use your exported certificate without it and there is no “recover password” capability.
WARNING: You should safeguard the certificate backup file by copying it off to a memory stick or DVD then storing it in a safe place and deleting the file from your computer’s hard drive. Anyone possessing this certificate file could potentially forge electronic correspondences in your name! You should not delete this file from your hard drive until after completing the rest of these instructions.
Run Outlook 2007.
Select Tools / Trust Center from the menu.
Select the E-Mail Security tab.
Check the Add digital signature to outgoing messages and the Send clear text signed messages when sending signed messages.
Click the Import/Export Digital ID button to get the Import/Export Digital ID dialog:
Click the “Browse” button and locate the digital signature file that you previously exported from the browser.
Provide the password that you used for exporting the digital signature and a friendly Digital ID name to identify it with (I suggest your email address or your name).
Press OK on the Import/Export Digital ID Dialog and you will be returned to the Trust Center Dialog. Press the “Settings…” button:
You will be taken to the “Change Security Settings” Dialog. Click on the “Choose” button to select a signing certificate:
Select the appropriate certificate from the “Windows Security” dialog box.
OPTIONAL: If you have more than one certificate, you can press on the “Click here to view certificate” link. Look for the “Subject” property on the Details tab of the Certificate Details dialog:
Press OK and you will receive the Importing a new private exchange key dialog:
I suggest keeping the default of Medium security; however, you can move the security up to High and this will require you to type in a password for each email that you want to digitally sign… this can be a pain, but it does help to reduce the likelihood that your digital signature might be used without your permission by a person at your keyboard or by a piece of malware. Press OK after you have made your selection, and them press OK again to close the Trust Center dialog.
Now that the configuration has been complete, you may send emails just as you normally used to. The only difference is that they will now be digitally signed:
When an email that has been digitally signed arrives, it will have a small icon just to the left of the paperclip (attachment) icon:
When you open a digitally signed email, you can see the certificate marker
To see an authentication of the certificate dialog, click on the certificate icon (circled). If the certificate is valid, this dialog will show you the message “Valid and Trusted” and the name of the person that sent the email to you. The actual certificate can be further inspected by clicking on the “Details…” button. You should keep the “Warn me about errors in digitally signed email before message opens” checkbox checked.
The details button will display the following dialog:
By making use of digital signatures in our emails we can significantly improve the trustworthiness of emails received through the Internet. Ultimately (assuming an eventual widespread adoption) this will significantly reduce spam and phishing attacks.

Setup Windows Identity Framework SDK Samples on Windows 7

2009-08-12T21:08:39Z

The setup and cleanup scripts that ship with the Windows Identity Framework SDK sample code do not properly operate under Windows 7. This is due to a check for the OS type that is used to determine if the scripts should use the netsh command or the older httpcfg command.

To correct this problem, simply modify both scripts and add “/c:6.1” to the findstr command. Do this for both the SamplesPreReqCleanup.bat and the SamplesPreReqSetup.bat files located in the Utilities folder as the Setup script calls the Cleanup script.

Google Notebooks and Bookmarks

2009-08-11T20:11:18Z

I really like Microsoft OneNote. It only stands to reason then that I would love the online Google Notebook application. Of course I do!… What’s not to love?! Nearly everyone that I know who has used the application loves it. If didn’t have a chance to add the application before it went away… you really missed out on an incredible productivity and organization booster…

Unfortunately, on January 14 of this year, Google pulled further development of it’s Notebook application stating that those using the application could continue, but that there would be no further enhancements and support for the browser extensions, which had concluded with the Google Toolbar 5.0 release http://googlenotebookblog.blogspot.com/2009/01/stopping-development-on-google-notebook.html). That wasn’t too bad by itself, but with the release of IE8… the ability to use the Browser Extension Object in the 5.0 toolbar that allowed me to right click and send material from my web browser directly to one of my Google Notebooks also went away… the 5.0 Toolbar won’t run under IE8! This was a terrible setback! I researched my options… but there really weren’t any good replacements that met my needs. I tried Zoho and Delicious as potential substitutes but nothing worked for me quite the way that I wanted. I am certain there are many people out there that find themselves in a similar situation and are absolutely beside themselves with dismay over why Google would kill such a great application. Others, who felt as I did even started a petition to bring Google Notebooks back (http://www.petitiononline.com/gnoteb/petition.html) I hope they are successful!

For those of you that were lucky enough to add the Notebook application but still miss the convenience and formatting of the context menu… here is what I do now… send the text to your Google Bookmarks application… then go to your Google Notebooks… the Bookmarks will listed as another Notebook (Yeah!!) and you can then move items from your Bookmarks into an appropriate Notebook. It takes an extra step, but it works good enough to get what I need done. Realizing that Bookmarks were just a scaled down version of Notebook… it really does boggle the mind as to why it went away…

This hasn’t been too bad, but it does demonstrates one of the worst fears of customers in relying upon “Software As A Service”… If a vendor changes their business model (or vanishes entirely) they can leave you high and dry. If the application is mission critical… they could have a very significant impact on your profits and even your survival.

Compiling and Installing AzureServicesManagement Tools on Vista

2009-07-08T04:49:00Z

Microsoft provides an MMC SnapIn for managing Azure Services as C# source code and a set of scripts. The tool allows the management and configuration of .NET Access Control Services, .NET Service Bus, and .NET Workflow Services [now on hold until .net 4.0 matures]. The download can be found here:

http://code.msdn.microsoft.com/AzureManagementTools

When following the Readme.docx instructions for compiling and installing the AzureServicesManagement Tools on Vista, the installation will fail. The error message will be swallowed and you won’t be told what failed because the script will continue execution past the error. Placing pause statements into the “build.cmd” script when an error occurs will help you to see what is going on:

Now when we run StartHere.cmd again, we can see the error… but we are still confused… here is what you will see:

To save you from pulling your hair out following this rabbit down the proverbial rabbit hole, the most important step that we need to accomplish is getting the compiled assemblies into the GAC. Visual Studio must run with elevated permissions because the AzureServices.Management project has two post build steps that run the InstallUtil utility. You will see this error if you open Visual Studio without Elevation and try to compile the solution directly:

To get the solution compiled and the assemblies you need registered in the GAC on Vista, run Visual Studio 2008 in Elevated mode.

Open the AzureServicesManagement.sln file from the Elevated (Administrator) Visual Studio 2008.

Compile the solution. This will cause the post build step to run the InstallUtil tool to install the compiled MMC SnapIn DLL into the GAC.

Exit Visual Studio and run the StartHere.cmd file as instructed in the Readme.docx instructions. The error will still occur, but it can be ignored because you have already registered the assemblies in the GAC. I did not verify where the “Debug|MCD” configuration was coming from as it seemed unnecessary to establish this work-around; however, if someone wants to take this a bit further, its my assumption is that the Microsoft supplied ConfigurationWizard.exe located in the Assets\Dependency folder is setting the configuration to the non-existent “Debug|MCD” value.

Add an additional PAUSE to the Install.cmd file in the scripts folder if you want to read the output of InstallUtil to ensure proper GAC registration (or simply check the InstallUtil.InstallLog file located in the bin/Debug folder).

You should now be able to use the AzureServicesManagement Tools as described in the Readme document by double clicking on the AzureServicesMMC.msc file in the AzureManagementTools directory.

One Small Landing, Another Giant Leap For Mankind

2008-05-26T02:43:00Z

I have vague memories of watching the Apollo moon mission in 1969 at age 6. I remember my father sitting with me on the sofa and telling me how history was being made as we watched Neil Armstrong take his first step on the moon. Later, we watched in high anticipation during the radio silence awaiting re-entry followed by the splashdown at sea of the space capsule. So here I am, a dad myself, sitting in front of my computer watching live streaming video on the internet with my two kids telling them much the same thing. In case you just "tuned in"… the fun is just beginning at NASA as pictures and other data begin arriving from the Phoenix Mars Lander which just touched down this evening (May 25). Although such an event can feel like background news because it doesn't seem nearly as extraordinary by today's technological standards or cinematic simulations. Sometimes we just have to slap ourselves a bit though and remember that this is really another planet we just landed on… and it is truly anything but ordinary. The information gained from Mars' geology and atmosphere will almost certainly prove invaluable as we study and compare our own planet in hopes of preserving its hospitability for humans many millennia from now. The very first pictures are up at: http://www.nasa.gov/mission_pages/phoenix/main/index.html

Vista SP1 for my 64-bit OS has fallen and it can’t get up!

2008-04-29T05:15:00Z

Vista SP1 for the 64-bit version of the OS has fallen on my computer, and it can’t get up! I’ve been without my primary computer now for several days due to the release of SP1 for Vista. The symptom is an infinite loop of installations and failures of SP1 (the specific details seem to vary by people reporting to be afflicted with this problem).

Nick White at Microsoft declares that the problem “affects a small number of customers in unique circumstances” on the Windows Vista blog here:

http://windowsvistablog.com/blogs/windowsvista/archive/2008/02/19/update-on-windows-vista-sp1-prerequisite-kb937287.aspx

but Google turns up nearly 36,000 hits for the pretty specific phrase of “configuring updates: stage 3 of 3 vista ultimate 64 sp1”; so I have to believe there are a few more customers being negatively impacted by this than just moi.

This blog was helpful in resolving the infinite patch installation loop in case you happen to get caught by this nasty thing:

http://forums.microsoft.com/technet/showpost.aspx?postid=2873378&isthread=false&siteid=17&authhash=7f2056e7cf93af49d9d8704602bc97be8b4c437b&ticks=633390464558901682&sb=0&d=1&at=7&ft=11&tf=0&pageid=5

It doesn’t “solve” the problem, but it will at least make your computer useable again for awhile so that you can backup your important files, etc. In my case whenever I install a new IIS feature, the pattern of installing/failing SP1 starts all over as soon as I reboot my machine. Everything having to do with attempting to resolve this problem seems to take hours of time (SP1 itself takes nearly an hour to install and then another hour after it aborts to rewind itself). KB937287 has been installed on my machine 5 times now! After SP1 fails in its installation, KB937287 shows back up again in Windows Update. KB937287 is apparently a pre-requisite patch to prepare your computer for the slaughter that is SP1.

I spent nearly two hours on an IM with Microsoft tech support this evening where they took remote control of my computer and declared that the problem was with my Norton 360 antivirus interfering with the service pack installation (even though my anti-virus had already been disabled per the SP instructions). They asked me to remove my anti-virus and re-install SP1 again and then scheduled me for a call back tomorrow. Unfortunately the results were an unsuprising repeat of the “configuring updates: state 3 of 3“ hell that was my weekend (it’s like groundhogs day all over again)! At this point I’m thinking I would be better off simply burning my computer to the ground and reinstalling the OS without the automatic patch feature disabled until MS can get this sorted out. In point of fact, this is exactly what many customers seem to be blogging that they did. It may only be affecting a small number of customers, but the impact to those minority customers seems severe.

Security Implications Of Services Impersonating Callers

2007-12-23T05:47:00Z

In my last post (Caller Impersonation for WCF Services Hosted Under IIS Appears Broken), I laid out my rationale for why I felt that the security of services impersonating a caller when hosted under IIS was broken. To be responsible, I feel it necessary to follow-up my previous assertion by noting that such a configuration is not a best-practice, even though many corporate staff developers may be tempted to secure their intranet services this way.

Allowing a service to impersonate the caller’s identity requires that the caller have a high degree of trust in the service that he or she is interacting with. Of course it’s very unusual for a user to be aware of the services their applications interact with, so this is a completely unrealistic expectation. When a service is configured to impersonate its callers, it could potentially perform clandestine operations within a network using authorizations granted to the caller’s identity without the caller being aware.

To further illuminate the security vulnerabilities of callers trusting services to impersonate their identities, it’s important to note that most services are focused on a specific area of business while application users typically have much broader and deeper sets of authorizations. A malicious service’s reach is magnified when the impersonated user’s identity happens to be trusted by other entities or business domains unrelated to the service.

Smaller organizations often take for granted that local intranet services are not doing malicious things using the caller’s identity; however, it’s likely not the best choice for security conscious intranet service writers to make. Even if you have complete trust in your own internal services, consider the potential for their unintended leveraged use by a malicious third party. In my opinion, it’s much safer for a service to act with its own identity rather than to impersonate the caller’s. Services can be run using an identity that has been granted the least privilege necessary for it to perform its functions. The “less is more” approach to security is considered a best practice, and should be followed whenever practical.

To summarize, even though I still consider caller impersonation under IIS to be defective, some may consider this to be an important security feature! Although a bug is still a bug no matter how much lipstick we slap on it, I have to agree that this particular bug might actually force better security choices to be made.

Caller Impersonation for WCF Services Hosted Under IIS Appears Broken

2007-12-17T02:26:00Z

There is a security feature of WCF services hosted under IIS that I find poorly implemented. In all honesty, it appears to be broken and non-compliant with its intended purpose. If you’re developing services for use in the intranet environment, then it’s quite reasonable for you to expect that a service can impersonate your Windows identity while it performs its work. After all, security personnel have no doubt already established your intranet authentication and authorization policies for corporate assets, and internal service security should be able to fit within this established paradigm.

Unfortunately, WCF will throw an erroneously worded exception about your attempt to use anonymous access when the IIS hosted service that performs caller impersonation is set to require a Windows identity (presumably a bug of assuming that “not equal to 1” identity means “equal to 0” identities). The reason why you likely had to read the first sentence more than once is that the error message complains about your use of Anonymous when you turn Windows security on, but turning Windows off and Anonymous on makes the error about using Anonymous go away! Confused by the apparent contradiction? Well, so was I when I first encountered it.

In my opinion, the recommendations from Microsoft for working around this limitation are completely idealized and wholly unrealistic. Thanks very much to Wenlong Dong for explaining the problem in his blog entry “Impersonation with Double Identities”; unfortunately, his blog post never addresses the fact that the two identities should be being used for different work in the transportation and consumption of a message (or perhaps he simply disagrees with my assertions).

Here's the logic for my assertions:

When a postal employee delivers a letter we can make some reasonable assumptions about the security of the letter’s transportation. It is safe to assume that the post office took measures to ensure that our letter was not stolen, snooped, or tampered with during their possession. This is transport security.

Now the postal person has his or her own identity. My postal delivery person’s name is Ralph (a pseudonym I’m using to protect Ralph’s true nefarious identity). I have my own identity, and my name is Paul (which is not a pseudonym, just in case you were wondering). Although Ralph seems like an okay person to me, I’m not ready to invite him into my home whenever he delivers letters addressed to me. I just want Ralph to leave my letters and then go on to his next delivery as quickly as he can. I would be extremely irritated (you might even say that I would “go postal”) if Ralph routinely handcuffed himself to my letters and insisted upon helping me open them.

My friend Russ, (a character I borrowed from Justin Smith’s WCF book), waits in line at a very busy U.S. Post Office. It’s the holiday season and he tires quickly of waiting so long so he decides to send my letter via UPS instead. While driving to the UPS office, Russ discovers that there is a Federal Express office on the way with no waiting line! Russ changes his mind again and immediately decides to send it to me via FedEx. No matter which carrier Russ decides to use, he reasonably expects that his letter to me will be secured until it can be delivered. To differentiate themselves from their competitors, some carriers may have stronger security than others; however, they all perform the same generic service of transportation.

It turns out that Russ is somewhat paranoid that the Department of Homeland Security might be interested in reading his private messages about family protected secret soup recipes and he decides to send his letter to me using some very strong public-private key encryption so that only I can read his message. Russ can now feel confident that if his letter is somehow intercepted that its contents for cooking a very delicious mulagatani will remain confidential. In short, Russ has applied message level security to his communication with me, while the mail carrier has applied transportation security.

If you agree with my analogies then you’re almost certainly hungry. You also likely agree that there are two separate and distinct identities that perform work during the lifetime of a message. There is the identity of the letter transportation carrier (e.g. Federal Express, U.P.S., or U.S.P.S.) and there is the identity of the message recipient. Both identities are responsible for their own distinct roles, and neither is responsible for accomplishing the work of the other.

Wenlong’s blogged solutions do not make a distinction between the postal worker’s identity and the recipient’s identity—they are just two non-descript identities stuffed into a single container and WCF cannot seem to figure out which one it should use for the message security.

The number one recommendation in Wenlong’s blog is to forget the entire matter altogether, and change your IIS security settings to “Anonymous”. In simple terms, Microsoft is asking us to turn transport security off and then depend entirely on message security (or no security). When making this decision, keep in mind that your network administrator cannot easily monitor or prove that message security actually exists or immediately prove its absence if it inadvertently gets turned off. Message security also does not protect the host from malicious anonymous traffic. Administrators likely won’t agree to this, and they may come along at some future time and lock your sweet little anonymous service down as a non-compliant and potentially hazardous risk to their corporate assets. As developers generally do not have carte blanche in networks to do whatever they want, and network administrators already have established security policies which work well for existing internal “asmx” web services, the “solution” of using anonymous is simply dead-on-arrival. Wenlong warns that Microsoft recommends against our use of multiple identities; however, I would argue that good security is generally layered and it certainly makes good sense to me that we would want to secure both our transports and our messages.

The second recommendation for working around this limitation is to have the service impersonate its own account instead of the caller’s account (as if the two could somehow be considered equivalent). Of course, impersonating the user of the service and impersonating the same account for all users of a service are entirely different security models, and it’s unlikely that such a solution would make the network administrators much happier than the “message security only” approach.

This leaves us with the last proposed solution of removing the transport identity from the evaluationContext.Properties["Identities"] collection. Ah… but there are two identities in the collection and you have a 50/50 chance of removing the right one! The problem here is that placing the two identities into the same collection without any way to determine the role that the identities are supposed to play is completely silly. Creating a production application that has to guess which identity to use seems like a very brittle and potentially hazardous approach to security—but it’s the only one we seem to have available when impersonation of a Windows caller is a requirement. In point of fact, why do we have a collection of identities anyway if we have no way of determining the purposes of the collection’s occupants?

This “double identity” problem existed with the .net 3.0 framework, and I’m sorry to report that it still remains in 3.5. If your network administrators aren't too concerned about an IIS endpoint being set to allow anonymous access, Sanjay Antony's blog entry may be useful.

Ten Tricky Timer Testing Tips

2007-10-16T18:48:00Z

The following are tips for testing Windows Workflow Foundation instances that contain delay activities (timers) when used in conjunction with a passivation store. This list of tips is certainly not exhaustive, but I believe that I’ve accumulated enough useful techniques to warrant sharing with others.

1. Purge all rows from your InstanceState table that may contain passivated workflows of the type you’re testing.

2. In order to support your ability to conveniently blow away InstanceState rows, each developer should have his or her own private persistence database. If possible, avoid using a shared database for this type of testing.

3. For testing (and perhaps even production) consider merging your passivation store with your application database to help facilitate referential integrity. You will undoubtedly have an associative table to map workflow instance ID’s against your application’s key field(s). This is not Microsoft's recommended production deployment strategy, but it is a useful configuration to utilize during development and debugging.

4. It’s impractical to test timers with intervals that are set to days, weeks or months—so allow the ability in your application design to modify these intervals in a data driven fashion.

5. Don’t test with time intervals that are too small. The default poling interval for determining expired timers in need of service is 2 minutes. See #10 below to adjust.

6. Don’t bump the poling interval for expired workflows down too low as you can create scenarios where you are poling faster than you are processing workflows or where you don’t have enough time to operate the debugger before the next timer cycle begins.

7. Be aware of the multi-threaded nature of the code you are working with. Timers will cause passivated workflows to rehydrate and begin execution upon timer expiration while you may be busy debugging other workflow instances. If you haven’t removed rows from your InstanceState table you may find yourself simultaneously debugging multiple workflow instances—which is okay if that was your intention, but it generally is not.

8. Add this.WorkflowInstanceId to your diagnostic output (Trace.WriteLine and friends) so that you’re aware of which workflow instance is responsible for the diagnostic output being generated. When multiple workflows begin executing at once, you’ll be glad you have this. You may also consider adding the managed thread id as well.

9. If you’re workflow operates on a shared resource, use a SychronizedScope container activity to protect access to it. This is true even for workflows that do not use delay activities, but it is mentioned here for completeness.

10. You can adjust the timer poling interval using the LoadIntervalSeconds attribute of the System.Workflow.Runtime.Hosting.SqlWorkflowPersistenceService configuration entry.

A Workflow Error One Should Never See In A Dialog

2007-08-23T04:59:00Z

I was surprised to receive a "Property value is not valid" error dialog when assigning a Ruleset to a Policy activity. There was nothing particularly unusual about this action other than the fact that the workflow itself seems to be pressing the limits of what my poor 1.7 Mhz Centrino Duo CPU with 2-gig of RAM can handle (350+ activities, 1000+ rules). The dialog box wasn't my biggest surprise though… upon clicking the details button, I discovered that the source of the error was being reported was an "OutOfMemoryException!" egads! This would seem to imply that the WF designer is catching and eating all exceptions from this property setting--and assuming that any and all exceptions are a "Property value is not valid" error. Tisk, tisk , tisk! Bad code monkey! Bad!