2010 Winter Olympics

2010 Winter Olympics Silverlight Smooth Streaming Health Monitoring System

During the Vancouver Olympics in February 2010, Microsoft Corporation and NBC provided live Silverlight video streaming for the Vancouver Olympics. This effort involved a dedicated data center (“Olympic Origins servers”) combined with Internet Information Server (IIS) smooth streaming servers to provide up to 30 concurrent live video streams. The system generated over 12 petabytes of data, and at peak loads streamed 374 gigabits of video per second and handled 2.4 million pages per second. With 82 million mobile page views, 1.9 million mobile video streams, and 5,000 hours of live and on-demand video, the Olympics streaming video system was a wonderful demonstration of Microsoft technology and media platforms. Wintellect supported this effort by creating the health monitoring tools to monitor and ensure the reliability of the overall system.

Microsoft required a strong level of high availability in the data center to ensure uninterrupted streaming during the Olympics. This data center was to be responsible for receiving live video data from the Olympics, encoding the data for the Silverlight smooth streaming format, and then publishing the data via a live circuit to a content delivery network (CDN) that then streamed the video to end users. To achieve this high availability, multiple primary and secondary failover server clusters were dedicated to groups of incoming and outgoing streams, as well as to the archive or “video on demand” content.

Microsoft needed a way to quickly and effectively monitor the health of this system in real time. Such a system would provide instant alerts when any system parameter that might impact delivery (including network bandwidth, disk space, CPU utilization, etc.) reached certain defined thresholds. Also needed was the ability to react in real time to problem situations by updating or pushing out new server configuration files. The data center was housed in a highly secured environment, and the solution needed to provide secure, encrypted, authenticated and centralized access to monitor and configure all remote servers in the environment.

Microsoft, already familiar with the Wintellect brand, decided to outsource the development of the health monitoring and configuration management portion of the effort. Wintellect, a Microsoft Gold Certified Partner and Silverlight Alliance Partner, had the experience with WCF, System Center Operations Manager (SCOM), and Silverlight to deliver the product within the timeframe desired by Microsoft.

Microsoft Systems Center Operations Manager, or SCOM, was selected as the base tool for monitoring the overall environment. Smooth Streaming was a new technology at this time, and so custom SCOM management packs were developed for the SCOM system to interact with the Smooth Streaming APIs and provide the required additional performance counters in real time. In order to monitor the health of the system and anticipate potential configuration adjustments, Silverlight was used to create a configuration and monitoring application. This application would parse server XML configuration files, prompt the user for needed edits, and then batch these changes for publication to multiple remote servers.

Windows Communication Foundation (WCF) was used to facilitate communication between the Silverlight application and remote agents that were installed on each server. These provided secure encrypted links by using common protocols such as HTTPS/SSL that were firewall –friendly in the data center environment.

The overall health monitoring system, including the WCF agents and Silverlight application, were rapidly constructed due to the flexibility and capabilities offered by XAML, Silverlight and WCF. The native Silverlight support for LINQ-to-XML made it easy to parse XML documents with unknown schemas, and then use the powerful XAML-based layout mechanisms available to provide visual views of the configurations for editing and updating. The Composite Application Library, commonly known as PRISM, was utilized to design the Health Monitoring application in a modular form in order to keep the browser footprint lightweight by dynamically loading only the modules currently needed by the operator.

The full solution was completed ahead of time and under budget within a fixed-price delivery schedule. This enabled the team to provide additional functionality such as a synchronization monitor that would scan replication servers and generate detailed reports if the file systems were out of sync. Silverlight’s data-binding model also made it easy to generate hundreds of unit tests to support the quality of the product in conjunction with its rapid delivery.