The MarketWatch site has recently recently been the target of a massive renovation project. A highlight of some new features from an editorial perspective can be found at the Site tour, but I wanted to take some time to talk about it from a technology point of view.

The initial work for engineering began some time last year, approximately December, we began to create systems known to be necessary for the functionality that would be eventually required. The middle of the first quarter saw the completion of the joint efforts of the design agency and business stakeholders in the form of new IA/UX and design. With an ultimate target date of May 12th I’m sure you can appreciate the fact a huge number of things about the implementation needed to go very well.

Many people on several teams were involved in it’s creation. I can’t possibly name everyone involved but I wanted to recognize a few people who acted in a technical leadership capacity.

Kevin Dotzenrod, Architect and Engineer Manager, is a good friend I’ve worked with many times before. He was instrumental in the success of the project in every way from the original solution design to final profiling and field integration.

Donn Felker, Senior Consultant with Magenic, is a developer I was working with for the first time but respect greatly. His fluency in the art modern software development, determination, and ability to knock down complex challenges is incomparable.

And the team leads, Kevin Boedigheimer, Peter Gaard, Bob Galuska, Ben Music, and Phil Whipps, have done a spectacular job of engineering individually as well as ensuring the people on their teams could be successful.

But enough cheerleading. :)

Platform

We made the decision in the very beginning to shift onto IIS7 on the Windows Server 2008 64-bit platform. Some notable technologies used in the site’s creation include ASP.NET MVC, Spark, WCF, Windsor container, and Active Record over NHibernate.

That may seem like a risky combination, but the ASP.NET MVC web framework was the only technology we hadn’t used in production previously. Everything else had a proven track record of stability and ease of use. Because we were transitioning from MonoRail+Spark to MVC+Spark even that change didn’t involve any differences in concepts and patterns. The learning curve was limited to the differences between in controllers, routing, and html helper implementations. The extensibility of System.Web.Mvc and System.Web.Routing was particularly impressive. With the ability to replace a service at exactly the correct place you can take command of your controller creation to introduce IoC, or your take control of the action/result execution process to inject site-wide filters or policy.

Although the technology was understood a few rough patches for productivity came during the earlier stages of development, which is why you always want to keep an eye out for opportunities to adapt and simplify. A large number of people were working in parallel, and once certain critical superstructure was in place the velocity of development was astonishing. One practice we adopted which worked quite well was to establish a canary deployment as part the last step of a successful build. Not to be confused with qa deployment, or staging for production, or a snapshot for internal demos - the job of the canary (in the coal mine) is to die first when there’s a new problem. It’s used as a shared resource to verify correctness of recent changes and the current state of the overall project.

Bluegrass

There’s no time to dive into all of the different aspects of the implementation but one I wanted to highlight specifically is Bluegrass. For a long time MarketWatch has had a LiveQuotes feature, which evolved into LiveSuite as news was introduced, both of which are now deprecated with with the deployment of Bluegrass - our asynchronous, comet-like distribution framework to deliver instantaneous json data and event information. The client forms at-most-one connection with the a message broker which either returns queued messages or enters an efficient aync wait state on the server until the arrival of messages for that particular caller.

The brokers themselves are ignorant of the meaning of the messages, a change from the design of LiveQuotes and LiveSuite, which will enable us to provide additional channels of information without updating and redeploying those systems. The scale at which they operate is also unprecedented in our organization. The servers will each process 3700 requests a second at sixty percent cpu (dual quad-cores) and will top out at six thousand. And that’s while each server averages over ten thousand concurrent requests executing as several calls to pickup are “on hold” pending the arrival of any messages for the caller.

Windsor

For the implementation of the web site itself there were many reasons to keep it simple and direct. Windsor is an Inversion of Control (IoC) container which are very important to understand if you haven’t used such a thing before. It’s impossible to describe why you would want to use an IoC container other than to describe it’s effect with a metaphor. Imagine in your mind the classic surgery scene - the surgeon says “scalpel” and whap she has one in her hand, properly sterilized and inventoried pre-op according to policy, so all she does is indicates the need and uses the tool. “Clamp” and whap she’s clamping. Doesn’t even glance away from the surgery site.

The effect this has on your code is dramatic. As you’re writing a controller it has a task at hand to worry about. It’s concern is receiving actions and their parameters and policy, composing view data, returning the appropriate type of result in response. The last thing you need is a lot of code, and the associated knowledge, for create and initializing the services it needs. Just as a surgeon doesn’t sterilize and inventory the O.R. equiptment she’ll be using, an IoC container enables your controller’s constructor to say “IQuotes,” “IHeadlines,” “IComments,” and have the appropriately initialized and lifecycled implemenations of those service components slapped into it’s hands ready to use.

It’s a beautiful thing and really helps you abstract away the complexity of interconnectedness with very little effort on what is a fairly large number of components coming together to service a request. We had declared a base IService interface to automatically enlisted the descendant components into the IoC container and that convention eliminated all configuration in the majority of cases.

There’s more I could go on about - but this is blog, not a short story, so at this point I believe I’ll thank you for reading this far and leave it at that. :)