In November, 2014 Weather.com launched on Drupal and became one of the highest trafficked websites in the world to launch on an open-source content management system (CMS). Mediacurrent and Acquia are excited to announce a new, 3-part blog post series that will share insight around how Weather.com was migrated to Drupal. Our team of experts will share best practices and what lessons we learned during the project.
There's an old saying, “Everyone talks about the weather, but nobody does anything about it.” While we are a long way from controlling the weather, Weather.com has done a spectacular job of delivering accurate weather news, as rapidly as possible, to all kinds of devices.
This is a small miracle, especially when you consider Weather.com served up a billion requests during its busiest week. Even slow weeks require delivering hundreds of dynamic maps and streaming video to at least 30 million unique users in over three million forecast locations. The site has to remain stable with instantaneous page loads and 100 percent uptime, despite traffic bumps of up to 300 percent during bad weather.
Page load times are the key to their business and their growth. When The Weather Channel's legacy CMS showed signs of strain, they came to Drupal.
On their legacy platform, Weather.com was tethered to a 50 percent cache efficiency. Their app servers were taking on far too much of the work. The legacy platform ran on 144 origin servers across three data centers. It takes all that muscle to keep up with the number of changes that are constantly happening across the site.
Traditionally, when you have a highly trafficked site, you put a content delivery network (CDN) in front of it and call it a day. The very first time a page is requested, the CDN fetches it from the origin server and then caches it to serve to all future requestors.
Unfortunately, it doesn't work that way for a site like Weather.com.
Consider this: If a user in Austin visits a forecast page, they see a certain version of that page. A visitor from Houston sees a slightly different version of that page. Not only are there two different versions of the page, one for each location, but much of the information on the page is only valid for about five minutes.
At the scale of three million locations, that's a lot of pages that have to rebuild on an ongoing basis only to be cached for 5 minutes each. Couple this with the fact that the number of served locations kept increasing as developers worked on the site, and you can see that things are rapidly getting out of control.
The first thing we did was break up the page into pieces that have longer or shorter life spans based on the time-sensitivity of the content. That allowed us to identify the parts of the pages that were able to live longest and that we could serve to the majority of users. The parts that varied, we no longer change on the origin servers, but instead delegate to systems closer to the user where they actually vary.
To accomplish that trick, we switched to a service-oriented architecture and client side rendering, using Angular.js, ESI (Edge Side Includes), and some Drupal magic. The combination of these three components boosted cache efficiency, page performance, and reduced the required number of servers to deliver it.
The result? After launch, we showed Weather.com a 90 percent cache efficiency. In other words, in going from 50 to 90% cache efficiency they reduced the number of hits to the origin servers, which means that you need fewer of them. Post launch, we were able to increase cache efficiency even further.
This cache efficiency was also measured only at the edge. Varnish (a caching proxy) further reduced the amount of traffic, meaning that Drupal itself and the Varnish stack were serving less than 4 percent of their requested traffic. The benefits of the service-oriented architecture also mean that scaling is simpler, architectural changes are less painful, and the end user can experience a richer user experience.
Doing something about the weather is still way out on the horizon, but Weather.com can certainly claim that it has improved the delivery of weather news.
