Peter Wolanin

Senior Application Architect

Peter Wolanin, Principal Cloud Security Engineer, had a previous life as a Senior Scientist before making the leap into working with Drupal, and eventually at Acquia. A Drupal code contributor since May 2006, and a listed Drupal core subsystem maintainer since November 2008, Peter has worked on helping many of Acquia's products and services throughout his tenure at Acquia, while continuing to contribute to Drupal and helping promote it by organizing meetups and DrupalCamp New Jersey, and by speaking at many other events. Most significant D8 contribution I was not working heavily on Drupal 8 until DrupalCon Portland. At the extended sprints in the Acquia office after the conference I worked on getting up to speed on the new plugin system and routing systems. That led to envisioning and working on executing the conversion of menu local tasks, actions, and contextual links to plugins. I also helped push the conversions from page callbacks to routes and controllers. However, I think my most significant single contribution came out of Drupal Developer Days in Szeged, Hungary. Towards the end of the event (after a week straight sprinting on Drupal 8 beta-blocking bugs) I developed a new idea as to how we could solve all the outstanding problems with the menu link system. Daniel Wehner and I spent about half a year executing on that vision with help from lots of other people <a href="https://www.drupal.org/node/2256497">https://www.drupal.org/node/2256497</a>. At one point the main patch grew to 617 kB (17,920 lines) before it was broken to make it more reviewable! Biggest challenge with D8 It was pretty hard to find enough time to get it done, and at times there was a lot of effort that went into just keeping communication open and complex, multi-part plans moving forward. Looking forward to most about D8 I'm eager for the release so I can start building fun modules and find time to port some of of my existing modules. I am really happy about the conversion of code to use a mostly OO approach. While the number of files increased, it actually makes the code a lot more fun to work with and write.

Minimizing maintenance time while updating thousands of Acquia Cloud Site Factory sites

January 18, 2016
6 minute read

Note (January 2016): While this blog post was originally written about drupalgardens.com (originally published April 06, 2010), the same basic strategy for updating sites has continued to be used in the Acquia Cloud Site Factory (ACSF) platform that emerged from it. Some of the very specific details in the original post are outdated (for example, we now use git instead of svn), but the strategy of using two docroots with different code versions and two different vhost ports to choose between them is how ACSF still works to update sites.

In his Drupal Gardens 2 month update Chris wrote about a number of the improvements to Drupal Gardens in our last sprint (see archived blog post). I want to focus on our solution for a system that now lets us perform Gardens site upgrades with only a couple minutes of maintenance (site offline) time per site when running database updates. The problem was made a little more difficult by the custom domain feature that was also going live, so each Drupal site (database) might be referenced by multiple sites directories. I describe here the solution we worked out, which involves using Drush, the (internal) Acquia Hosting APIs, and communication with the http://www.drupalgardens.com/ site so that each site in our multi-site installs could be seamlessly moved between two different versions of Drupal 7.

Prior to this new system, the early beta testers sometimes experienced a fairly long period of their site being in maintenance mode during any Drupal Gardens upgrades. The reason for potential delays before is that we are using Drupal's multi-site feature to run thousands of Drupal Gardens sites off the same code base. The multi-site configuration gives us better PHP opcode performance and we also decided it was the easiest configuration for us to maintain. However, when we deployed new code, the safest approach was to take all sites offline since they all started immediately using the new code but their databases were not yet updated.

Drupal Gardens is deployed on top of Acquia Hosting in a high-availability configuration involving (at the moment) each site being served by redundant load balancers, three or more web servers, and a cluster of database servers. At the time of this project, the Aegir Hosting and Provision projects were not suited to work in such an environment, so we have our own system that uses Acquia Hosting's internal API calls to provision new databases, and Drupal Gardens-specific code to deploy them. However, Drush (an integral component of Aegir) was the obvious tool to use to run database updates and other commands across a large number of sites.

For each multi-site installation the load balancer has two name-based virtual host (vhost) configurations. Each of those vhosts directs requests to the same set of Apache web servers, but on different ports. Apache is configured with a vhost for each port that has a different document root (docroot). So, a host name that matches vhost #1 on the load balancer will use the Drupal code present in docroot #1 on the web server.

Having in place an easy way to route requests based on the host name to a different docroot is the key to speeding the process. Obviously there are other ways we could have achieved this including name-based vhosts on the web servers, or virtual document root configurations for the web servers, but by limiting the configuration change to the load balancer we get faster changes that are easier to verify, and it means that we can easily have the two docroots on different web servers if desired.

From there the rest of the process is mostly coordination, including a few Drush commands run from within docroot #1 on one of the web servers.

Update the svn repository for the Drupal Gardens code and tag the new code.
Prepare a drushrc file so that there is just one alias for each Drupal Gardens site, even if it has multiple hostnames configured. The information for this comes via an API call to http://www.drupalgardens.com/ that returns a list of all sites and the associated host names for each.
A custom Drush command calls the hosting API to move all domain names from vhost #1 to vhost #2 on the load balancer. This means they all use docroot #2 which has the current code.
Switch docroot #1 to use the updated code corresponding to the tag in svn.
Run a wrapper script that iterates through the site aliases and for each site invokes:
1. Custom Drush command puts the site into maintenance mode plus causes a message to appear for the site owner with estimated offline time.
2. Custom Drush command calls the hosting API to move the domain names for one site from vhost #2 to vhost #1, and hence request go to docroot #1 with the new code.
3. Standard Drush commands run the database updates and then clear all caches
4. Custom Drush command insures the site is out of maintenance mode.
The output of all these steps and whether they succeeded or registered any failures is logged in case follow-up action is needed.
Finally, switch docroot #2 to the new code to be ready to start the cycle again.

Keeping current with Drupal code and database updates is key for the long-term success of any Drupal project. As we scale Drupal Gardens from thousands of sites today, to hundreds of thousands and maybe to millions, we will continue to refine approaches like this that minimize the impact of maintenance on Drupal Gardens sites and users.