On the afternoon of Monday, April 7 2014, the Heartbleed bug was made public and demonstrated that the majority of encrypted traffic on the Internet was vulnerable to eavesdropping.
In this post we wanted to provide insight into the steps Acquia took to handle this event. First, though, we want to state:
- Acquia’s infrastructure is now safe and hardened
- Acquia has been working to ensure our customers’ Drupal sites are secure no matter how or where they use Drupal
- Mitigating Heartbleed involves more than just software updates or server patching
- We are working with our customers to ensure that their sites are as secure as possible
Handling the crisis
After learning of the vulnerability, we immediately recognized the severity of the OpenSSL issue and activated our incident response plan. This kicked off internal procedures to begin mitigating the vulnerability. This is where our internal incident response plan paid off, as our preparation and training allowed us to act fast and methodically. Our team ensured in parallel tracks that 1) our infrastructure was secure 2) our partners’ infrastructure was secure 3) our customers were well informed of the event and their responsibilities were understood.
Our goal was to ensure that all of our customers across all of our product lines were protected. This included customers across our Acquia Cloud, Acquia Cloud Enterprise and Acquia Cloud Site Factory product lines, as well as the support we provide for Drupal sites wherever they may run. Hundreds of thousands of Drupal sites hosted the world-over depend on us for security and resiliency best practices and we knew we had to provide excellent support during this crisis.
A realistic and adaptable incident response plan is crucial to dealing with events such as Heartbleed. The plan has to be comprehensive enough that it ensures you are hitting every detail and flexible enough that it can be adapted to any situation. Having pre-determined chat rooms, dial-in conference bridges and a communication plan ready to roll gave us a jump start. A plan that is not comprehensive or is out of date may do more harm than good, however. Acquia’s rigor in internal compliance and external auditing through efforts such as SSAE-16, PCI and FedRAMP initiatives ensured the plan was effective, allowed us to formulate our approach and put it in motion very quickly.
Eradicating the Heartbleed bug
Acquia terminates SSL for its customers using two mechanisms: nginx and Amazon Elastic Load Balancers (ELBs). We secured nginx in our infrastructure by building a new version with an updated OpenSSL and deploying it to our cloud servers. Our automated build system and tooling for deploying updates to servers in parallel allowed us to efficiently push the updates out to the affected servers. This was key in our ability to ensure all 8000+ instances we run were hardened against the attack. At this point Acquia Cloud servers were effectively secured from the outside world against the vulnerability.
Acquia Cloud is built on the Amazon Web Services (AWS) infrastructure and Acquia worked closely throughout this entire event with the AWS team through our strategic partnership in ensuring that all Acquia resources were secured. Throughout this period we worked with AWS to ensure the vulnerability was resolved on our entire ELB fleet. To Amazon’s great credit, the ELB update process was completed without service disruption, demonstrating the power of the cloud overall.
Additionally, we worked with partners such as Akamai to ensure that customers who are fronted by Akamai were not vulnerable. Although the Akamai infrastructure was secured at the time the Heartbleed bug was released, there was still the chance that Heartbleed was exploited prior to being announced. Thus, Acquia and Akamai worked to inform our joint customers about the vulnerability and recommend they deploy new rekeyed SSL certificates to the Akamai infrastructure.
How we supported our customers
A large part of fully mitigating Heartbleed is ensuring that SSL certificates are rekeyed and deployed. As part of our response plan we created a guide to help our customers understand how they can easily ensure their sites are secure. However, Acquia knew that a guide was not enough. We also directly reached out to many customers to assist them in the process.
Acquia runs hundreds of thousand of Drupal sites across our product lines. Most of those sites can be updated relatively easily because they utilize wildcard SSL certificates. In order to meet the challenges of upgrading thousands of enterprise production sites, Acquia improved the user experience of our SSL management interface and worked hard to help our customers update their certificates. Our 24/7 support and operations capabilities and global coverage enabled both broad and targeted support to ensure customer success.
Next Steps
We will continue to work with our customers who have not yet updated their SSL certificates or followed our other recommendations to ensure that they are protected. Security is a shared responsibility between infrastructure providers, platform providers, website creators and maintainers and, indeed, the open source community which powers most of the Internet. We believe that an Open Cloud Platform improves security and reliability for all and we are committed to ensuring our customers stay protected and resilient.
