After I joined Acquia in January, I was given the task of assisting in the port of the Acquia.com/Acquia Network web properties to Drupal 7. Although other people were involved in the process, including Jakub Suchy and his team, there was a lot that needed to be ported. One such item was the Acquia Network Connector module, which links a website to the Acquia Network and passes website data after it validates its keys. On one fateful gardening day, I began the task of porting its existing code to Drupal 7, and I found myself quite impressed with its potential.
The Acquia Network Connector module was designed specifically for users running Acquia Drupal, and only those whose core distributions were unmodified. It wasn’t long before Acquia decided to expand the module’s scope by supporting any Drupal distro or code (contrib or custom). This caused some very impressive features in the Connector to have less value, and to stagnate over time. During the port process, I examined these portions of the code and realized that the Connector could be enhanced by exposing more website information to website owners and to our Support team.
I rebuilt the server side of the Connector on a Drupal 7 site, and then created an accompanying analysis site. I used separate sites because I’ve learned that using the same website/codebase for both the back-end processing system and the user display site was a bad idea. Separating the two makes for a more robust configuration. But how do you synchronize them? Answering this question proved to be nearly as large a task as creating Insight.
The biggest problem was that the implemented architecture involved no fewer than 7 different websites! Websites running Drupal 5, Drupal 6, and Drupal 7 all needed up-to-date information on customer accounts, roles, access, and products. Oh... and they needed a single-sign-on between them all. After I looked at it, and I talked to the other big brains at Acquia, I decided on a mixture of the Services module and Bakery. We also created the Services Client module, which is an arbitrary client to services that can be used to synchronize data between sites. I won’t use this post to get into the nitty-gritty, but I think that this architecture can be used to power interconnected systems and sites in most enterprise environments.
Using the new architecture, when content is updated on one site (for example, a user’s Acquia Network subscription), the changes are sent to other sites by passing them through a master data warehousing site. Because each site has a full, local copy of the data they need, the system doesn’t have a dependency on any other particular site.
At the same time (caffeine is helpful for multitasking) that I was making incremental progress on this enterprise data management setup, I was working on the Acquia Connector. The key was a system of passing a larger dataset between the connector and the server. Ultimately, I was able to have the connector pass a large amount of data from a client site back to our servers, including hashes of all of the site’s modules and files. We then parse and store the data, which is used by the Insight engine to rate sites, and provide recommendations for website administrators.
We use the data for two different website grades. The first is its Insight grade, which denotes a site’s performance, security, and adherence to Drupal best practices. While the number of tests can differ based on your installed version of Drupal, at present, a website is evaluated against approximately 50 tests. The second grade is a website’s SEO Grader rating, which has five separate testing areas. A given website can see as many as 30 different SEO tests run against it. Also, because we used an extensible system for storing and modifying tests (you may know them as nodes), we can easily add more tests for Insight and SEO Grader. Our Support team is already creating a list of items that can be tracked, and scenarios that they’ve found to cause customer problems.
After a lot of testing (including a 3-month beta, a redesign, user testing, and usability studies -- not in that order), we launched Insight and SEO Grader on November 29, 2011. Since that time, we’ve seen our users install the Acquia Network Connector on hundreds of sites, which have then displayed thousands of website alerts for our customers to resolve. Most notably, many of these displayed alerts have been fixed on hundreds of Drupal sites -- a fact that makes Drupal better for everyone.
We’ve only just begun. As Insight’s release neared, many other folks at Acquia saw its benefit for our customers. It was exciting to see developers, designers, testers, marketers, sales people, customer advisors, professional services team members, and dozens of other people at Acquia pitch in to help with the project. At its launch, Insight was more than just an idea I had a year ago, but the culmination of an amazing collaborative effort by over fifty people. Both the product and how it was created give me great pride.
In order to create Insight, the six developers on the Acquia Network team had to create or significantly enhance over thirty modules, many of which we contributed back to the community. Over the next few weeks, I’ll blog about the roadblocks we hit, how we worked around them, and our inspirations. I’ll give you the skinny on how we manage the synchronization of hundreds of thousands of pieces of data over hundreds of inbound connections, and how we message that data back to you.
While there were a lot of people who’ve directly or indirectly contributed to Insight, I want to thank some folks (...and the Academy... and my fans...) who were instrumental in creating the product that you get to use: Ben Jeavons, Charlie Gordon, James Elliott, Mariano Asselborn, and Martin Hravbocin. Without each of these people, Insight would still just be an idea.
Stay tuned! I’ll post more soon for those of you that want to look under the hood.
... and if you want to check it out, get a free 30-day trial.
