When managing a whole government platform, as we do in Australia with govCMS, there need to be layers of testing, staging, and assurance before you push code to production.
Automated testing allows us to delegate repeatable and menial tasks to a robot in a scripted fashion. These tests deliver a way of detecting basic syntax errors, breakages of coding standards, build failures, and regressions with a high degree of surety. We can also be assured that functionality persists through new builds, security requirements are met, and that upgrade paths work as expected.
This has the benefit of allowing developer time to be focussed on delivering new high quality features. It also ensures that the QA process is consistent across all proposed changes to the platform.
govCMS ships with a comprehensive and extensible automated testing suite to protect the platform against bugs and regressions. It also offers a managed procedure for running smoke tests over all sites prior to the production push. It’s worth noting however, that not everything is covered by platform tests and a continuous improvement approach is taken to improve coverage.
As the codebase itself is publicly available, we can observe both the automated tests and the results of them in the continuous integration (CI) system, Travis CI.
With every requested change to the govCMS codebase, created via a pull request, a significant number of tests are run. The aim is to protect both the platform and all sites on it against bugs, coding errors, and regressions in functionality. Once all tests have passed, code is deployed and everyone on the platform benefits from the new features and functionality.
As soon as a change is suggested, test runs are spawned for both PHP 5.5 and 5.6. As long as each of these completes without triggering an error, the pull request is considered successful and the govCMS maintainers are notified to provide a manual review.
What we’re using
In keeping with the open source ethos that drives both Drupal and govCMS, we’re using a suite of community contributed tools to build and test.
Amongst other things, the toolkit consists of:
The composer.json file acts as instructions for Composer to download all dependencies used by govCMS. Once these packages have been pulled down, Phing uses build.xml to download Drupal, install all modules using Drush, and run tests.
Functional testing against the govCMS platform uses Behat, a behaviour-driven development PHP-based test framework, with all tests written in the natural Gherkin language.
Using Behat empowers non-technical users to define acceptance criteria to the user stories that form the basis of the agile development methodology.
Testing locally
After building govCMS locally, navigate to the docroot directory. The following commands can be used to initiate the built in a PHP Web server and run all tests in the same manner Travis CI does.
$ php -S 127.0.0.1:8888 "../build/tests/drush/router.php" &> /dev/null & $ phing -f build/phing/build.xml run-tests
Parameters in both Phing and Behat can be overridden if needed for different ports, URLs, database backends, etc. This can be done by altering the build.properties and behat.local.yml files.
Writing new tests
As recommended in the project’s CONTRIBUTING.md, any new features added to the platform should be accompanied by a test to ensure it is checked for regressions. New Behat tests can be written and added alongside others.
Features that require javascript support can be tagged with @javascript. This will ensure headless PhantomJS is used to test not only text output onto the page, but also JavaScript events.
Testing existing sites
The continuous evolution of the platform means that new code also needs to be tested against existing sites prior to production deployment. To test new releases, the Acquia CloudAPI is used to deploy all govCMS production sites into the stage environment, and run smoke tests using PhantomJS to detect changes.
