Acquia Migrate: Accelerate — now open source!

Now that many Acquia customers have made the leap, it's time to open source Acquia Migrate: Accelerate (AM:A): drupal.org/project/acquia_migrate — any Drupal site owner can use this tool freely 🥳

Countless blog posts have been written about migrating from Drupal 7 to 8/9/10. Some recommend how to begin breaking down this rather overwhelming endeavor into manageable pieces. Others provide concrete technical recommendations for very specific Drupal 7 setups.                       
The many hours spent writing up valuable lessons learned and sharing them is a very valuable contribution to the Drupal community! But it still requires many individual Drupal developers to find, read, understand and apply these lessons.

For Drupal sites with big budgets, this is feasible, but for many among the long tail of >350,000 Drupal 7 sites, it is not. Too much technical expertise is required, which is often out of (financial) reach.

What if that could be different? What if we could collect all these lessons learned and make them not just available to the Drupal community, but apply them automatically?

Enter: AM:A! 🦾🤖

What is it? #1: a friendly UI 🤩

It's a friendly UI built on top of Drupal core's migration system, but without the low-level technical details. It provides a dashboard that lists only high-level concepts such as "Tags taxonomy terms" or "Blog post content items" and visualizes the dependencies between these concepts:

(This allows the user to first migrate tags, inspect them on the real Drupal 9 site, and only after confirming that they look fine move on to blog posts, in contrast with Drupal core's migration UI, which is all-or-nothing).

It provides live previews before even executing migrations:

You can navigate around while the migration is happening, with live updates:

Finally, there is a central console showing migration errors, including solution suggestions tailored to those migration errors 🙏 (You'd be surprised how many data integrity issues exist in Drupal 7 — AM:A helps catch these.)

What is it? #2: a Drush command for faster migrations 🚀

For users not afraid of the command line, AM:A provides a drush ama:import command, which provides a CLI representation of the same migration dashboard as above, allows scripting and provides faster migration performance. (A single long-running process has far less overhead and is hence a lot faster at executing migrations than the UI above — especially for big migrations.)

Our favorite thing: the UI still works fine, and a little robot will show up in the progress indicator at the top right! 🤖 So you can use that to not only observe progress in a nice UI, you can also browse the site live while the migration is happening.

What is it? #3: module + patch recommendations 📕

AM:A has grown to provide >2500 recommendations, >700 of which are hand-curated, and 280 are vetted, which means we've manually tested, verified and patched the migration path from >400 Drupal 7 modules to >250 Drupal 9 modules. Over a hundred patches to Drupal core and contributed modules are included.

Many of recommendations are opinionated: AM:A for example migrates a dozen or so ways of storing and embedding media in Drupal 7 to a single new way: Drupal core's Media module + Media Library module + Embed media filter. The goal: take any existing Drupal 7 site and get them on the path of strongest Drupal core support, both to enable a better end user experience and to simplify future Drupal upgrades.

These patches have always been publicly available. It's just been hard to gather them all together, until now.     

(Most migration plugins will work fine out-of-the-box with AM:A, but many migration plugins are lacking dependency information. Without that information, the AM:A module puts it in the catch-all "Other" migration on the migration dashboard.)

What is it? #4: Drupal 9 migration project generator ⚙️

Acquia CLI's new app:new:from:drupal7 command is the glue between the recommendations and the module: it inspects a Drupal 7 site, interprets the (module + patch) recommendations and generates a composer.json file for your brand new Drupal 9 site. With all modules and patches automatically listed!

In other words: this removes a lot of the painstaking research that you'd otherwise have to do manually, before you can get started. Anybody can download and install Acquia CLI and use AM:A, even if you are not an Acquia customer.

You'll be up and running in minutes: all that remains is following the on-screen instructions to let AM:A find your Drupal 7 database.

Who is it for?

It's for all sites still on Drupal 7. The simpler the site, the more benefit AM:A brings, the more time it saves.

• Drupal 7's long tail of relatively simple sites will especially benefit, since they are fairly likely to have a 95–100% complete migration.
• More complex sites will mostly benefit from AM:A getting you to a great starting point: pretty much all known migration best practices are encoded in AM:A's recommendations!

But you'll still need to debug your migrations when the inescapable mystic migration errors turn up. 😮‍💨 I recommend PHPStorm + Xdebug. Try to not get mad at the world and take a deep breath 🧘‍♀️ — there are at least 2 very good reasons it doesn't Just Work:

1. the Drupal 7 site has data integrity problems (fix them there, not in Drupal 9, migration plugins assume valid input data!)
2. this site has a hitherto unseen complex combination of modules, field types and configuration — it's up to you to debug it, figure it out, and contribute it back upstream 🙏

Your Drupal 7 site is up-to-date, right?! Make sure all modules are on the latest version and database updates have been applied! Countless hours have been lost during migrations due to this! ⏳

Want it now?

Great! 😄

In 5 minutes you'll have a live AM:A instance for your Drupal 7 site.

1. [60 s] Choose a Drupal 7 site that you have running locally. Figure out the path to the root of Drupal (the directory containing index.php). Let's say it's /var/www/example.com.
2. [5 s] Create a directory where your Drupal 9 site will be generated. Let's say /var/www/d9.example.com.
3. [5 s] Create a new database for it. Let's say example-d9.
4. [50 s] Fiddle with your web server setup to allow you to visit http://d9.example.test. (There's nothing there yet, we'll change that 🤓.)
5. [30 s] Install ACLI. (It is assumed that composer and jq are already installed.)

6. [30 s] Generate a Drupal 9 site for it (you can also omit all parameters and it'll ask you instead):

   # From any path:
   acli app:new:from:drupal7 --drupal7-directory=/var/www/example.com/docroot --directory=/var/www/d9.example.com

   yields this output:

   🤖 Scanning Drupal 7 site.
   👍 Found Drupal 7 site (7.59 to be precise) at sites/default, with 235 modules enabled!
   🤖 Computing recommendations for this Drupal 7 site…
   🥳 Great news: found 9 recommendations that apply to this Drupal 7 site, resulting in a composer.json with:
   	- 11 packages
   	- 1 patches
   	- 5 modules to be installed!
   🚀 Generated composer.json and committed to a new git repo.
   ⏳ Installing. This may take a few minutes.
   No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information.
   Gathering patches for root package.
   ✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂
    - Applying patches for drupal/core
      https://www.drupal.org/files/issues/2020-02-24/core-derived_rdf_mapping_migration-3115073-7.patch (Issue #3115073: Convert RDF mapping migration to derived migrations)
      https://www.drupal.org/files/issues/2021-02-24/core-derive_path_alias_migrations-3122649-41.patch (Issue #3122649: Derive path alias migrations per entity type (and bundle))
   ✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂✂
    - Copy [web-root]/sites/default/default.settings.php from assets/scaffold/files/default.settings.php
    - Copy [web-root]/modules/README.txt from assets/scaffold/files/modules.README.txt
    - Copy [web-root]/profiles/README.txt from assets/scaffold/files/profiles.README.txt
    - Copy [web-root]/themes/README.txt from assets/scaffold/files/themes.README.txt
   New 💧 Drupal project created in /var/www/d9.example.com. 🎉
   

   That's right, composer install already happened for you!

7. [15 s] Now let's install Drupal with user 1 named admin, password admin and pass in both the database you just created (example-d9), the directory where the site was generated

   # ⚠️ From inside /var/www/d9.example.com! ⚠️
   vendor/bin/drush site-install --account-name=admin --account-pass=admin --site-name="example.com on D9 thanks to AM:A" --no-interaction --db-url=mysql://root:root@localhost/example-d9

   followed by:

   jq -r '.installModules[]' < acli-generated-project-metadata.json | xargs php -d memory_limit=512M vendor/bin/drush pm:install -y
   

   (to install all modules whose migration recommendations have been vetted, to ensure those migrations are available out of the box)

   followed by:

   vendor/bin/drush state:set --input-format=json acquia_migrate.initial_info - < acli-generated-project-metadata.json

   (to provide all that metadata to the AM:A Drupal module's Module Auditor UI by storing it in Drupal's state )

8. [5 s] Visit http://d9.example.test again, you should see this screen:

   

9. [60 s] Follow the instructions on that screen: tweak /var/www/d9.example.com/docroot/sites/default/settings.php. You already have the location of your Drupal 7 site handy. In my case, that required adding:

   $databases['migrate']['default'] = array (
     'database' => 'example',
     'username' => 'root',
     'password' => 'root',
     'prefix' => '',
     'host' => 'localhost',
     'port' => '3306',
     'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
     'driver' => 'mysql',
   );
   // The directory specified here must contain the directory specified in the
   // "file_public_path" Drupal 7 variable. Usually: "sites/default/files".
   $settings['migrate_source_base_path'] = '/var/www/example.com';
   // The directory specified here must contain the directory specified in the
   // "file_private_path" Drupal 7 variable. Usually outside the web root.
   //$settings['migrate_source_private_file_path'] = '/somewhere/private';
   

10. [30 s] Choose which high-level concepts you want to import: maybe not the webform submissions for example?

11. 🚀 Voila, the structure is already being imported, you're ready to start migrating!

    

12. Choose something and start migrating, with still time to spare to reach the 5 minute mark 😎

FAQ

• Why Drupal 9 and not 10?

  See the part above about manually vetting migrations, which resulted in "vetted migrations". Unfortunately, PHP 8 behaves differently and Drupal 10 requires PHP 8.1. Plus, there are still more modules available for Drupal 9 than for 10, so the migration would be less complete.                       
  Fortunately, thanks to us Drupal having made upgrades easy forever (since Drupal 8), migrating from 7 to 9 is 90% of the work. From 9 to 10 is trivial in comparison.

• Will AM:A ever target Drupal 10?

  The AM:A module itself already works fine on Drupal 10. But not all migrations do (see the previous question). We'd love to make it possible to migrate directly from 7 to 10, which will be much more feasible now that it's all open source!

• Why now?

  Balancing business needs with open source contributions can be difficult. The most important thing is that it's available now, ~1.5 year before the Drupal 7 EOL in 2025. (We always intended it to be open sourced, that's why d.o/project/acquia_migrate was created by webchick 3 years ago, even before development started! We've also tried to get parts of the infrastructure into Drupal core.)

• What's the name of that pretty thing in the top right? 🤩

  We're thrilled you like the Swooshy Wooshy 😁 — yes, that is a bona fide webchick term! (👋 Angie!)

• How does AM:A choose which high-level concepts to present?

  AM:A analyzes the dependencies between all installed migration plugins. It organizes migration plugins into clusters using heuristics that have been hardened over the years.                       
  Without that dependency information, this cannot work. Unfortunately, many migration plugins do not specify dependencies, including migration plugins in Drupal core. So the AM:A team contributed countless patches. See above.

• Why is the Drush command faster?

  Because the in-Drupal UI must use requests, and PHP has a max_execution_time limit. (Basically the Swooshy Wooshy is a really fancy alternative batch UI!) PHP code executed from the CLI can run as long as needed, removing the need for: i) bootstrapping Drupal, ii) fetching lots of information, iii) having the migration system figure out what the next row to migrate is.

• Does AM:A help with my theme?

  No. But … why not switch to Drupal core's new default Olivero front-end theme, which likely has far better accessibility than your Drupal 7 theme 🙊 Soon you'll be able to clone & customize it.

Want more detail?

See the tutorial, in which I showcase migrating my personal site from D7 to D9 in an hour using AM:A 🤓

Thanks to the team

Being the tech lead on this project was a dream experience thanks to an amazing team to get it off the ground:

• Angie "webchick" Byron — for all things product & UX, and asking to make the impossible possible 😅
• Gabe Sullice — for all things API & hypermedia (decoupled_pages), building entirely new toolchains from scratch (acli app:new:from:drupal7 is all him) 🦾
• Peter "zrpnr" Weber — for all things React, he translated Angie's instruction "Swooshy Wooshy" into the informative delightful experience you see on screen 🪄
• Zoltan "huzooka" Horvath — for making impossible migrations a reality (media_migration, location_migration, bean_migrate, and even … paragraphs_migration 🫳🎤)

These fine folks are all wizards! 🧙

Once the infrastructure was ready, most of them worked on other things, but we fortunately got people helping to grow our recommendations:

• Narendra "narendraR" Singh Rathore
• Srishti "srishtiiee" Bankar
• Omkar Podey
• Yash Rode

Together, we massively grew the recommendations, and put tooling into place to automate as much as possible: from testing the >2,000 recommendations daily (to ensure it results in an installable composer.json), to automatically detecting when patches landed & shipped in releases, and much more! 👩🏻‍💻