Tightly integrated, loosely coupled

If you haven’t already seen Acquia Migrate: Accelerate, take a moment to watch this short demo video and pay special attention to the UI after the 40-second mark.

What you’ll see is a full-page React application. Except for the administration toolbar at the top of the screen, every visible element is rendered in the browser with React, even the breadcrumbs. This architecture allowed us to create a user interface that looks, feels, and is an integral part of a Drupal 9 site without compromising our ability to create a maintainable and modern UI.

 

,

Why didn’t we use Drupal’s existing back-end rendering system, a.k.a. Twig? Here is a sample of all of the features that we incorporated into the UI. Given the same resources and without React, we could only have implemented a few of these:

1. Multiple, simultaneous progress meters
2. Notification dots that appear when new content is available on an out-of-view tab
3. Batch operations that do not block a user’s navigation of the application
4. Cancellable, queued operations
5. A streaming error message log
6. Complex conditional user inputs
7. Live filtering
8. Floating, minimizable task tracker
9. Responsive table-like listings with live updates and interactive components

While we believe the feature-list above is very impressive when compared to Drupal-rendered interfaces, they’re not very impressive if you’ve been building decoupled applications for a while. Here are some technical elements that we believe make our approach fairly novel within the decoupled Drupal community:

1. Zero-config authentication
2. Low-bandwidth HTTP polling using Etags and 304 Not Modified responses
3. Seamless navigation between Drupal-rendered pages and React-rendered pages
4. No custom blocks
5. No use of drupalSettings (i.e. doesn’t rely on global state)
6. Drupal is aware of all front end routes 
7. Zero custom server configuration
8. Dismissible handling of HTTP errors

We believe many of these features deserve posts of their own, but four key design choices—made early in the project—enabled most of them. We’ll go into each one in their own section below. They were:

1. Application-specific HTTP API endpoints
2. Hypermedia controls
3. The Decoupled Pages module
4. Rigorous functional testing

Not everything went according to plan. In the last section, we’ll cover a few cases where we struggled with the decoupled architecture.

Application-specific HTTP API routes

The back end of our application uses JSON:API, but not the JSON:API module. JSON:API is an open source API specification independent of Drupal. Since our application deals with migration plugins, messages, and batch operations, we could not use Drupal core’s JSON:API module since it only provides endpoints for Drupal entities.

Here’s a sample of some of the routes that our application handles:

• Migration collection
  • Returns a list of migrations with available actions, their state, category, and progress
  • Path: /migrations
  • Methods: GET
• Migration detail
  • Similar data as the endpoint above—for a single migration
  • Path: /migrations/{migrationId}
  • Methods: GET
• Create migration import process(es)
  • Used to begin a migration; returns a link which should be followed in order to monitor and continue the migration process
  • Path: /migrations/import
  • Methods: POST
• Create migration rollback process(es)
  • Similar to the endpoint above—to begin a rollback
  • Path: /migrations/rollback
  • Methods: POST
• Monitor and continue migration process
  • Incrementally processes the requested batch and returns its progress and state
  • Path: /migration/process/{processId}
  • Methods: GET

,

Hypermedia controls

Despite the predictable URLs described above, our JavaScript application does not have any API URLs or URL path patterns in it. Instead, it learns about each endpoint from the back end. Here’s how it works:

1. The back end serves an index.html with a root div into which the React application renders itself.
2. The React application looks for a data attribute on that div which points to the HTTP API route that serves the current page of the application (more on that below).
3. The React application makes a GET request to that route for its initial data.
4. The React application parses out links encoded in JSON. These JSON-encoded links have types and certain types of links do one thing, other link types do other things. The JavaScript application understands these link types and how to follow them.
5. The React application renders visual elements based on the presence of those links and the user experience it wants to provide. That is, some links cause the app to show a button, others cause the app to show a new dropdown option. Those choices are a front-end concern.
6. When a user selects one of those elements, it triggers the React application to follow the associated link according to a documented protocol. E.g. send a PATCH request with some other data provided on the encoded link, then request the primary HTTP API route again and re-render the application’s components accordingly.

Following this pattern meant that we were able to build complex behaviors out of a few basic building blocks. Our back-end engineers focused mostly on business logic and our front-end engineer focused mostly on the user interface (there was overlap, of course).

For example, one of our link types was called update-resource. We documented that, when followed, the JavaScript application MUST send a PATCH request to the href of the link with the data provided by that particular link. Each resource representing a single migration might have many of those types of links associated with it. Here is one such link:

{     "href": "/migrations/comments",     "rel": "update-resource",     "title": "Skip",     "data": {       "type": "migration",       "id": "comments",       "attributes: {         "skipped": true       }     }   }

,

Once this link was followed, the next request for that same resource would no longer have that link, instead, it would have a link whose title was “Unskip” and the data.attributes.skipped field value would be false. As far as the React application was concerned, these links were indistinguishable. Moreover, the React application did not have to understand that a migration could be skipped or unskipped. That business logic was kept on the back end.

Using hypermedia to control the application also meant that we were able to add new behaviors without changing any code in the React application. This was used several times to prevent our engineers from being blocked on one another: truly decoupled collaboration was a big productivity boost!

For example, if we want to add the ability for a user to postpone a migration in the future, a back end engineer could implement the business logic and add a link with the title “Save for later” along with some different data attributes. Since the React application will simply render the new button alongside the “Skip” button just like any other update-resource link, a front end engineer doesn’t have to get involved. In that way, we can say that the React application learns about new endpoints at runtime.

The Decoupled Pages module

The Decoupled Pages module provides a quick and simple way to define new Drupal routes which can be taken over by a single page application (SPA), often written in Javascript using React or Vue.

Without going into too much detail since the project page provides technical documentation, the concept is that by declaring a route and any additional SPA-routed paths in a module’s routing.yml file, Drupal can serve the SPA to the user’s browser. Since Drupal knows about those paths, the SPA will load even if the user follows a link directly to an SPA-routed path. This eliminates the need for custom Apache or NGINX server configuration in order to serve the application’s main JavaScript file (often named index.js). This also means that non-decoupled pages can easily link to decoupled pages and vice versa: Drupal can link to internal routes rendered by React.

Finally, since Drupal serves the SPA, there were no cross-origin request obstacles to overcome, Drupal’s built-in user login pages were still accessible, and the React application could rely on the browser's built-in cookie processing. In other words, the React application didn’t need a single line of code dedicated to authentication concerns.

Rigorous functional testing

Throughout the development cycle, we made continuous improvements to the user experience in response to multiple rounds of user tests, a private beta, and a public beta. At the time of writing, Acquia Migrate: Accelerate has reached limited availability. After each of those experiences and milestones, our understanding of our users’ needs changed—as did our understanding of the product itself.

Since we knew from the beginning of the project that we would be actively discovering what exactly the product should do, we knew we had to be especially careful of technical debt and regressions. In other words, we needed automated testing, but we needed those tests to be very tolerant of large refactorings and a quickly evolving user interface. Therefore, we heavily preferred high-level tests to low-level tests.

Overall, we were happy with this approach. No new feature was added and no bug fix was accepted until it had at least one test. This isn’t to say we only used functional tests, any custom JavaScript functions were covered by unit tests using Jest.

Functional UI tests

For the UI, we used Drupal’s Nightwatch testing system. Nightwatch.js emulates a user clicking through the product to exercise different product features. By using this tool, we could move and rewrite React components without throwing away their tests (and the time we spent writing them). And we had a high degree of confidence that we weren’t breaking any feature that we had already gotten to work earlier in the development lifecycle.

However, these tests were not perfect. We were occasionally bogged down by the confusing behavior of its web driver (i.e. the library that uses a standard browser API to pretend to be a real user). This meant that we had to write our markup with the web driver in mind. For example, we had to add unique classes or IDs so the driver could find a button or link to “click”. These selectors needed to be maintained and updated as the UI evolved.

Functional API tests

For the HTTP API, we used Drupal’s Browser testing system. We used the JsonApiRequestTestTrait to account for the fact that we were not testing HTML responses, but JSON:API ones. To simulate an actual migration, these functional tests had access to a Drupal 7 fixture database. This meant that we could be confident the tests were realistic and and truly exercised the migration system.

Our API tests used the hypermedia controls described above as if it was the React UI’s JavaScript. For example, our tests would make an initial HTTP request, deserialize the JSON, search for an import link, then follow the protocol documented for those types of links to execute a migration. Finally, our tests would assert that some content had actually been imported.

This testing pattern meant that we could change the underlying implementation without fear of breaking the behavioral contracts made with the JavaScript side of the product. In addition to preventing regressions, these types of tests also provided some HTTP request examples that the JavaScript side of the product could follow.

What didn’t work

To be upfront: we would not approach this problem differently if we were asked to build the product again. However, some things were more difficult with our decoupled architecture than if we had gone a more conventional route. They were:

1. We had to replicate the administration theme
2. We struggled with managing long-running operations
3. We had duplicate routing configuration

Replicating the administration theme

In a conventionally built administration interface, module developers rarely have to worry about the theme. For the most part, they can use Drupal’s Form and Render API to generate markup which inherits its look and feel from the site’s administration theme.

Since we were not using Drupal to render markup, we had to replicate markup and copy style rules into our codebase in order to make the application feel integrated into the rest of the site. Making a completely custom UI gave us the freedom to implement new design patterns, but that meant we had to implement new design patterns.

Drupal core comes with many design elements like checkboxes, dropdowns, draggable lists, etc. These have gone through a rigorous review process which means that they’re accessible and internationalizable out-of-the-box. Since we weren’t using those, our front-end engineer had to be especially mindful to avoid accessibility issues. At this time, the application isn’t fully internationalizable either, although we did try to keep that possibility open by isolating any translatable strings.

Managing long-running processes

One of the features that we implemented was the ability to add many long-running migrations to a queue and continue navigating the application while they were being processed.

Unfortunately, PHP and Drupal do not have a convenient solution for keeping long running processes, like a migration, alive. Drupal works around this shortcoming with its Batch API. This system uses an archaic browser behavior to repeatedly redirect the browser; it uses each subsequent request to process a fraction of the remaining items in the batch’s queue. Those pages have a premade progress meter.

Had we not made a React application, we would likely have reused this system and avoided a lot of work. Instead, we had to create our own batch processing system and overcome a number of challenges with a client-side queue. However, we feel that the final user experience was worth the trade-off. Unlike the built-in batch API, our implementation empowers users by displaying fine-grained information about the state of each queued migration, by enabling them to add and remove migrations from the queue, and by providing them with the ability to pause or cancel the entire process without difficult to remedy consequences.

Duplicated application routing

This was more of a tedious annoyance than a major shortcoming: we had to maintain our application’s UI routes in two places in order to enable bookmarking and shareable links to pages handled by our React application. That’s because Drupal needs to know that a request for /migrations/migration/{migration}/{tab} should serve our SPA’s code instead of responding with a 404 Not Found.

Many decoupled architectures work around this by using separate domain names for their UI and their HTTP API. For example, if a request is made for any route at www.example.com, the back end always serves the SPA code. If a request is made for any route at api.example.com, the back end always serves JSON.

We could not take that same approach because we wanted our UI to be integrated with the rest of the site’s Drupal-rendered routes and we couldn’t require special server configuration since we wanted the app to work locally without any special set up. Therefore, for every route managed in our React application, we also had to declare that route in a routing.yml file within our Drupal module so that Drupal would know to serve the SPA code instead of a 404.

 

Piqued your interest? Learn more!

All Acquia Cloud Enterprise customers are eligible for Acquia Migrate Accelerate! Check out our product overview or request a demo today! Talk to your Acquia Account Manager to get started, and then our getting started documentation will get you off and running!