Building an Open Source Photo Gallery with Face and Object Recognition (Part 2)

In part one of this two-part series, I explained why my Hackathon team wanted to build an open source photo gallery in Drupal 8, and integrate it with Amazon S3, Rekognition, and Lambda for face and object recognition.

In this post, I'll detail how we built it, then how you can set it up, too!

tl;dr: Check out the open source Drupal Photo Gallery project on GitHub, and read through its README for setup instructions so you can build an intelligent photo gallery powered by Drupal and AWS Rekognition.

Storing images on a Amazon S3 with the S3FS module

Once we had a basic Drupal 8 site running on Acquia Cloud with a 'Gallery' content type and an 'Image' Media type, we switched the Image's Media entity image field to store images in Amazon S3 instead of Drupal's public files directory.

The S3 File System module makes this easy. We had to use Composer to install it (composer require drupal/s3fs) since it has a few PHP library dependencies. We then configured the S3FS module settings (/admin/config/media/s3fs), and pasted in an Access Key and Secret Key for our team's AWS account, as well as the S3 bucket name where we'd store all the files from the site. We changed the Image field in the Media Image entity to store files (the 'Upload destination' on the Field settings page) on 'S3 File System'.

Note: For security reasons, you should create a bucket for the website in S3, then create an IAM User in AWS (make sure the user has 'Programmatic access'!), and then add a new group or permissions that only allows that user access to the website's bucket. The CloudFormation template, mentioned later, sets this up for you automatically.

We stored all the files in a publicly accessible bucket, which means anyone could view uploaded images if they guess the S3 URL. For better privacy and security, it would be a good idea to configure S3FS to have a separate private and public directory in the S3 bucket, and to store all the gallery images in a private bucket. This is more secure, but note that it means all images would need to be passed through your webserver before they are delivered to authenticated users (so you'd likely have a slightly slower-to-load site, depending on how many users are viewing images!).

Why did we store files in S3 instead of on the webserver directly? There are a few reasons, but the main one in our case is storage capacity. Most hosting plans offer only 20, 50, or 100 GB of on-server storage, or charge a lot extra for higher-capacity plans. With photos from modern cameras getting larger and larger (nowadays 10, 20, even 40 MB JPEGs are possible!), it's important to have infinite capacity—which S3 gives us for a pretty minimal cost! S3 also makes it easy to trigger a Lambda function for new files, which we'll discuss next.

Using AWS Lambda to integrate Drupal, S3, and Rekognition

This is the automated image processing workflow we built:

1. A user uploads a picture (or a batch of pictures) to Drupal 8 using Entity Browser.
2. Drupal 8 stores each picture in an Amazon S3 bucket (using S3FS).
3. An AWS Lambda function is triggered for each new picture copied into our S3 bucket (more on that in a bit!).
4. The Lambda function sends the image to Rekognition, then receives back the object and facial recognition data.
5. The Lambda function calls a REST API resource on the Drupal 8 site to deliver the data via JSON.
6. Drupal 8's Rekognition API module parses the data and stores labels and recognized faces in Drupal taxonomies, then relates the labels and faces to the Media Image entity (for each uploaded image).

This was my first time working directly with Lambda, and it was neat to see how Lambda (and other 'serverless' infrastructure) can function as a kind of glue between actions. Just like when I built my first Node.js app and discovered how it's asynchronous nature could complement a front-end built with Drupal and PHP, Lambda has opened my eyes to some new ways I can act on different data internally in AWS without building and managing an always-on server.

The Lambda function itself, which is part of the Rekognition API module (see index.js), is a fairly straightforward Node.js function. In a nutshell, the function does the following:

1. Get the S3 bucket name and object path for a newly-created file.
2. Run rekognition.detectLabels() to discover 'Labels' associated with the image (e.g. 'Computer', 'Desk', 'Keyboard'), then POST the Labels to Drupal 8's Rekognition API endpoint.
3. Run rekognition.indexFaces() to discover 'Faces' associated with the image (and any 'FaceMatches' with other faces that have been indexed previously), then POST the facial recognition data to Drupal 8's Rekognition API endpoint—once for each identified face.

So for a given image, Drupal can receive anywhere from one to many API calls, depending on the number of faces in the image. And the Lambda function uses basic HTTP authentication, so it passes a username and password to Drupal to authenticate its API requests.

On Drupal's side, the Rekognition API POST endpoint (set up in Drupal as a REST Resource plugin) does the following:

1. Verifies the callback is for a valid, existing File entity.
2. Stores any Labels in the body of the request as new Taxonomy terms (or relates existing Taxonomy terms, if the Label already exists in Drupal).
3. Stores any Faces in the body of the request as new 'Face UUIDs' (since these are what Rekognition uses to relate Faces across all images), and also ensures there is a corresponding 'Name' for every unique, unrelated Face.

That third step is critical to making an 'intelligent' gallery—it's pretty easy to be able to detect faces in pictures. You could use something like OpenCV to do this without involving Rekognition at all.

But if you want that data to mean something, you need to be able to relate faces to each other. So if you identify "this face is Jeff Geerling" in one picture, then in the next 5,000 photos of Jeff Geerling you upload, you shouldn't have to keep telling the photo gallery "this face is Jeff Geerling, too... and so is this one, and this one..." This is what Rekognition and it's fancy machine learning algorithms gets us.

So in Drupal, we store each FaceId or each of the Faces in FaceMatches as a unique face_uuid node, and we store a separate name node which is related to one or more face_uuids, and is also related back to the Media entity.

If you're interested in the details, check out the entire Rekognition API module for Drupal.

Displaying the data on the site

It's nice to have this structured data—galleries, images, faces, labels, and names—but it's not helpful unless there's an intuitive UI to browse the images and view the data!

Our Hackathon cheated a little bit here, because I had already built a basic theme and some views to display the majority of the information. We only had to touch up the theme a bit, and add labels and names to the Image media type's full entity display.

One of the best front-end features of the gallery is powered by Drupal 8's built-in Responsive Image functionality, which made responsive images really easy to implement. Our photos look their best on any mobile, tablet, or desktop device, regardless of display pixel density! What this means in the real world is if you're viewing the site on a 'retina' quality display, you get to see crisp, high-res images like this:

Instead of blurry, pixelated images like this:

Most photographers try to capture images with critical focus on the main subject, and having double resolution images really makes technically brilliant pictures 'pop' when viewed on high-resolution displays.

We display names below the image, then a list of all the labels associated with the image. Then we display some other metadata, and there's a back and forward link to allow people to browse through the gallery like on Facebook, Flickr, etc.

We wanted to add some more functionality to the image display and editing interface, but didn't get time during the Hackathon—for example, we hoped to make it easy to click-and-update names associated with images, but we realized we'd also need to add a feature that highlights the face in the image when you roll over a name, otherwise there's no way to identify individual names in a picture with multiple faces!

So I've added issues like Add edit link next to each name on image page to the Drupal Photo Gallery project, and if I can spare some time, I might even work on implementing more of these features myself!

All the configuration for the site is in the Drupal Photo Gallery project on GitHub, and if you want to get into more detail, I highly encourage following the instructions in the README to install it locally using Drupal VM's Docker image (it should only take 5-10 minutes!).

Next steps

There were a number of other features we had in our original "nice-to-haves" list, but didn't have time to implement during the Hackathon, including:

• Per-album and/or per-photo group-based permissions.
• Sharing capabilities per-album and per-photo (e.g. like Google Drive, where you can share a link to allow viewing and/or editing, even to people who don't have an account on the site).
• Photo delivery via private filesystem (currently the S3 bucket is set to allow public access to all the images).
• Configure and use different REST authentication methods besides basic HTTP authentication.
• Easy enablement of HTTPS/TLS encryption for the site.

I may implement some of these things for my own private photo sharing site, and I hope others who might also have a passion for Drupal and Photography would be willing to help as well!

If that sounds like you, head over to the Drupal Photo Gallery project page, download the project, and install it yourself!

Or, if you're more interested in just the image processing functionality, check out the standalone Rekognition API module for Drupal 8! It includes an entire AWS CloudFormation template to build the AWS infrastructure necessary to integrate Drupal and Rekognition using an S3 bucket that triggers a Lambda function. The AWS setup instructions are detailed here: AWS setup - S3, Lambda, and Rekognition.