tl;dr
Contrib Search maintainers are committed to make Drupal 8 kick ass with Search API.
Search is a massively cool technology spectrum with loads of really tough problems such as language stemming, delivering search as a site scales and helping customers actually find what they want. However, solving these is not so easy. Let’s look at the history of Search and Solr in Drupal.
Drupal 6
Drupal 6 got a great Apache Solr Search module, thanks to PWolanin and RobertDouglass. There was a 1.x and even a 2.x branch early on. The 2.x branch was loaded with cool ideas but only saw most of its conversion in real working code in the Drupal 7 branch. The 2.x branch got deprecated and now we have 3.x. This is a straight backport from the Drupal 7 branch. At its most successful time the 6.x branch of the module was installed on around 7000 Drupal sites.
Drupal 7
Drupal 7 was the big start for me personally to get deeply involved in Search for Drupal. I had a 6 month internship at Acquia to improve the Apache Solr Search module and Acquia Search platform. The Drupal 7 module got a complete overhaul, fixing many bugs, improving the developer experience, and adding features like entity support and much more. The module is now used on more than 9100 sites.
At the same time we saw the Search API module emerging. For a while now both modules solved similar use cases with a very different architecture. Each module focussed on different use cases and were the right tool for the right job for many. However, some of the architectural decisions made it very hard for the Apache Solr Search project and the Search API project to combine efforts. Search API is now used by 24,000 websites, and 6300 of them use Apache Solr.
Not everyone understood the differences between the Apache Solr Search and Search API and this has led to confusion. Due to these architectural differences, some problems were solved twice by the community. For example, modules such as the facetapi_slider and searchapi_ranges tried to solve the same problem, which was to create a graphical slider for facets, twice. This duplication is lost effort in our collective goal of making Drupal the world class product for your search solution; we are basically slowing ourselves down with these two completely separate codebases.
Combined efforts
Around 2012 and 2013 combined efforts between the two projects began taking place, such as:
- Facet API was created to become the common way to show facets blocks.
- Presentations were given to show the difference and to encourage collaboration, for example the Solr Wizardry at Dev Days Barcelona 2012
- A common schema and solrconfig.xml were created with Acquia sponsorship. This effort made it much easier for users to switch between modules when trying them out on a shared hosting such as Acquia Search. It also offered many other benefits such as combined performance updates and more when we change something in solrconfig.xml. This change benefits more than just Acquia — it makes it easier for all hosting companies to support both modules.
- Both maintainers worked together and migrated the connection classes from the Apache Solr Search module into Search API.
Drupal 8
With the release of Drupal 8 we are at an important point in time to decide if we want to step up and have an even bigger shared roadmap. At DrupalCon Portland, we started thinking, but things got really interesting when we had meetings at DrupalCon Prague. In one of the meetings, we tried to identify common parts in the two systems and to see what we could share in Drupal 8. We were still going our separate ways, somewhat, but decided we could at least share the views query class and some other components.
At a second meeting at the DrupalCon Prague Sprint, between drunkenmonkey and myself, we tried to identify why we couldn't make a unified solution. I explained some of the architectural problems some websites encounter with Search API and why it can't be used for the same use cases the Apache Solr Search module had in mind. This meeting was a big step in helping us align visions and explain to each other what we wanted to achieve.
We then took some time to consider what we (as Acquia, but more importantly as a community) were going to do with the Apache Solr Search module for Drupal 8. Because we were able to see so much progress in sharing our vision and components, and to acknowledge and address some of the architectural challenges in the Search API, we decided to go fully on-board to make the Search API module a unified, kick-ass solution for search in Drupal 8.
All of the Apache Solr Search and Search API maintainers are aligned on this goal. There is a lot of work ahead of us if we want to make sure that all the good things of both worlds get combined and some bugs can be squashed. Naturally, we are also starting to learn the new way of working in Drupal 8, so it'll be a wild ride.
We have already started to fulfill this promise. We are working together with drunkenmonkey and the rest of the Search API crew to identify some of the current problems our customers have with Search API, and to collaboratively solve those. If you want to see or help with some of these issues, I invite you to join the Search API issue queue, start reviewing, and chip in. One of the issues that we are tackling is a better handling of the errors of indexing during cron runs. The solution that was proposed is to get rid of the Queue system, and while it sounds like a dramatic change, I can assure you it is not as dramatic as you'd think ;-).
There are some action items in order to move forward. I’m sure many more will be added in the course of the following months. There is already an issue in the Search API issue queue to list what is needed to upgrade Search API to Drupal 8 (https://drupal.org/node/2044421 and https://drupal.org/project/issues/search_api?version=8.x ).
Here is a list of some of these issues that I think we need to have in Search API to make it a first-class search-solution.
- Add more flexibility in how we show data in an index. This data can come from sources outside of the Drupal site. This data is also not necessarily indexed by a Drupal application.
- Reduce the amount of code that unnecessarily relies on Drupal structure (for example, the field listing).
- Add more view modes support, even for content that is not an Entity. Boost multi-site search support massively.
- Make search pages faster by not loading the entities if not required. The data can live in the index.
- Allow multiple entity types to be selected when creating an “Index”
- Add more RDF support
- Add more multilingual support.
- …
If you have questions, comments, fireworks, please let us know below! And we hope to see you in the issue queue to make Drupal 8 kick ass with Search API!
