It is an exciting time for Drupal search. Thanks to jhodgdon, the search community is finally communicating with each other. Looking at projects such as Search API Facet API Integration, we are actually starting to collaborate as well. With collaboration comes healthy and welcomed differences of opinions, but in order to keep discussions productive we must be working towards common goals. I am very passionate about Drupal and search, and I want to see if there are common high level goals we are all trying to achieve. I'll start by sharing some of my thoughts in an effort to be transparent as to why I am focusing on some areas of search and not so much on others.
Mission statement
Every initiative must have a mission statement. Every contribution I make towards search will be geared towards the following ideal:
Make search a differentiator when choosing Drupal over other CMSes
Goals
My time in Professional Services has taught me that Drupal search competes with enterprise providers such as Endeca and Autonomy and is measured against experiences like Amazon, Google, and Sharepoint applications. Obviously we are behind the eight ball. In order to catch up, I feel the following criteria must be met:
- User Experience: Create out-of-the-box solutions that can build search experiences like Endeca, Amazon, Sharepoint, etc.
- Scalability: Provide an upgrade path to solutions with enterprise-grade scalability.
- Accountability: Offer hosted solutions that take the risk out of building search-enabled applications with Drupal.
Arguments can be made that we have all three, however the major problem is continuity between them. For example, if you use Views + core Search there is no easy way to port the UI to Apache Solr Search Integration, Search API, or Acquia Search. That is why I have been focusing most of my efforts on the Facet API module.
The strengths of the competitors mentioned above are the interfaces they provide for users to refine their search in meaningful ways. If those configurations can be ported between the various Drupal projects, we are 90% of the way there in terms of achieving continuity. By consolidating efforts of the various search sub-communities, I am confident we will create differentiating display widgets in no time.
Core search
None of the goals mentioned above are dependent on changes to the core Search module. Our new found collaboration is proving that shared solutions such as Facet API are possible, and I expect additional projects to pop up in the future (i.e. Converter module). To me, a massive refactoring of the core Search module is a lot of effort that doesn't directly serve the mission statement above. As a result, the only goals I have for D8 search are the following:
- Make sure it works out of the box with no configuration.
- Ensure that it knows how to get the heck out of the way so contrib can pick up where it leaves off.
Once tools are proven to work with all available solutions, maybe we can start thinking about their inclusion in Drupal 9. Core Search currently suffers from a lack of maintainership, so I am not sure that a massive influx of code is the best idea at this time. I think it was Barry Jaspan who jokingly said, "core Search is where good developers go to die". It has been passed around more than illegal drugs at a Phish concert, so I am weary that we won't have enough eyeballs to ensure that a truly generic solution can be achieved and maintained in the near future.
Search Lucene API
A lot of people ask me about Search Lucene API. I think it is an important module that has spawned some cool things such as Facet API and the "search pages" feature now available in Apache Solr Search Integration. However, maintaining yet another isolated API is detrimental to the search community as a whole. Therefore I will be stripping out much of the functionality into generic projects for other backends to use if they so choose. As these projects become available, so will a stable version of Search Lucene API. I would very much like to drop the "API" from its name before 7.x-3.0 is released.
In addition, my hope is that the Faceted Navigation for Search module will kill the need for Search Lucene API, as its main goal is to provide an advanced search solution to smaller sites primarily on shared hosting. In short, Search Lucene API isn't dead, but it is not a focus of mine right now. Hopefully Faceted Navigation for Search will provide a viable alternative for people who have benefited from SLAPI in the past.
Community focus going forward
This post is simply a conversation starter and a summary where my head is currently at. I would love to get feedback on what other people think are important high level goals in order for Drupal to be viewed as a search-ready platform. There is an unprecedented opportunity here, let's make sure it doesn't slip away.
Comments
Core search
I have to disagree with you regarding the role core search could/should play in this, especially regarding continuity, consistency, scaling.
I don't really see why Facet API would help much with that. When switching the backend, you'd still have to reconfigure everything, re-create search pages, users would see new UI, etc. The only thing you get is a consistent way of setting the facets, but you'd still have to re-configure them.
With a framework like the Search API in core, this could easily be changed. Then, search project developers could (and, hopefully would) finally build upon core search, not around or against it, leading to a consistent interface and vastly simplifying changing of backends, or other parts. When a user wants to upgrade from database search to a Solr server, only a few clicks (and setting up the Solr server, of course ;)) are necessary for having exactly the same search as before, only with a much more powerful backend.
Such a core search framework would essentially serve the same purpose as the Facet API, but for all aspects of searching: As with Facet API, developers of new facet widgets, facet types, etc., don't have to develop their features for all search modules separately, a flexible core search framework would lead to developers of additional features being able to develop for just a single system and thereby produce a solution for all kinds of backends.
To me, this seems to support your goal of scalability much better than separate solutions could. And with a framework flexible enough, there would be no need for it to "get the heck out of the way", as it would already allow the use cases where you currently need to drop core search. But I of course agree with you that there also needs to be a search that works out-of-the-box for inexperienced users. I'd just separate it from the "real" core search module – core search, the product, clearly separated from core search, the framework. And later, if the site grows large enough, the user can with a few clicks upgrade to a Solr (or Xapian, Sphinx, MongoDB, whatever) server, without having thought about it at the beginning.
And regarding maintainership: I'll have to port and maintain the Search API in any case. Doing that in core would of course mean more work, but hopefully also more help. And like you, I'm quite passionate about search in Drupal, so would be willing to invest some additional work in what I regard as a huge step forward.
Oh, just saw our combined core conversation – seems like we'll get a chance to discuss this publicly and in person, too.
Thomas, Thanks very much for
Thomas,
Thanks very much for posting. Disagreements = progress, so I take this a super positive thing for Drupal search and very much welcome the discussion in London. Just a point of clarification, this is Peter's core conversation, as usual I am just riding his coattails :-).
To clarify where I am coming from, my role as a member of the Professional Services team has given me insight into what various organizations are trying to implement with search and where their priorities lie. More importantly, my new role as a Solutions Architect allows me to see where Drupal competes and why it wins or loses the selection process. Because of our shared passion for search, I am sure you can understand why I pay special attention to it when speaking with people about Drupal.
Right now Drupal is seen as an inferior search platform to solutions like Endeca and Sharepoint. The reason for this belief is that the user interfaces of the other solutions can be tweaked and modified kinda-sorta easily to produce very interesting displays, mostly to what we would consider to be facets in the Drupal world. Not once has anyone cited the developer experience of the core search API as a factor as to why Drupal was not chosen. Focusing on the faceted displays will solve 90% of the use cases that drive people to other platforms, mainly because Apache Solr Search Integration and now Search API have done a great job providing a competitive baseline of functionality. Whether or not these solutions are in core is really a non-factor in terms of Drupal adoption. The remaining 10% of differentiators are things like indexing non-Drupal data, federated search, etc.
I agree with you that a unified API in core would facilitate the goals I mentioned in the original post, however it is by no means necessary as the community is proving that the gaps can be filled rather quickly without a generic API in core. I want to make it clear I am NOT saying the work you are doing with Search API is unimportant. In fact I will go so far as to say it is critical to the success of Drupal search. I just don't think it is critical to get it into core, and I would rather focus on work that could be leveraged immediately as opposed to concentrating on something that conceivably won't be a viable option for another 3 years (3 more months of D7 maturing, 18 months of D8 development, 6 months for D8 to mature). Search is a small community, so I am not sure we have the (wo)man power to focus the proper attention on both.
What I would like to see happen is for the community to cherry-pick the quick wins that would facilitate improvements of Facet API, Apache Solr Search Integration, Search API, etc. and put them into core. As this begins to happen, I feel the various projects will organically begin to merge (as they are starting to do by sharing common code) in which case the generic, core-ready API will begin to emerge without us having to force it.
Anyways, look forward to catching up in London,
Chris
Partial agreement
OK, with these clarifications I really see your point, this does make a lot of sense. And thereby puts me into kind of a tight spot, as I could end up with 15 minutes of core conversation to fill with nothing to say. ;(
I do feel there are advantages of fixing this in core – however, I have to agree that those might not outweigh the additional efforts. (Of course, I wouldn't neglect the D7 work in any case, so no worries there.) Also, seeing as the Search API is primarily a framework for other contrib modules to build upon: if it does that job well enough, it will be used, and otherwise it won't – no matter whether it's in core or contrib. I'd have said it was a central point to leverage other API modules, like the Facet API – but again, this in no way depends on it being in core.
I guess that I hve before seen it more from the point of view of simple Drupal users who just want to build a simple website – those are much more likely to use core search instead of a contrib solution. However, I guess for those the current solution is already quite good, and they stand to gain little from a cleaner framework or the like. With search_facetapi, there are now even facets available for it!
Thanks for this discussion, and the food for thought! I hope there'll be lots of feedback by others, too, so we can form a broader opinion here, or maybe see it from other angles, too.