Solr Next Gen - the 7.x-2.x refactoring

Look forward to seeing the next version happen! The objectives are really spot on.

Can we set a vague though doable time frame for an alpha release?

I haven't been following the progress on the D8 Search API (e.g. http://groups.drupal.org/node/71988), but as some of our efforts will overlap, we should share ideas with that group.

Awesome ideas. Excited to see how this shapes up in addition to helping out in any way I can. There are a couple of projects I am working on that I think apply to this new architecture initiative.

The first project is called Facet API. It's main goals are the following:

• Provide a simple API that allows any search backend to create and manage facets. Currently the Search Lucene API and Apache Solr Search Integration projects are supported, and the plan is to add an adapter for Searchlight.
• Make the API backend agnostic so facet configurations can be shared across the various modules.
• Introduce the concept of facet realms which render facets in a similar fashion, for example as a list of clickable links or series of form elements.
• Provide a simple, pluggable widget API that allows developers to build varying displays of facets within a realm. One example is a tagcloud widget.
• Provide a UI component that serves as a GUI to the underlying APIs which will allow for non-developers to create interesting faceted displays via the admin interface. Ultimately this is a facet builder.
• Integrate with Context and Features so the facet configurations can more easily be packaged and shared.

The benefit of Facet API being outside of Apache Solr is that it is really stress-testing the developer APIs and making sure they are as friendly and flexible as possible. In addition, it is backend agnostic which allows non-Solr users / developers to build rich faceted interfaces that the Solr community can benefit from as well.

The second project is Search Lucene API which targets a completely different different audience, but there is a ton of overlap in terms or architecture and overall goals. Some of the features in Search Lucene API (current and future iterations) that I would love to see in Apache Solr are the following:

• The ability to more easily add new index connections. For example, a site may want to connect to an additional Solr instance that is managed outside of Drupal. Search Lucene API attacks this problem by putting to basic administrative forms in the core Framework. Any module that implements the index info hook gets the basic administration interface with connection parameters, index information, etc. for free. For the sake of this post, lets call the new hook hook_apachesolr_index_info(). If we add a hook, we can easily add a UI component that will allow users to add a new connection via the admin interface.
• The ability to add sub-indexes, or separate more targeted searches. Sticking with the hook above, there could be a "base index" key which could, for example, point to the normal apachesolr_search index and inherit it's connection parameters. Either by using the Facet API module, creating an option to automatically apply filters, or finding some way to hook this into the Views interface, users could create things like blog or forum searches without having to do any custom coding.
• Implement a hook to define boost fields. There is currently a variable that stores boost fields which you can work with to add boost fields, but it would be a little more transparent to add the fields via a hook. Sticking with the hook_apachesolr_index_info() above, the boost fields would be one of the administrative settings you would get out of the box by implementing it.

There are a few ore things that could be cool, but this post is getting really long. I think the three things above require some refactoring at the lowest level of the module, and the flexibility provided by them could be an enabler for some other cool features to be build on top of Apache Solr in contrib.

First off: Great initiative! The module really needed a revamp and this seems to move in exactly the right direction. I hope I can be of help regarding this process.

Past experiences

Robert asked me to write a bit about my past experiences with writing contrib modules for the apachesolr project. These are two GSoC projects I did, the Views integration in 2008 and the Apachesolr RDF module in 2009. Sadly I have already forgotten half of it, but maybe what I got can still help to flesh out some requirements for the Solr Next Generation.

• The basic problem I had with the module in several places, was the assumptions it made about you wanted to do with a function, class or the whole module. Everything was built to index nodes on a single server and then to display a page with a single search on it. Creating different servers or no server at all, indexing something other than nodes, not using the standard search page with URL parameters or not wanting facets to display could all be only done in slightly hack-ish manners. E.g., something like executing a search with the Drupal_Apache_Solr_Service class depended on a variable ("apachesolr_index_status" or the like – telling, basically, whether indexing was turned on by the user) being activated, even though I wanted to search a completely different server.
  What I've read here seems like most of this is already addressed, but since this was my main isssue with the module, I still wanted to mention it. I also don't think this is still quite as bad as back then.
• Another thing was a confusing API, with some functionality being split between several magic functions, and other functions on the other hand doing several things at once (where a programmer of another module might only need a single one of them).
  For Solr NG the scope of single functions should be better thought through, and also important API functions that could or will be useful for other projects should be well documented.

OK, this is pretty poor as far as feedback goes, but it's been quite some time. I also think, the current apachesolr module already fixed some of these problems. Anyways, maybe this helps you.
But what I wanted to mostly talk about was my current project.

Search API

Before implementing anything other than the Solr connection library, I strongly urge you to take a close look at my Search API project (if you haven't done so, already). This already takes care of several important features and reading this discussion I really think that basing your module upon the Search API would remove a lot of duplicated effort from this project. For example, my module already takes care of the following:

• Management of different search servers and seperate indexes.
• Indexing management (list of unindexed and updated items) [OK, this one is very basic]
• Indexing any kind of entity (basically your whole "Drupal Adapters" item)
  Right now, all core entities are supported out-of-the-box, a programmer who wants to index and search for his own entities just has to implement hook_entity_info() and hook_entity_property_info() (which he should do in any case) to achieve that. All search functionality is then available for his entities.
• Very flexible indexing (and searching) workflow
  The fields that should be indexed, as well as the way in which they should be indexed, is completely configurable. All fields from related entities, e.g. a node's author's mail address, are also available if desired.
  Almost any special features in any other search module can already be implemented with the Search API by writing custom preprocessors or "data alter callbacks".
• Backend independence
  With a Solr module built upon the Search API, e.g. users previously content with a database-based search could easily switch to a Solr search when their site gets larger, while keeping all their settings and just needing to re-index their data. Especially once more features are available for the Search API, this will be critical for making switching the search backend as painless as possible. With a completely seperate Solr module, users would have to set up additional fields, facets, views and the like all over again.

Additionally, I'm at the moment starting to implement three additional important features (which will be completed over the course of the next few weeks):

Facets

The module will soon support creating facets for any non-fulltext field, as far as possible with backend-independence and for any kind of entity.

Views integration

Once this is completed, users will be able to create views with search keys as filters or arguments, searching or filtering on any indexed property defined for the index and displaying all of an entity's properties the way they want. Facet support will also be included right away, so that facetting views queries can easily be set up.

Apache Solr backend

In fact, I'm already developing this one, and it will be completed in one or two weeks. So after finishing the base library, you could just test out the Search API's Solr backend. If it meets your basic requirements, you could then just take it over for the apachesolr-7.x-2.x branch and apply any modifications or add any additional features you had in mind. This would be both a great test and chance for the Search API, as well as a considerable ease for you, as you would only have to implement Solr-specific features and maybe help to improve the Search API in a few places.

It would be really cool, if you thought about it. I think the Search API can help you a lot with builidng a better Solr search integration, and keep you from wasting time with duplicating things I've already implemented.
But in any case, I'd be glad to help in any way I can, as far as my time permits.

Another thing, regarding my task of implementing a Solr backend for the Search API: I'm currently thinking about which library to use for communicating with the Solr server, and Robert told me it would be best to ask here.
Of course, using your new library, once it's finished, would be best, but since I'm already working on it I have few options but to stick with another one now and then switch to your library later. So, do you have any recommendation on which library you would use, if you wouldn't implement your own?
The PECL library looks great, but it has to be added in the php.ini, which won't be an option for a lot of users. And your current Solr API looks cool, too, but relies on the apachesolr module at the moment, as far as I can see.
So, any tips? Or is the SolrPhpClient used by the apachesolr module right now the best option anyways?

@ Chris: The Facet API looks interesting, too, yeah. A pity it's not usable, it would be great to build on it when implementing facets for the Search API. But I'll take a good look at it and try to keep as much compatibility as possible, so I can switch later when your module is released.

http://drupal.org/node/915626

It goes back to the fl and hl.fl parameters that Solr expects to be comma separated (a weirdness of their API). Our new API needs to make the addition of parameters and values easy and safe in our hook and drupal_alter based architecture. The array based parameters work out ok, but comma separated parameters have the tendency to get clobbered, so our API needs to wrap this and manage the values in a safe way.

I am wondering if this initiative is still under work or if it has been cancelled. Any idea?

For 8.x, Nick_vh and I have discussed the idea that the main apachesolr module could displace search_api_solr as the connector to the index.

So you would want to base the D8 version of your module on the Search API? We should discuss this in Prague – it would be good to know before I begin to port the Search API Solr module. (Although this probably won't happen for another month or two, so you still have time to decide in any case.)
Also, you might want to get involved in the Search API porting process to identify any issues you might encounter, or have already noticed with the D7 version.