Multisite Search using ApacheSolr module

Yes :)

Kyle Mathews

I am far from an expert on Solr (ok, not at all) but I'll give a go at describing what you'd do and rely on people who actually understand how this is done to correct me where I stray.

First you set up Apache Solr on a server somewhere.

Then you install the ApacheSolr drupal module on each of your drupal sites. Make sure to install apachesolr multisite search module as well. I've never configured this so you'll have to figure it out but basically each Drupal site needs to point at your Apache Solr server.

Finally, for each non-drupal site, you'll have to write a custom indexer. Solr is a web service. To index your other sites, basically you just need to create a big XML document with all the necessary information and send it to Solr. I haven't done this before but I'm sure the Solr documentation explains this very well. You could probably look at the ApacheSolr module code as well for some hints although most of that is probably Drupal specific.

Once all of the sites are indexed, Solr will let you search just one of the sites or all of them together. The drupal apachesolr module is pre-configured to do all that.

That'd be really powerful, to have Nutch and Drupal's ApacheSolr working closely together. I work at my university right now. Universities tend to be a hodge podge of different website technologies. It'd be really powerful to be able to crawl all the sites using Nutch and then search across the sites from a Drupal install.

Kyle Mathews

Hey all,

I found the above article extremely helpful and was able to get nutch and solr working together. I then tried to create a schema.xml that was an amalgamation of the one from Nutch and the one from the ApacheSolr Drupal module - this essentially just meant adding some of the Nutch fields to the Drupal schema, and a copyField. Nutch didn't complain, it sent its docs to solr no problem. However the Drupal indexing no longer worked. The conflict was the copyField, which is as follows <copyField source="url" dest="id"/> - Nutch uses the url as the unique key for documents, whereas Drupal uses a hash of the nid. When I removed that, I could send Drupal content to solr again. So I'd basically have to keep changing this one line in the schema depending on which content I want to index.
I think there's probably a clever way around this that hopefully doesn't involve delving into the guts of the SolrIndexer plugin in Nutch - probably all that's needed is a nutch-specific id field (one that will always be empty for Drupal docs) that can be copied into the id field - the problem with the url field is that Drupal docs use it too.

There are other problems with trying to use the two together though. I can't see how to get much control over the html parsing and field-mapping of nutch content and can't imagine how you could get facets covering your content from Nutch. Oh and to get Drupal to even show the nutch docs in search results I had to hack the ApacheSolr module in a couple of places.

We're probably not going to use nutch after all anyway so I'm currenlty investigating other options... just wanted to share my fndings ;-)

Katherine

Hmm thanks for the post, I definitely didnt know about the copyField issue. Facets were a concern but ultimately from my Googling Im lead to believe that facets are not possible/viable in the case of a Multi-site search.

That plus the lack of control over parsing(like you mentioned) and the extra time effort involved made us ultimately go with a different solution. Im still looking into this as and when we get I get time....

I will be presenting a solution to the Nutch and Drupal problem at Drupalcon SF http://bit.ly/dftBpU. I am madly trying to finish the Nutch Module for this date. On the multisite side of things we are currently developing a module that should make this a little easier

Regards,

David Stuart