Hello Community,
I am currently working on a non-commercial research school project. We are facing the following decision and I was wondering if you have some advice or experience in this manner.
Problem: Creation Search engine (+Community functions etc) for our research field. Additional Tagclouds, coauthoring etc to enhance the search.
Solution: We have created a MySQL DB which contains 2.5M entries (one entry is one paper). It includes the Metadata as well as the abstracts.
We were thinking to create an index with LuSQL and load this into Drupal, access it with Luceneapi. However we could not archive. Reasons for that we are thinking are the incapability of Index Versions (LuSQL creates 2.9, Drupals Luceneapi can only handle 2.2).
A second approach could be to creates nodes of the abstracts, and let create the index by Drupals Lucene.
However, we are talking about 2.5M Abstracts, as well as frequent cronjobs in order to keep the index up to date.
The integration of the nodes should be autmated, parsed with a pdf parser.
Do you have any advice for us how to proceed / any best practices available?
Thank you very much in advice.
Patrick
Comments
CCK
Have you thought about using a CCK text field for the abstracts?
Why not LuSQL Solr and
Why not
LuSQL
Solr
and Drupal.
Data could go directly in solr server for best performance. Drupal only as frontend for the solr server. Not the typical way but best performance. You must write some code to get the data out of solr.
Views 3
Look into views 3 for directly querying solr
http://www.angrydonuts.com/views-3-x-roadmap
http://drupal.org/node/695302
My company has created a
My company has created a heavy-duty journal database system using Drupal which sounds like it might be the sort of thing you are after. It is written with research and library standards in mind. The URL is http://www.sciquest.org.nz. Take a look and if there is anything there that might be of use to you send me a private message and we can disuses further and I can show you the back end.
Thank you guys
Thanks a lot guys.
I also had posted an issue regarding bix indexes and received a useful answer. http://drupal.org/node/739294
So as it looks like we need to shift really as you said Mikey and Bennos. We installed an Apache solr who already has successfully created an index. We are now about to modify the drupal integration module to let it shown correctly in our page.
We dont quite trust views 3 cause its only an alpha2 version.
I wrote you an Email Aaron, you page looks very promising!
Does anybody has other experiences regarding this?
Thank you!!
Views 3
Hey patjov,
We are currently in the process of using views 3 on our production system in conjunction with apache solr views. There were a few bugs that should be pushed back upstream but mostly feature ready (for our purposes at least). I has involved upgrading some of our more custom views 2 implementations.
It is going into regression/load testing hopefully Thursday so I should be able to report back next week on the success of that.
Just to give you a sense of siize this client has about half a million unique visitors a month with ~20million page views
Regards,
David
Hello guys, thanks for the
Hello guys,
thanks for the help.
We adopted it slightly differently but very successfull.
you can check it out under http://wireportal.informatik.uni-mannheim.de/drupal/
The solution looks sth like this:
creating lucene index with apache solr with DIH from mySQL database
accessing it via drupal module.
Customizing all layers within search and result process within the drupal system in order to get through non-standard or cck fields but rather our own DB fields..
No views involved so far, cause we adopted the search module and the templates.
Regards,
Patjov