Search

Let's improve Drupal's core search. With higher MySQL requirements in 6.x we should be able to do better.

Core search modules: search.module, but also node.module and user.module. Existing contrib search modules: views_fastsearch, porterstemmer, semantic search, facted_search, Apache Solr (I'll add more as people point them out to me)

Here are pending search issues that need review.

There are a few looming search indexing bugs, and some good ideas about performance, refactoring and better hooking. Please help review the above issues. My preference is to focus on improving the queries and search performance, but if you have other ideas and are willing to jump in with patches, please do so.

Chat Module

private
Sciera - Thu, 2009-07-02 01:26

Hi Guys,

Can anyone recommend me a good chat solution for Drupal 6. I am trying to evaluate different solutions like DimDim, 123FlashChat Server, avchat(avchat.com)..

Has anyone used any of these solutions.. Will U recommend any?

I will be using this for a Learning Management solution. I would like to host online presentations, do whiteboarding, have the ability to control the time the chat sessions are on; All chat sessions will need to be stored and indexed for later retrieval. I will be using the apache solr module. That part seems to work fine.

RDF for Solr: Possible implementation strategies

drunken monkey@drupal.org's picture
private
drunken monkey@... - Fri, 2009-06-12 17:29

(For information about my project, see here. Put shortly, it's about enabling Solr to index RDF data via drupal.)

Before starting the actual coding, even on prototypes, the basic options for implementing this will have to be discussed. At the moment, my mentors and I see the following three possibilities:


custom search form and search filter as that in orkut

private
group: Search
deepthi - Sat, 2009-05-16 18:10

HIi

i have a custom search form which wil display users based on cck fields like
first name,gender,location etc .How do i filter these search results based on
search filters like age,location ,whos online etc..same as like that in orkut search.

Thanks.

Adding RDF Support to the ApacheSolr module

private

Project information

Project page on drupal.org: ApacheSolr RDF Support
Student: Thomas Seidl (drunken monkey on d.o)
Mentor: Robert Douglass (robertDouglass)
Co-mentor: Stephane Corlosquet (scor)
Local mentor: Wolfgang Ziegler (fago)

Current status: Implementing first prototype

Description

This project will improve the ApacheSolr module by enabling it to handle (i.e., index and search with a comfortable UI) any kind of RDF data. This will instantly make it possible to provide meaningful searches for all site content that isn't node-centric, as well as content from anywhere else on the web. Only an RDF class description and a way to access the data would have to be provided (apart from the normal Solr requirements) and the module would automatically do the rest of the work.
More information can be found in the original discussion.

Search api sprint

public
group: Search
aufumy@drupal.org - Fri, 2009-03-06 21:44
Start: 
2009-03-07 09:00 - 17:00 America/New_York

Abstract out the sql bits from the core search module into a separate sql search module.

http://drupal.org/node/282192

http://groups.drupal.org/node/13145

Provide the hooks that other search alternatives can plug into.

Help backport ApacheSolr D6 to D5

robertDouglass's picture
public
robertDouglass - Thu, 2009-02-05 21:48

Hi Everyone,

Today I took a couple hours and attempted a backport of ApacheSolr D6 to D5. This will bring all of the cool features that D6 has to D5. The patch needs work. It doesn't yet work. If anyone has time to chip in and work on the indexing of nodes (currently not working) that'd be great. With a little momentum I think we can do the backport relatively quickly, after which keeping the two in sync will be much easier.

http://drupal.org/node/337735#comment-1240921


replacing drupal 6 core search with views search

public
group: Search
viper - Mon, 2009-01-26 21:02

Hi all

I am trying to replace drupals core search with my own views search. the search in views work but only if you type its exact url would you see the views search. and this is where I am having hard time putting the two together.

I am using the function hook_menu_alter to replace the drupal core search with my views search path. heres the code thats on my custom module file:

<?php
function searchy_perm() {

return array('access searchy', 'administer searchy');

} // function searchy_perm

function searchy_menu_alter(&$callbacks) {

Apache SOLR to index a HUGE forum

claudio's picture
public
group: Search
claudio - Tue, 2009-01-13 18:21

I'm quite busy trying to get the latest (dev) Apache_Solr module to work with our complex multi site setup and it all went fine, until we had to import an huge phpbb2 forum (28k topics, near 1 million comments - some topic has thousands of comments).

Now I'm puzzled: which route should I go through? Does anyone has succesfully implemented something like that?

From the top of my mind, I'd consider each comment as a separate document to feed SOLR with, with a rightly cooked "indexer" script that will extract them AND the topic, of course (first the topics, then the comments).


g.d.o. Group Name Searching

andremolnar@drupal.org's picture
public
group: Search
andremolnar@dru... - Wed, 2008-12-03 06:20

Was just catching up on some sessions I missed in Szeged and watched:
http://www.archive.org/details/drupal_search

And it inspired me to find this group here on g.d.o. And while trying to find the group I got this amusing result:

gdo groups search result


OR Search Relevance Ranking

T1TAN23@drupal.org's picture
public
groups: Austin · Search
T1TAN23@drupal.org - Mon, 2008-11-24 19:09

I need some help figuring out the best way to implement keyword relevance ranking in my search results set when conducting an OR search on node content. I have implemented a theme_search_form_alter to build a custom search_block_form and I have everything working with the exception of getting the results to rank properly based on keywords entered into the search_block_form.


N-Gram indexing?

Scott Reynolds's picture
public
group: Search
Scott Reynolds - Sat, 2008-11-15 23:06

So I have been tasked with solving the issue of stemming, mispellings, multilingual and partial matching for a site. I decided to create a quick prototype for n-gram indexing. (http://en.wikipedia.org/wiki/N-gram among other resources) . It is a relatively proven technique for these issues, but creates a larger index. So of course not sure if it is the best route in terms of performance, but I plan on running tests.


Washington DC Search talks

public
group: Search
aufumy@drupal.org - Thu, 2008-10-30 23:47

Is anybody going to give an apachesolr talk at Washington DC Drupalcon?
Earnest Berry, Robert Douglass, Chad Fennell, Doug Green, Michael Hess, Djun Kim, David Lesieur, Blake Luchessi, Jacob Singh, Thomas Seidl?

I will add a session on distributed search and apachesolr, but feel free to join, or run the session.

==============

Current list of proposed sessions on search at Washington DC Drupalcon as of Oct 31, 2008:
* Xapian Search (3 votes)

How do I use text input to search a taxonomy, in a custom search form?

droople's picture
public
group: Search
droople - Tue, 2008-10-07 17:46

I would like a custom search form with search "input form" to query a certain vocbulary.

I have looked at views, fastsearch, filter block

They are ok to use when you want a drop down, not a "text input". I have about 450 terms, from a vocabulary I want to be queried.

I believe a drop down would not be user friendly, also it would load as fast.

On my site, a directyory listing, the search form would be of this nature

|keyword| |location| |business type| Search Button


Test case for advanced search form

robertDouglass's picture
public
group: Search
robertDouglass - Sun, 2008-09-14 09:53

I've created a test case that tests the advanced search form. There are currently no other tests that do this. The patch needs review. Please jump in and give it some attention: http://drupal.org/node/308233


Search Ranking Module

BlakeLucchesi's picture
public
group: Search
BlakeLucchesi - Wed, 2008-09-03 16:28

Hey everyone. I just finished up work on my Summer of Code project that was targetted at further developing the core search. During the course of the project I created a module for Drupal 6 that back ports DougGreen's hook_ranking() patch that made it into HEAD. The module uses hook_menu_alter() and some other work to override the core search implementation completely. This module also includes some other cool new things listed below. I'd like to get some feedback on the changes that have been made because I want to create some patches for the new features to be integrated into HEAD.


Google, Yahoo & Microsoft search services

mfb's picture
public
group: Search
mfb - Thu, 2008-07-17 19:48

I now have Drupal modules for all the big-three dot.com search services, Google CSE, Yahoo BOSS and Microsoft Live. All three can integrate with the core search API as search tabs. Lots of features are still on the to-do list (more accurately, are waiting for feature requests to be filed) like adding advanced search options.

I setup a (temporary) demo of all three here.


Patch to pull core's custom search indexing into a separate backend

public
group: Search
nedjo - Mon, 2008-07-14 12:57

I've posted a first cut at a patch to refactor core search into a generic search module that accepts different backend handlers and a specific handler, core's custom indexing. It's a very basic patch--it just does the minimum to decouple the generic search interface from the backend.

The aim is to permit e.g. apachesolr and other search backends to plug in seamlessly to core search.

Ready for review: http://drupal.org/node/282192.

ApacheSolr Search and other 3rd party Drupal search options

robertDouglass's picture
public
robertDouglass - Fri, 2008-07-11 10:38

This is a BoF discussion that is planned for August 24, 2008 at 16:30 - 17:45 as a part of the FrOSCon conference in Sankt Augustin, Germany.

  • What improvements have gone into Drupal search as a result of the Minnesota Search Sprint?
  • ApacheSolr demonstration
  • Xapian search and Drupal.org

Search is cool. Come talk about it.


Search session for Drupalcon Szeged

robertDouglass's picture
public
group: Search
robertDouglass - Thu, 2008-07-03 20:05

I've proposed and extended search session for Szeged. The timing of the session is important as a way to keep momentum going from the Minnesota Search Sprint and focus on what we can achieve in Drupal 7. It will also provide a chance for people to talk about alternatives to Drupal core, what core can do to support the growing number of 3rd-party solutions, and what core can learn from things like ApacheSolr.

http://szeged2008.drupalcon.org/sessions/drupal-search-where-are-we-wher...

Please vote if you like the session.


Multiple Stemmers seem to conflict

public
group: Search
ducdebreme - Wed, 2008-06-25 07:43

We have a multilingual website providing German, English, French and Italian content on a Drupal 5 instance.
We used to use Drupal's internal search and had many issues about terms that were not found. I found, that the problems might be solved using stemming.
So i installed stemmer for all the languages: porterstemmer, de_stemmer, ...

Node Link Ranking Factor

BlakeLucchesi's picture
public
group: Search
BlakeLucchesi - Fri, 2008-06-20 01:31

Just wanted to let everyone know that I submitted a patch that allows you to rank nodes with more inbound links than other nodes higher in search results using the new ranking hook.

Take a look, give it a try, and please give feedback.

http://drupal.org/node/257216


Improving Drupal's Search Speed Under Load Conditions

techsoldaten's picture
public
techsoldaten - Wed, 2008-05-14 02:04

Trellon recently released the Xapian search module for Drupal. This replaces Drupal's standard search feature seamlessly (except for a core patch) with an interface to the Xapian search engine. There's a post about it over on Trellon.com.


Xapian Search for Drupal - Metrics and Benchmarks

techsoldaten's picture
public
groups: Enterprise · Search
techsoldaten - Wed, 2008-05-14 01:38

I posted some internal benchmarks from the Xapian module we are working on in the Trellon.com blog. This post describes the performance gains using Xapian can have and does some benchmarking on a really, really slow server.

Would love it if someone could verify those results. We are about to take the sample size of the data up to about 1mil nodes and repeat.

M


Some brainstorming notes from the sprint

public
group: Search

Totally disorderly and mostly here for our own reference =)

  • Refactor node/user search implementations into own modules.
  • Control over the search interface.
  • Moving stuff between adv. search form and main search form and/or block.
  • Full search building interfaces. Create search environments; Each env. has own settings. eg. What content types are in search? What does the interface look like? Analog to building a view w/ fastsearch.

Drupal's Search Framework: The execution of a search

robertDouglass's picture
public
group: Search
robertDouglass - Fri, 2008-04-25 07:26

Drupal’s ambitious search module provides a framework for building searches of all kinds. By isolating the tasks involved in searching, and allowing the actual search implementations to be handled by other modules, the search framework sets the stage for all sorts of creative search applications. This article, which applies to Drupal 6, explores the structure of the search framework by following the steps needed to execute a search.

Full article: http://acquia.com/blog/drupals-search-framework-the-execution-a-search


Summer of Code 2008: Search Scoring Improvements

public
groups: Search · SoC 2008

Project Information:

Current Status: Working to complete tasks outlined below and get them ready for acceptance to D7 core. The Drupal project page can be found here: http://drupal.org/project/search_score_improvements. The project page will contain a downloadable package of all code contributed from this project.

Description:

Minnesota Search Sprint

robertDouglass's picture
public
group: Search
robertDouglass - Mon, 2008-04-21 16:14
Start: 
2008-05-09 09:00 - 2008-05-11 20:00 America/Chicago

Continuing the great and growing tradition of bringing people together in small groups to attack focused problems, a search related code sprint will take place from May 9-11 on the campus of the University of Minnesota. Goals of the sprint will be to enhance the search framework, identify improvements to the core search implementations, and consolidate efforts towards making Drupal 7 a fantastic release for search. Sponsors for the search sprint include University of Minnesota Libraries, Acquia, McDean, Inc. / OpenBand, Workhabit, CivicActions, The University of Michigan, Laboratoire NT2, and BoldSource. Attendees include Earnest Berry, Robert Douglass, Chad Fennell, Doug Green, Michael Hess, Djun Kim, David Lesieur, and Blake Luchessi.
Search Sprint Sponsors


Test spelling suggestions for D7

robertDouglass's picture
public
group: Search
robertDouglass - Wed, 2008-04-16 17:49

http://drupal.org/node/247482

This patch adds spelling suggestions to the page that is returned if no search results are found.
Spelling suggestions

This is done by utilizing the Levenshtein algorithm for calculating the nearness of words.


Test spelling suggestions for D7

robertDouglass's picture
public
group: Search
robertDouglass - Wed, 2008-04-16 17:49

http://drupal.org/node/247482

This patch adds spelling suggestions to the page that is returned if no search results are found.
Spelling suggestions

This is done by utilizing the Levenshtein algorithm for calculating the nearness of words.


Search issue queue - dig in!

public
group: Search

This is a list of issues in the D7 issue queue that pertain to search. They've been vetted and confirmed to be real issues. They cover the entire spectrum, from advanced feature requests to bugs to theming issues. This is a great place to start your quest to make search in D7 a better thing. The table should be re-organized a bit. We should sort it into bugs and features, and make some priority judgements. Remove rows as issues close.

Google Search Appliance Module Released!

public
group: Search
JacobSingh@drup... - Tue, 2008-04-08 19:09

Hi folks,

Please see Google Appliance Module for Drupal .

This module currently supports keymatches, keywords, i18n compat (see the module page for more info).

I will be updating it regularly with new features, as the base class is robust enough to handle an advanced search, sorting, etc. It's currently in beta because I'd like some more feedback on the architecture / integration stories before rolling out more features, but a version of this is in production on a large site.

Google Search Appliance Module Released!

public
group: Search
JacobSingh@drup... - Tue, 2008-04-08 19:09

Hi folks,

Please see Google Appliance Module for Drupal .

This module currently supports keymatches, keywords, i18n compat (see the module page for more info).

I will be updating it regularly with new features, as the base class is robust enough to handle an advanced search, sorting, etc. It's currently in beta because I'd like some more feedback on the architecture / integration stories before rolling out more features, but a version of this is in production on a large site.

Search Scoring Improvements

BlakeLucchesi's picture
public
groups: Search · SoC 2008
BlakeLucchesi - Mon, 2008-03-31 04:44

Background


Search Scoring Improvements

BlakeLucchesi's picture
public
groups: Search · SoC 2008
BlakeLucchesi - Mon, 2008-03-31 04:44

Background


hook_search

mihksoft's picture
public
group: Search
mihksoft - Thu, 2008-03-27 21:36

Hy!
I have a table author with author_id, author_first_name, author_last_name, author_date_of_birth, author_date_of_death, author_website, author_email, author_language_id, author_isDeleted.
I have make a module author, implement the hook_menu and make a table view for my table and a modify form. From the content I have make a new content type called Author. I have implement the hook_search in my module

function author_search($op = 'search', $keys = null)
{
switch ($op)
{
case 'name':
return t('Content');
case 'search':
$find = array();


hook_search

mihksoft's picture
public
group: Search
mihksoft - Thu, 2008-03-27 21:36

Hy!
I have a table author with author_id, author_first_name, author_last_name, author_date_of_birth, author_date_of_death, author_website, author_email, author_language_id, author_isDeleted.
I have make a module author, implement the hook_menu and make a table view for my table and a modify form. From the content I have make a new content type called Author. I have implement the hook_search in my module

function author_search($op = 'search', $keys = null)
{
switch ($op)
{
case 'name':
return t('Content');
case 'search':
$find = array();


SearchAPI Module

BlakeLucchesi's picture
public
BlakeLucchesi - Wed, 2008-03-26 19:10

The following is my first revision of a proposal to create a search API module. I'd love to get some feedback.

Project Details
A Drupal search API would allow for separation between the search interface that end users interact and the back-end indexing and retrieval work that a search engine performs. The advantages to creating a search API are:


SearchAPI Module

BlakeLucchesi's picture
public
BlakeLucchesi - Wed, 2008-03-26 19:10

The following is my first revision of a proposal to create a search API module. I'd love to get some feedback.

Project Details
A Drupal search API would allow for separation between the search interface that end users interact and the back-end indexing and retrieval work that a search engine performs. The advantages to creating a search API are:


Improving the Apache Solr Search Integration module

drunken monkey@drupal.org's picture
public
drunken monkey@... - Wed, 2008-03-26 16:47

I am planning to hand in a proposal on improving the Apache Solr Search Integration module.
The project would include:

  • Porting the module to drupal 6 (if necessary)
  • Integration in Views 2, enabling the use of Views 2 as a front-end to display the search results
  • Writing simpletest unit tests for this module, especially for the new functionality

What's your opinion on that? I have already contacted Robert Douglas to ask for his.


Improving the Apache Solr Search Integration module

drunken monkey@drupal.org's picture
public
drunken monkey@... - Wed, 2008-03-26 16:47

I am planning to hand in a proposal on improving the Apache Solr Search Integration module.
The project would include:

  • Porting the module to drupal 6 (if necessary)
  • Integration in Views 2, enabling the use of Views 2 as a front-end to display the search results
  • Writing simpletest unit tests for this module, especially for the new functionality

What's your opinion on that? I have already contacted Robert Douglas to ask for his.


Refactoring core search

public

Drupal's search APIs received some good attention at the recent Boston Drupalcon. Following up on discussions there, here is an attempt to draw together ideas on directions for refactoring core search. Please wade in and add your ideas and observations.

Existing core search

Drupal core search is implemented in an integrated way, providing a powerful working solution but little flexibility. Core search integrates several distinct pieces, among them:

  • For nodes, a custom SQL-based indexing solution.
  • For nodes, an SQL-based search algorithm.

Dynamic content view

garthee's picture
public
groups: SoC 2008 · Search
garthee - Thu, 2008-03-20 08:12

I think the way information is presented and search results are displayed can be optionally enhanced. This somewhat related to the following discussions, however a new idea
1. http://groups.drupal.org/node/9934
2. http://groups.drupal.org/node/9946

A short description of what I would love to see with this module


Dynamic content view

garthee's picture
public
groups: SoC 2008 · Search
garthee - Thu, 2008-03-20 08:12

I think the way information is presented and search results are displayed can be optionally enhanced. This somewhat related to the following discussions, however a new idea
1. http://groups.drupal.org/node/9934
2. http://groups.drupal.org/node/9946

A short description of what I would love to see with this module


Sphinxsearch integration

johsw@drupal.org's picture
public
johsw@drupal.org - Fri, 2008-03-14 14:41

Following Yelvington I'm crossposting this to the following groups: SOC2008, Knight Foundation, Newspapers on Drupal and Search

One thing that would be really cool, is a module integrating http://www.sphinxsearch.com/ and Drupal. Our experience is that core search doesn't play nice when you have alot of nodes (we have 150.000+). Indexing simply kills the server.

So instead we use Sphinx. It's REALLY fast, both when searching and when indexing. BUT everytime we alter our content-types we have to manually reconfigure the sphinx configuration. This is why I propose this as a module for the SOC08 - a module that integrates Sphinxsearch and Drupal


Sphinxsearch integration

johsw@drupal.org's picture
public
johsw@drupal.org - Fri, 2008-03-14 14:41

Following Yelvington I'm crossposting this to the following groups: SOC2008, Knight Foundation, Newspapers on Drupal and Search

One thing that would be really cool, is a module integrating http://www.sphinxsearch.com/ and Drupal. Our experience is that core search doesn't play nice when you have alot of nodes (we have 150.000+). Indexing simply kills the server.

So instead we use Sphinx. It's REALLY fast, both when searching and when indexing. BUT everytime we alter our content-types we have to manually reconfigure the sphinx configuration. This is why I propose this as a module for the SOC08 - a module that integrates Sphinxsearch and Drupal


Building a killer search for Drupal

public

We've had a good discussion today at Drupalcon, in a BoF session led by Robert Douglass. Here's the plan that emerged to build a killer search for Drupal that will help take us Drupalers further towards world domination. ;)

Building a killer search for Drupal

public

We've had a good discussion today at Drupalcon, in a BoF session led by Robert Douglass. Here's the plan that emerged to build a killer search for Drupal that will help take us Drupalers further towards world domination. ;)

Finduser - a new search module for .... finding users

robertDouglass's picture
public
group: Search
robertDouglass - Sun, 2007-11-25 22:04

http://drupal.org/project/finduser

This is a custom search module for users. It provides a search page that can search for users by username, email, or a custom text field. The custom text field can come from any content type, thus this module plays nicely with any user-as-node strategy.

The module does one thing and does it well, and is independent of search module. It has enough customization options to allow you to bend it to your will, as well as a healthy number of themable functions.

On the administer settings page for the Find user module you have the chance to set many configuration options. If you have CCK enabled and have added any text fields to any content types, these fields will show up as options to be searched. This allows you to use a module like nodeprofile or usernode or bio to extend your user profile and make it searchable via the Find user module.


Finduser - a new search module for .... finding users

robertDouglass's picture
public
group: Search
robertDouglass - Sun, 2007-11-25 22:04

http://drupal.org/project/finduser

This is a custom search module for users. It provides a search page that can search for users by username, email, or a custom text field. The custom text field can come from any content type, thus this module plays nicely with any user-as-node strategy.

The module does one thing and does it well, and is independent of search module. It has enough customization options to allow you to bend it to your will, as well as a healthy number of themable functions.

On the administer settings page for the Find user module you have the chance to set many configuration options. If you have CCK enabled and have added any text fields to any content types, these fields will show up as options to be searched. This allows you to use a module like nodeprofile or usernode or bio to extend your user profile and make it searchable via the Find user module.


Fuzzy Search Needs Testing

BlakeLucchesi's picture
public
groups: Search · SoC 2007
BlakeLucchesi - Sun, 2007-08-19 08:00

As I am entering the home stretch of the project I have gotten a small amount of feedback from various testers regarding my fuzzy search module but so far have gotten no feedback on how good the algorithm is at returning relevant results. I have just recently reworked my algorithm to tighten up the matching. I'd really appreciate it if I could get a person with a site that has over 400+ nodes or so to test out the module and see how fast it performs and how relevant they believe the results are.


Syndicate content