Views pager for sites with large datasets

Posted by nrambeck on July 7, 2010 at 5:36pm

One of the newspaper sites I work on has several hundred thousand nodes of content. Because of the fairly large dataset, Drupal's core paging system can take several seconds just to run the count query required to return the total number of pages in a list of content. As a solution, I created a simple module to solve this problem for Views pagers. The module adds an additional pager option called "Lite" that produces a simple pager without executing the expensive COUNT query required by the default pager.

I'm posting this here to see if there is enough interest to add the module to Drupal.org. For now, the code lives at Github. For more information, checkout the project page on my site below.

http://nathan.rambeck.org/projects/views-litepager-module

Comments

Duplicate Module?

Posted by mikeytown2 on July 7, 2010 at 7:08pm

http://drupal.org/project/prev_next

No

Posted by kbahey on July 7, 2010 at 7:26pm

I am the author of the Prev/Next module. It has a different use case, where you want links or thumbnails from one node to the one before and after it so you can "page" through the nodes by chronological/posting order. You are on a node already, not a list of nodes.

The lite pager solves another performance/scalability issue, when viewing the pager on a list of nodes generated by views.

So, yes, go ahead and contribute it, and bonus points if you can extend it to apply to core's pager as well to avoid this query.

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

subscribing. this sounds

Posted by ball.in.th on July 14, 2010 at 10:41am

subscribing. this sounds interesting!!

--
http://ball.in.th - ชุมชนคอบอลพันธ์แท้, ผลบอล

There is another solution for

Posted by ajayg on July 15, 2010 at 5:44pm

There is another solution for pager problem. It does not completely eliminate the count(*) query as your solution but atleast caches the query. There is already patch for views 2.x but that can not be commited till views 3.x patch also submitted.

If you are familair with views 3.x please consider helping to get this patch commited. This will help thousands of sites using views.

http://drupal.org/node/550962

Definitely interested

Posted by spearhead93 on July 17, 2010 at 12:58pm

Anything around reducing the load of Views (pagers, page generation) for media sites is more than welcome.
I'll have a look about this module and will review it as I'm facing some perf issues in a similar context.

You might be not that

Posted by dawehner on July 22, 2010 at 10:52pm

You might be not that familiar with views 3.x :)

There is a solution already which really integrates into views. It's called pluggable pagers for views. You can write your own pager for views!

If you need just a "next page"-pager you don't need to know the count of items.
So just query one additional item.
If you get one result more, there is a next site, if not then not.

This is already implemented on a more or less tested version: http://drupalcode.org/viewvc/drupal/contributions/modules/views_plugins/...

If someone has another idea i could implement it as preperation for drupalcon copenhagen :)

From very fast scanning of the code i think litepager does something else, but i'm not sure.

Dude

Posted by bflora on August 5, 2010 at 7:14am

I just want to go on the record and say that this module has already cut load time by SECONDS on some of my key views. Sir, I am grateful for your contribution. Seriously. This is a life saver.

Can I help in any way to get this ported over to a proper project? This needs to happen.

Module Released

Posted by nrambeck on August 5, 2010 at 3:11pm

I finally got around to adding this as an official project.

http://drupal.org/project/views_litepager

--
Nathan Rambeck

Thanks!

Posted by cashwilliams on February 10, 2012 at 2:52pm

Thanks for solving my problem a year before I even found it!

Working on a site that has 20mil nodes, so uh... yeah... paging search results doesn't go over well

Recommended release

Posted by comat0se on August 5, 2010 at 5:55pm

This sounds pretty interesting. You might wanna take a look at this though.

drush dl views_litepager
There is no recommended release for project views_litepager on [ok]
Drupal 6.x. Ask the maintainer to review
http://drupal.org/node/197584 and create/recommend a release in order
to be compatible with drush and the drupal.org security broadcast
system. A recommended development snapshot release is sufficient.
Alternatively, run pm-releases command and explicity pm-download any
non-recommended release that might be available.