Posted by hellohamish on June 18, 2015 at 11:51am
Hi,
I'm writing a report for a local government client about the suitability of migrating their internal and external sites to Drupal.
They have a very document heavy site with a large number of documents (up to 500,000). The documents are static and won't change so there's no requirement for Document Management functionality. They will require metadata and security as well as the ability to search by metadata.
Has anyone got experience or seen any case studies about storing large volumes of documents in Drupal? Anyone aware of any practical limits on the number of documents Drupal can store without affecting performance?
thanks for looking,
Hamish
Comments
Works fine
I managed a site for a daily newspaper with almost exactly 500,000 nodes when I left. All the base functionality works fine. Occasionally I'd have to hand optimize the queries that Views would generate (with hook_views_query_alter or similar), but in general, no issues.
One thing that can slow down
One thing that can slow down sites with large numbers of nodes are pagers included. Prev Next aims to solve some of this.
On almost any site, the database is the bottleneck. If you use MariaDB with some high-availability optimizations, it will get you a long way.
Other tools like Varnish and NGINX can improve anonymous user experience too.
Optimize with Solr
You could use search_api and search_api_solr modules to improve views performance. However, search_api doesn't load whole node content from solr, it just do the search on solr and solr's json responses is a nodes ids list. Then, search_api get this node ids list and load all nodes (node_load_multiple).It is not cool but if your views have a lot of contextual filters and text search, then Solr comes in handy