Posted by bart.hanssens on March 16, 2012 at 8:09am
Hi,
I'm looking into stock / inventory software, so I'm just wondering if anyone has experience with Drupal sites containing several millions of entities / nodes, but with relatively few users.
The system would need to cope with
- a few hundred storage facilities
- each containing hundreds of of storage racks
- each rack containing perhaps thousands of "pigeon holes", where items can be stored
- several types of storable items (content type with different fields)
- each physical item would get a unique QR code
- a history of several years is to be kept, while items enter and leave ("unpublish" ?) the storage facilities
- from time to time, an overview (Views ?) of all the items currently in a "pigeon hole" or an entire rack.
It seems relatively simply to build a small demo, but I'm worrying about the scalability.
Thanks
Comments
Well, I for one have seen
Well, I for one have seen some awful code in places which would kill scalability for millions of nodes.
The first place that springs to mind is in admin/content which does a "table scan" type query to determine if there are any nodes with alternative languages. (can't remember the exact location but remember seeing it and thinking "eek" ...)
This particular example would be relatively easily fixable but I have no doubt there are other areas of code where developers haven't thought through the scalability implications fully.
Why not use the devel module to have a go at creating a few million nodes and see what happens ?
(and report back !)
edit: for clarification, at some point, I'm going to want to build a site with lots of nodes / taxonomy terms. I believe it's possible, but I would expect to have to delve in and start contributing core patches.
I think that the "node" and
I think that the "node" and "fields" concept is great for a flexible system - but each field is stored in a different table - so a simple query could involve joining many tables together - when you multiply this by millions... well, in my opinion, it is just not designed for that!!!
BUT... that does not mean you can't create your own tables for the BULK data - you could even enable them as entities, etc.. - and use the "nodes" for your upper level structure.
In fact, if you are keeping a history for serveral years, this historical data should be in its own tables away from the "active" data. If I remember correctly, Amazon keeps only a small portion of your purchase history active. Older hisory is slower - but by that time, the user is looking for specific information and is willing to wait a little longer!
http://geoff.com.au - http://redronin.com