Posted by geoffb on November 30, 2007 at 7:54pm
I'm wondering if anyone has any thoughts on whether Drupal will ultimately buckle under the weight of too many modules.
I turn to this forum because, as newspaper sites wanting to entice readers and enhance their experience with all sorts of 'bells and whistles', we are more likely to want to add a lot more modules than a lot of other websites.
But, we're wondering how adding more and more modules will affect future performance and whether Drupal will ultimately be able to handle it.
If you read Server indigestion: The Drupal contributed modules "open buffet binge" syndrome (http://2bits.com/articles/server-indigestion-the-drupal-contributed-modu...), the prognosis seems to be dire.
Will the "sky start falling" if we keep on the modular path?
Comments
big media newspaper companies
have money to throw at the problem via hardware, though. ;)
You do bring up an interesting point for discussion, though. I'm currently looking at this because on one of my Dsites (Anderson Free Press) I recently gained 60+ new members that contribute ... a lot. I hadn't really looked to close at the site set-up before because it hasn't had much traffic in its six or so month life so far. It's different trying to optimize the site for lots of registered users versus lots of anonymous readers/grazers.
Might be nice for someone to put together a Handbook page about how to optimize a Drupal install? Best practices for module use?
That might be a way I can contribute back to the Drupal community - as time allows.
Unless someone else wants to step-up and write it. (Or maybe it's around here somewhere already...)
K. Paul Mallasch - Publisher
kpaul media
Bits here and there
If you feel like searching the forums, I've seen optimization tips here and there in forum posts. I'm not aware of a handbook page for it but the handbook is pretty big and I havent' seen it all. I suggest searching there (use advanced search to just search the handbook) and go ahead and write something up if there isn't anything. Anyone can create a handbook page. If you find something already that's lacking, just ask to be on the doc team and you can edit it.
I, for one, would welcome a document on this as I have a site with around 100 modules that is likely to have 120+ by the time I'm done. I finally got it moved off shared hosting but I'm worried it will kill the VPS as well if it ever gets popular.
Michelle
PS: What's your secret to getting members to contribute? :)
See my Drupal articles and tutorials or come check out life in the Coulee Region.
MySQL slow query log identifies bottlenecks
Throwing money at a problem has a downward sloping marginal return. Yes, ample RAM makes Apache happy but some modules flip massive SQL queries like they were pancakes. In the case of GazetteNet.com, monitoring the MySQL slow queries log helped us find our bottle necks.
http://dev.mysql.com/doc/refman/5.0/en/slow-query-log.html
I suppose it would be handy if there was a module that parsed these logs and was able to trace the queries back to the specific module/page load.
ram usage
Server memory is key.
Drupal will run as many modules as you ask, but your php config may need tinkering. If you get the white screen of death, inadequate RAM is often the cause.
We run APC as well, which of course requires extra RAM as well.
And for best performance you can have mysql store the sessions table in RAM.
more on ram
IF you don't have access to your httpd.conf file, you can bump it in your .htaccess... Don't go too crazy with it, though.
php_value memory_limit 30M
Just stick that (with whatever value you want) near the top of your .htaccess.
re: more on ram
Alas, when you start using a lot of modules you can easily need 90MB RAM, in which case each Apache process is eating nearly 100MB. To get Web 2.0-ish functionality you need to use a lot of modules (or create custom modules encapsulating this functionality, in which case your upgrade path is more complicated).
db optimization, too
We also convert many of the heavy-use tables (sessions, cache) to InnoDB instead of MYISAM.
Check the various Handbook threads on optimization and the group: http://groups.drupal.org/high-performance
--
http://ken.therickards.com/
http://savannahnow.com/user/2
http://blufftontoday.com/user/3
--
http://ken.therickards.com/
Memcache is worth considering
As for the heavy-use tables, I would also recommend memcache [0]
Advcache [1] is also very cool, but you will need to patch Drupal core.
For more documentation on Drupal optimisation check out there:
http://2bits.com/
2bits is providing high-performance consultancy for Drupal, and I am strongly recommending them if you need Drupal optimisation. (I am not from 2bits ;) )
[0] http://www.drupal.org/project/memcache
[1] http://www.drupal.org/project/advcache
--
http://www.rue89.com
Damien Cirotteau
http://www.rue89.com
memcache
Yes. Definitely memcache.
--
http://ken.therickards.com/
http://savannahnow.com/user/2
http://blufftontoday.com/user/3
--
http://ken.therickards.com/
thanks guys,
this is all great info. none of my sites have really hit the level where i need something like this yet, but i'm getting close - especially with so many active users on AFP.
-kpaul
kpaul media
Server load
Unless there's something I don't understand about Drupal, I can't imagine number of modules would really be an issue -- it's more about server load ...
In other words, a 1,000-module site should perform better than a single-server, one-module site that gets 1 million visitors in a day .
H.
from what i know
I'm more of an old school webmaster - little knowledge in lots of areas rather than a hardcore techie, but AFAIK for every module you have in the /modules/ directory (even if it's turned off or disabled!) the more RAM Drupal uses to load the site for each pageview. So, it's entirely possible for a shared server environment (or even a VDS/VPS class server) to be taken down with relatively few pageviews but lots of modules.
HTH.
-kpaul
kpaul media
Yes, very
I have a social networking site that has around 100 modules and barely any traffic and it just kills shared hosting. I was getting WSOD all the time and the load times were terrible. I had to move it to a VPS, and that's with maybe 1 or 2 visitors a day. So the memory issue really is a big one.
Michelle
See my Drupal articles and tutorials or come check out life in the Coulee Regio
Not exactly
That used to be true, but in Drupal 5 things changed. Modules that are disabled are not loaded except when viewing Administer -> Site Building -> Modules.
In D6 there are a lot of performance optimizations, including changes that make it easier for developers to conditionally load PHP files only when they're needed, and many core modules have been split up for performance improvement. I'm unclear exactly how this impacts sites that use an in-memory performance cache like APC, however. It may be that the primary beneficiary is sites on shared hosts with memory limits.
Thanks...
I really did mean to say on the /admin/build/modules/ page... That's good to know about D6, which I haven't followed too closely (only so much time!)
I've seen a comment here and there about the PHP5 discussion. I hear there are lots of performance gains with taking that leap...
Speaking of shared hosting, anyone out there with knowledge about mediatemple's new 'replacement' to typical shared hosting? Any of their clients have anything good or bad to say about them in general?
I'm thinking of moving one of my sites to a VDS with them in the near future...
-kpaul
mediatemple grid server
I'm currently trying out the mediatemple grid service option. I haven't done much business with other hosting companies and I'm not a professional sysadmin, but so far it has been a good experience and seems to be a good value at $20/month.
The customer support has been prompt, their knowledge base is fairly detailed, and the server itself seems to be performing reliably. The UI tools to manage the creation of databases, user accounts, etc. are all working fine and I'm feeling pretty good about launching a single drupal site on a grid server instance. I think it would be pretty easy to add new "grid containers" to add db capacity as well but I haven't tried that yet.
That said, I haven't actually launched any live sites yet so we'll see how it turns out when hopefully something is live in January and I'm (hopefully) burning through 'GPUs' with all the site traffic.
Hope this helps...
media temple
Hi All,
My name is Keven Ages and I'm the lead web developer for FourthWallMedia.com in Toronto, Canada.
Last year our company decided to evaluate a few different hosting companies by purchasing low-end space on their servers. Our strategy was to just be treated like a 20 dollar a month customer before we made any partnerships or married one hosting company.
I pitted two major hosting companies against each other, 1and1 vs MediaTemple. MediaTemple's package was the low-end 20/month package (Grid Server) while 1and1 was the developer package (also 20/month).
First things first, we stuck Nagios on both hosting companies to observe uptime/downtime.
While Media Temple has some nice 'features', and I was excited to know you can run Mongrel and Ruby containers, while 1and1 had more of a traditional host offering in terms of technology.
1and1 has won hands down.
We have had approximately 25 hours of downtime with Media Temple (this is totally unacceptable) compared to 100% uptime with 1and1. Client sites on Media Temple will often arbitrarily go down for 15 min to 1 hour, with no heads-up from Media Temple. They also do a lot of 'server upgrades' which take the sites down for up to 3 hours a month. Now given these are done at 3 am, one of our clients is a hotel property which has clients from all over the world. We started getting complaints from their clients in Europe that they were unable to make reservations at peak times in their country.
The support at both hosting companies is top notch, but Media Temple often gives hazy answers to downtime when it is not scheduled, making us feel like we were a low priority for their support team.
I also find Media Temple to be surprisingly slow and sites that are optimally built are sluggish even when the 'grid' is apparently performing well.
Also if you compare the 2 hosting companies/packages you will notice that the 1and1 offering is much better.
Hope that helps someone with the same question.
Great timing!
Thanks so much for your comments, Kevin! I also have Media Temple and 1and1 on a short list. I have heard other similar reports about Media Temple. 1and1 has just leapfrogged ahead on my list.
Get what you pay for?
Maybe it was just the GS service from media temple? I know the technology on that was relatively new.
I've had a DV with them and have been really, really happy. Especially like the auto-updates of backend software and ability to grow w/out changing DNS or moving files. I guess you have to think about long term growth too?
K. Paul Mallasch - Publisher
kpaul media
it was a GS account. I
I realize it was relatively new but after almost a year it hasn't improved. A service being 'new' in the world of web hosting (imho) is not an excuse for bad service. If said company wasn't ready to release their product cause of 'bumps' they shouldn't be using customers to do beta testing w/o compensating them -- that's a very M$ way to do business.
1and1 has also been incredible about our clients growth. We've had 2 clients start with a beginner package and eventually scale to full root packages w/o any hiccups, their support team was in constant contact with us during the migration and (again) there was no downtime.
The web hosting industry is extremely competitive and allowing 25% downtime is completely unacceptable IMHO, whether it's 3.99/month or 20/month.
It shouldn't cost people 50-100/month to get 'decent' service.
For what it's worth...
Our CEO personally wrote an open letter to all customers and blog readers regarding the troubles with the (gs) Grid-Service, and all affected customers were credited for two months of service from the last major outage: http://weblog.mediatemple.net/weblog/2007/12/04/we-apologize/
We realize that saying "it's new, give us a break" is not an acceptable stance to have on a product that people are paying for. By no means do we think this relieves us of the duty of delivering a fantastic hosting product to our clients. Having been with this platform from its beta stages to initial public release and to today, I can say first-hand that not only has every issue been taken seriously, but considerable work has been put into constantly improving the platform and fixing major problems all around.
After our last major issue, some changes were made to the underlying storage subsystem that have improved performance and availability across the board. That said, if you or anyone is experiencing issues at this time we want to know about it. Feel free to contact me directly and we'll do anything we can to help.
Best,
Arman Zakaryan
(mt) Media Temple, Inc.
http://www.mediatemple.net/
Best,
Arman Zakaryan
(mt) Media Temple, Inc.
http://www.mediatemple.net/
Thanks
Thanks, really good to hear your findings on this.
2bits' module memory detail
Hiya,
2bits wrote up and provided a patch to see exactly how much memory each module takes. Has anyone here used that?
-Joe
Joe Murphy
Senior Developer
The Denver Post
not yet
Hadn't seen that. I'll add it to my todo list and report back if I try it...