Will drupal handle my 600.000 nodes and 100.000 users ?

Willy:

You are right to be concerned. Until the end of 2008 I managed a Drupal site with 200 modules, 2M nodes, 350K unique visitors each month, 2M page views a month, and a 10%/90% authenticated vs. anonymous site visitor mix. We had to jump through all sorts of hoops - caching, load balanced web servers, master-slave-slave-slave DB servers, and a CDN - to get the site to offer an acceptable level of performance. I am not sure what you consider to be "low cost", but we were running two load balancers, four low end servers (web and image), three mid range servers (DB slaves), and one high end server (DB master) and I had a very smart and motivated (and not inexpensive) Director of Operations that worked 24x7 keeping it all running.

When we got the rights to more data and realized that the 2M nodes were going to jump to 25M, we gave up on Drupal and moved to a custom Grails based solution. Don't get me wrong - I like Drupal and think it is the right solution for many problems, but it is not the right solution for all problems.

What functionality is your site going to have e.g. what modules do you expect to use? What is the mix of authenticated vs. anonymous site visitors going to be? Do you (or someone on your staff) have a deep understanding of caching? Who are your site visitors, and will you lose traffic and revenue if site performance is lacking? Or is your site compelling enough that site visitors will tolerate poor performance and be forgiving of the occaisonal speed bump? There are some very talented Drupal performance consultants available, but they are not cheap - can you afford to purchase help when it is needed? Are you comfortable outsourcing your site search to Acquia? Do you have PHP development staff that can recode poorly performing modules without waiting for the community to fix any performance problems?

Andy

I think that frustrates a lot of people, myself included, that an ASP.NET is able to handle 1.2 billion of pageviews per month, with most of them authenticated users (dating site) with only a handful of servers (7-8 servers) -- still with rooms to growth.

That's about 500-600 pageviews per second. Last time I heard it is close to 2 billion pages per month now.

http://highscalability.com/plentyoffish-architecture

With Drupal, I haven't been amazed by the figure. Even a site with only 25 million pageviews per month needs a few servers. PoF, on the other hand, aims to server 1 billion pageviews per month per server.

http://plentyoffish.wordpress.com/2006/12/07/scaling-sql-server-2005-nat...

That practically puts Digg -- which is LAMP -- into shame.

Markus claimed to have solved the scalability issue, but no one really knows how he did it.

I know, it's entirely different. But Kieran, while Drupal is just one of the stacks, its efficiency determines speed. I've seen a Drupal page with close to a thousand queries per page. Obviously with memcached and other caching methods, it still helps to optimize and reduce the queries.

Assuming everything else is the same, the other stacks are known to perform very well. But the question is, how is Drupal compared to others? I think that's what most people try to figure out first before picking it as a content framework of choice for their project.

Will choosing Drupal cause scalability issues in the future, or will developing an app with lower level framework save a lot of cost later? After all, we all in the business for the long haul.

I personally use Drupal in various different projects. But currently as I have ideas for many bigger projects, I am in the position to reconsider if Drupal is the right tool for the job.

Waw!

I'm a bit scarred when reading your posts guys. Yes, the first comment gave me a "quantum of hope". And so many posts I read (I still can't find them again).

Let's start with Andy. I must admit this is my real first project with caching system. Few days after we launched the version 2.0 of our website, we had load issues with our server, and I could fix it with cache_lite, which is a good tool. But not as "complex" as those things you explained. The maximum I will install is 100 modules. And I think it's enormous for us. Now, we hardly have 500 members authenticated at the same time. Then, we have much more anonymous than authenticated. But all content are moderated or posted by moderators.

Modules Because of the mix of content types, I will use cck and its relatives. To allow real versatility (we're a site of classifieds ads but we have other contents which generate a good traffic), I want to use views and panels. For url friendly, token, pathauto and global redirect. Those are the main modules i want, plus some of the defaults installed.

Caching > No, nobody have a deep understanding of caching. But as I said, I read some articles which helped to understant the necessity of a good architecture (7 stages of scaling web applications). But I think the guys managing our server can handle it.

Traffic, Revenue > Of course, if we lose traffic, it will be dramatic for us. So are revenues.

Kieran, I went to your link, but couldn't understand anything. You speak too fast for me lol. I will have a class of english soon. Then my ears should be trained by the end of october lol.

Let me give a summary of where we are now. One dedicated server which handles everything: mysql, http, files...everything. Then, although we are on a dedicated server, it look like we are on a shared hosting since we also have 3 of 4 other sites running on the same server.

Going on drupal, I intended to have a reverse proxy (like squid), apc and memcache installed. I also planned to ask for another server for mysql only, if necessary. That's all I planned about optimization. But i'm afraid it's not enough.

Andy, I think we can afford more servers . It don't depend on me, but on my boss. I forgot to add that, even if I'm not an expert on drupal, I know php a bit, and I can read drupal documentation lol.

Now, I have two sites running under drupal (http://www.bamena.com/ and http://www.journaux.ma/). The first one runs with drupal6 and the second with drupal5. And i'm satisfied with them both. Though bamena.com causes me some worries.

Right. This is where we are.

Thank you for your posts.

The most you can squeeze is when you are building custom scripts with custom database structure. You cannot compare Drupal to custom script. For example if I am using Drupal for personal blog, with most of the features disabled, I can write custom php scripts which will do the same but will get >10x performance. I see Drupal as a tool to get you idea running - then you will have the funds for very very custom solution. PlentyOfFish is the example of later - It is a hard work of building system from scratch.

I had a social site with ~1,000,000 million pageviews per day, which was designed and coded by very gifted teenager, who at that time was good at social engineering but very poor php programer. Site grown up pretty quickly and we moved from shared to dedicated server with 2gigs of ram. It was overloaded in minutes... I started to look at:

1. Mysql data structure and queries - most of the queries were nonsensical and not needed at all, indexes were built not the best way. After couple of days of tweaking, loads were reduced by 10-20 times.
2. Experimented with MySQL server settings as much as possible, gave another 10x boost. You will be surprised, as there are more options than query_cache_size that do impact performance...
3. Added jpcache support for php scripts - another 5x, as most scripts supported that. eAccelerator gave another boost.
4. Tweaked apache settings, mostly caching for static content and removing unneeded modules.
5. All the other tweaks gave another 2-5x boost...

After all that the same server was running on 0.2-0.4 load, without any problems for months. Yes, it was ~1,000,000 pageviews/day site, with most of the registered users ( I do not remember exact count)!!!

It was not Drupal experience but the principles are the same:

1. Do not use what you do not need - reduce features by 5%, and you may get 5x performance... Reduce modules count as much as possible.
2. Remember, that Drupal is an universal tool for everyone, customize it! Even if you need to hack core, mysql data structure etc. You are risking to get compatibility and update nightmares, but when we are talking about 10.000.000 pageviews/day site - it is worth it. Why? Because extra X000$/month server bill is what matters. Hire skilled programmer. Hire LAMP admin.
3. Reduce amount of dynamicly generated data: caching, caching, caching. Use everything you can get working. I would start from hardware level, because it is the least chance you will screw something and go all the way to app level. Unfortunately, at this moment I do not see a universal Drupal cahe module "to rule them all".
4. Already done "1"? Do that again - simplicity is what rocks. Remember, you are selling the main idea of that site, so dedicate most effort to it. Then, you will see what to do next. Overloading with features kill hardware AND visitors. I can bet, that your web app is not Microsoft Office of web sites.

Its all true what you say, just was that not the point of the posts above.
Its clear that you get drupals power for a prize as you get the asp speed for another.

But it would be interesting how things would be IF we then see a comparable non php CMS.

ATM no other base system offers a out of the box free CMS complete like drupal, joomla or typo3.
They are good 3-5 years behind. But there are already some good commercial non php CMS out.

I am sure we will see in the near future other full featured CMS beyond php and i am also sure
that will be good for the php world too. Competition is always good.

The tweaking is another problem - custom will not save you from bottlenecks of the core.
For example that drupal turns off the block cache when you have installed custom node
access control (even when your blocks don't use any node data), the uncaching of
user profiles and file pathes which scale badly... A small patch there works wonder.

We have ONE script, which is invoked to create a site out of a repository of modules,
core files and medias. It will do all the copy and apply the patches. The goal is really
to have a kind of "robot script". To update a site, i just copy the new files in the file/module
base directory, then i invoke back, create and install - 3 scripts.

I only have to check that manually when a patched module or core file is changed,
but in 9 of 10 cases you see on the first glance an adjustment is needed or not.
It works like a charm, i must say.

Any install step which can't be automized is changed to a way which fits to the 3 script step.
In that way i avoid the human error factor and i have no work beside one time a bit more to
create the scripts lines and the patches.

Hendry, I think you got me wrong. The statement was meant for users.

I meant that the users out there do take such clues. And they are not to be blamed too. Technology is complex as it is. It is difficult for people to understand these many terms. What I meant was a general statement that people should not imagine what works for PoF will work similarly for them. "But one thing you should never do, is take example of PoF in hopes of doing same.": that is an advice for users.

brainless,
Sumit Datta,
brainlessphp.blogspot.com
Web Developer