I recently (yesterday) got my hands dirty with a huge news-site in Greece that suffers from really bad performance!
I checked the filesystem and saw that the previous admins placed all files under the "sites/default/files" folder, without creating subdirectories. Now there are more than 20000 files under there, increasing by 100 per day.
Could this be the source of all evil? The owners of the site say that the problems are getting worse every month so I imagined that by re-arranging files in the file-system I would gain something in performance.
I thought of two solutions, one of them not being so professional...
a) use something like filefield_paths and try to fix the file-system
b) create a new content type, design it very carefully and let the old nodes perish
Should I try the first option? Do you believe it is worth it?
As for the second option, apart from making me shy even by saying it, it solves many other problems since it would allow me to redesign the Views and/or Panels. I am really thinking about it seriously, but for the near future, once I get most of the other errors fixed.
The site is live for more than two years.
3 months ago Varnish started serving the anonymous users, but many problems still remain.
There are occasional 500-503 errors and sometimes it doesn't even fetch text and/or images. It actually seems like the SQL returns nothing! I'm now searching for a solution to these too...
Comments
Hi, Monitor your website will
Hi,
Monitor your website will be the best way to find where your website sucks (php code, sql query...). Very difficult to say like without any data.
Julien Didelet
Founder
Weblaa.com
Julien thnx for the reply! Of
Julien thnx for the reply!
Of course you are a 100% correct, but monitoring it might be difficult, the previous admins stopped the "statistics" module in order to gain something from perfomance. Furthermore there are php-snippets all over the place (2-3 per block, 30 blocks in the front page)
I am just trying to find where I should start from. Probably I should enable statistics!
Bill Seremetis
http://srm.gr - working with Drupal in Greece
The statistics module is a
The statistics module is a performance killer and you might want to avoid it if possible. You can use Google Analytics or similar to monitor traffic.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
I understand! Please allow me
I understand!
Please allow me to correct myself: statistics and dblog are enabled. For some reason that I don't know yet nothing is being recorded in admin/reports/dblog
I don't understand why, but it would help if drupal was speaking to me!
Bill Seremetis
http://srm.gr - working with Drupal in Greece
Unlikely
While this could be a problem, it's quite unlikely that this is the main or largest source of performance problems. You should isolate what is causing the bottleneck at a kernel level (are you blocked on CPU, RAM or disk I/O?) and/or isolate the problem in PHP or the Database.
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com
I see!
I understand what you say.
Do you have any particular tools/workflow to suggest? Any ideas are more than welcomed!
Bill Seremetis
http://srm.gr - working with Drupal in Greece
tools
you could try new relic.
Is monit installed on the server? it will give you an idea just by looking at the graph.
And i reckon also the mysql conf could be a bottleneck.
I'm looking at new relic now.
I'm looking at new relic now. I don't know if monit is installed. I'll ask
you are referring to this one: http://mmonit.com/monit/
right?
Bill Seremetis
http://srm.gr - working with Drupal in Greece
not too sure
no i was thinking about the combination of munin and monit.
What is the server linux flavor?
for new relic, which is better and easier, look here: http://buytaert.net/playing-with-new-relic-on-acquia-hosting
I'm looking at it right
I'm looking at it right now.
Do any of you have any experience with droptor?
Bill Seremetis
http://srm.gr - working with Drupal in Greece
lookin into it
Nope but it seems interresting to monitor multiple site in one place. Can be useful, thx for the tip mate.
i'm looking at http://drupal.org/project/boost now and testing it see if can gain in performance
The site in question is now
The site in question is now using Varnish, is there any reason to combine Boost with Varnish?
Bill Seremetis
http://srm.gr - working with Drupal in Greece
Not really. I would stick
Not really. I would stick with Varnish as that will give better performance since it bypasses Apache all together on anonymous views.
Of course that could be a problem. Maybe Varnish isn't actually serving the pages? You can check by monitoring the response header from the webserver and looking for the X-Cache header. It should be set to HIT. Make sure you aren't logged in when doing this.
The big thing is going to be pinpointing where the bottleneck is happening. When it starts slowing down, run TOP and see what's taking up the most resources. Most of the time when I get into a problem like this, I find it's MySQL causing the problem (non-optimized configuration, missing indexes, performance killing queries, etc.).
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
PID USER PR NI VIRT
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3477 mysql 20 0 22.5g 15g 4232 S 42.5 67.2 37102:41 mysqld
The numbers for mysql are high as you can see!
The server is using php v5.3.5 running as a CGI script. Lighttpd is used for the http-server.
I didn't quite understand what you say about X-Cache. I install httpfox module on firefox but can't really find the X-Cache. I'll try with Dragonfly in Opera in a minute.
Bill Seremetis
http://srm.gr - working with Drupal in Greece
When a page is actually
When a page is actually served from Varnish's cache, it will add additional response headers like X-Cache or X-Varnish.The easiest way is to check with CURL via the command line:
curl -I http://{yourdomain.com}
You should get an output like this:
X-Varnish should always be there. X-Cache and X-Cache-Hits depends upon the configuration. On X-Varnish, if you have 2 sets of numbers like above, then it's serving from Varnish's cache. You might want to run it twice incase you hit a file that has been purged from the cache already.
You can also check it in Firefox with Firebug, by using the net tab and checking the headers, but the command line route is much quicker and easier.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
HTTP/1.1 200 OK X-Powered-By:
If I understood you right Varnish seems fine
Bill Seremetis
http://srm.gr - working with Drupal in Greece
Yeah Varnish is serving the
Yeah Varnish is serving the pages. It has to be somewhere else in the stack, which would require some investigating. I also noticed you said it was a VPS. Have you talked with the hosting company? There's always a chance it's a problem lying in the host server. I've experienced that before - tearing my hair out thinking there's a problem in the VPS, only to find out it's the actual host machine or another VPS running on the machine causing all the headaches.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
Dedicated server
Just talked to the hosting company, the server is a full managed dedicated server (16GB RAM and 12 cores x 2.8Ghz).
Not bad!
They have done some monitoring on their side, before I got involved with the site, and they located the source of the problem to be the sql queries, especially on the front page
Bill Seremetis
http://srm.gr - working with Drupal in Greece
Aha! Have they sent you the
Aha!
Have they sent you the slow query logs? Has the database server been optimized? Is it on localhost or a separate box? Does it have enough RAM for its query cache? Are you running multiple DBs (master-slave)?
all I can tell you for sure
all I can tell you for sure is the person behind the hosting company follows this discussion!
I could ask for the logs (and will), but the client doesn't want to change many things on his site. Meaning that I haven't really found where to start (eg. I followed your advice to stop the statistics module but it is currently under discussion).
This is currently a setback for me.
Bill Seremetis
http://srm.gr - working with Drupal in Greece
100% agree with joshk.
100% agree with joshk. Isolate and check your php code first and all your sql query (and with some contribute modules). After, if everything is ok, check what's happen at a kernel level. And if everything is ok, check what kind of solution you can implement to improve your performance. But first, check what's happen with your code ! :). You can install varnish, boost, APC and memcache if you want but if your code is bad, it's useless...
Julien Didelet
Founder
Weblaa.com
I can't say that you made my
I can't say that you made my day! The code is bad and I know it!
I'm starting the debugging process, it will take a few weeks/months!
In the meanwhile I'm waiting for the host to check new-relic and give me a reply
Bill Seremetis
http://srm.gr - working with Drupal in Greece
wow!
wow! many helpfull replies and many ideas! I wasn't expecting so much help, I'm amazed
keep it coming
thank you all, I'll make sure to track down my work so as to write a case study in the future ;)
Bill Seremetis
http://srm.gr - working with Drupal in Greece
boost + nginx
For anonymous users, look to put nginx in front of the apache / drupal server, then offload images, js, css and any files created by Boost.
All the above posts are good, optimize the code and queries. Nginx will buy you head room to debug the rest and put you in a good place to continue to grow the site traffic.
you suggest to drop varnish
you suggest to drop varnish in favor of nginx?
the server is a full-managed VPS and I'm not sure they are willing to trysuch changes, I'll forward your ideas along with the all the posts untill now
Bill Seremetis
http://srm.gr - working with Drupal in Greece
I actually suggest you to
I actually suggest you to drop apache altogether. Nginx can serve Drupal just as well, but certainly faster, and with boost it is as powerful and quick as (maybe more) apache + varnish.
fidelix I'll keep your
fidelix I'll keep your suggestion in mind but I must say that at the moment I don't feel easy with doing such huge changes to the site. The reason is simple: now it works, with many problems but it works. I would like to get it to be more stable (reduce the number of queries etc) and then I'll start examining changes to the server after discussing it with the hosting company.
After all they've done a great job up to now and it is a credit to them that the site is working! It was their idea to install lighttpd and varnish instead of apache.
ps: you intrique me to try nginx on my pc, instead of apache
Bill Seremetis
http://srm.gr - working with Drupal in Greece
I am running 4 very big sites
I am running 4 very big sites with nginx, and 17 medium/small projects. When you are able to study it, I'm sure you'll like it.
I'll sure test it, it sounds
I'll sure test it, it sounds intriguing
Bill Seremetis
http://srm.gr - working with Drupal in Greece
Varnish is generally easier
Varnish is generally easier to configure than nginx. Nginx won't be faster than varnish, so it's not worth the switch.
On the performance page you should consider upping the max age of cached pages. Your curl response (cache-control: max-age = 600) shows a max age of 5 minutes, meaning that varnish only keeps pages in cache that long. I assume you're running pressflow. What you should experiment with is setting the min age to 5-10 minutes and the max age to 6 hours or so.
I should set these settings
I should set these settings from Pressflow (yeap, the site is using pressflow) and Varnish will get updated? or do I need to change some settings in Varnish too?
Should I use Block-cache (for Blocks created by Views) also? Will it benefit the overall perfomance in a site that uses varnish?
Bill Seremetis
http://srm.gr - working with Drupal in Greece
Varnish will respect the
Varnish will respect the cache control headers it receives from the backend. You can change them if you want in the Varnish VCL - we often do - but it's not mandatory.
The Views block cache is a different caching mechanism than Varnish. They won't really affect each other (positive or negative).
Well
if you consider the complete ugliness that is VCL easier than a logically clean declarative language then I think we have very different understanding on what the word easy means.
Also you're lefting out the resource usage of Nginx vs. Varnish. Which one is lighter?
Both have merit
We run both Nginx and Apache+Varnish in our infrastructure (more than 100 sites). Depending on the situation, either may prove a better fit. Both have their benefits and drawbacks. I don't think anyone can categorically say one is better than the other in all cases.
Filesystem
One note on the filesystem - different filesystems handle lots of files differently :) Assuming you are on Linux, you may want to check if you are using an EXTx (ext2/3/4) filesystem, XFS, or something else.
There's good info here...
http://stackoverflow.com/questions/466521/how-many-files-in-a-directory-...
http://serverfault.com/questions/43133/filesystem-large-number-of-files-...
http://stackoverflow.com/questions/5019371/storing-accessing-up-to-10-mi...
Nice info on those links! I
Nice info on those links! I now want to try XFS on my pc to see if it hanldes things differently!
After reading the above I got the idea that although 20k files (or 60k files that a subfolder has) isn't that bad for perfomance it is definately not the best idea.
I won't be dealing with it for now as far as the site is concerned. I will dwelve into sql-queries and see what I can do there. It is definately on my to-do list to optimize the filesystem soon.
However after 7 days with the site and having read all the info you guys told me I'm leaning towards migrating the site to drupal (from drupal!). I'm thinking of:
a) creating a clean drupal installation
b) add the node-types from scratch, this time design them as to avoid sql JOINS (use many cck fields, and not share them between content types)
c) create the filesystem the way I want to (filefield_paths)
d) use community-modules instead of custom modules (eg: simplenews instead of custom module for newsletter etc)
e) finally migrate data (migrate module)
This solves many problems at once and has the extra benefit that the resulting site would be something that I could maintain.
First of all I get rid of custom modules that have no documendation of what they do.
I have most sql-queries re-designed and avoid many JOINS.
I fix the filesystem.
Furthermore I won't have to work on the bad design that previous admins had created.
I think of it as a nice idea, hope the client agrees.
Bill Seremetis
http://srm.gr - working with Drupal in Greece
Created this module to cache filesystem for imagecache images
This helped to speed up our page loads by a lot. Works best when using memcache. Results depends on how slow the filesystem is.
http://drupal.org/project/imageinfo_cache