So now we've evolved and moved forward in the development of the Special Olympics site, and some of you will remember my questions about how to choose a web hosting service. Although we chose Webfaction, we may need to move to a dedicated server and VPS, but in truth, I am VERY confused about how to KNOW what our site performance needs, and memory needs, truly are. PLEASE offer advice if you can.
We are have purchased 160mb of extra RAM allocation from webfaction. They say that the first 80, that comes with hosting, is equivalent to 256mb of VPS- I then don't know what it means when our total is actually 240mb- I guess that would be, hopefully, at least 512mb vps equivalent, maybe more. one of you might know better. We're also on the same server as our MySQL databases
Our issues are:
-
Page Execution times are general over 1000 and MUCH higher than that on admin pages- even with APC and Fast CGI and Drupal cache and no Admin menu, still over 1000. We do have a fair number of modules and I've tried shutting offer everything I can. A study of Drupal_load() shows that the total load time for modules seems to be just 32-35ms, so hardly a big part of the total page execution. And even when 800 MySQL queries run on one screen (View or permission or some other complex screen) the total time for that many queries (when that many occur) is about 370ms and that's usually as bad as query time gets. So there are MANY more ms of page execution not clearly explained by module loading or MySQL. Our theme doesn't seem to cause MUCH worse page execution time than Garland, though when we add more blocks to the sidebars, it gets a bit slow, but it's not a major difference I don't think.
-
Despite page execution, there's also simply a bit of a hover as you go to a new page. You know, it says waiting for server... for about a second or two rather than just popping over there- I'm wondering if it's a script load issue and I HAVE considered moving js script loading to the bottom of the page template rather than letting it stay in the head. But I don't know yet if that's the issue
-
I can't figure out a way to set APC so that it never fragments- the TTLs are 0 so shouldn't it fully purge? Instead, it seems to do a messy purge which increases the size of the php-cgi instance after a "cache full" and then apc shows up as fragmented. And no matter how much memory I give to APC, it eventually seems to use all of it. What I'd like to see is that the php-cgi recreates itself more often- if APC cannot purge cleanly, wouldn't it be better to re-initiate the php-cgi? The only problem is what will happen when the site gets a lot more visitors when it goes live.
What is the real impact of:
MaxRequestsPerProcess 500
DefaultInitEnv PHP_FCGI_MAX_REQUESTS 500-
I can't determine the best httpd.conf configuration of MPM Worker and i'm not clear on the most powerful settings in that file. I know that MaxClients is important but I'm not sure what it means so I can't figure out a good setting for it. I'm also confused by MaxRequests, both kinds- for PHP-FCGI and for Apache. And what does ServerLimit really mean? I keep finding web sites that simply repeat the Apache documentation but they never explain ir real world scenarios the actual impact of many settings.
-
We installed memcached 2.0.0b1 because it allows for socket (rather than port) based connections. Then we installed the Drupal Memcache API. When it's activate, some simple page views seem faster, but THEN- WE CAN'T SAVE VIEWS AND OFTEN CAN'T SAVE NODES. It just hangs and we get an Internal server error- so we have to shut down memcache. what could be causing that? Sadly the Unix/server error log doesn't show an entry for what happened.
-
Any opinions regarding igbinary serializer? We are using it with apc and we're assuming it's a good thing.
I've seen the link at:
http://www.morningtime.com/Drupal-6x-Performance-Guide/513
But it overwhelms me. I've tried a lot of those things, and most are only helpful during production or for static pages or anonymous users. We also need an overall greater speed on the admin side.
The FIRST things I want to learn to tweak are php.ini and httpd.conf.
**Httpd.conf fcgi and mpm worker setting areas look like this right now but we've tried a lot of different things:
### mod_fcgid PHP Configuration:
LoadModule fcgid_module modules/mod_fcgid.so
# MaxProcessCount and DefaultMaxClassProcessCount should be set to the same value.
# They set the number of burstable processes.
# General guide: Shared1=6, Shared2=9, Shared3=12, Shared4=15
MaxProcessCount 1
DefaultMaxClassProcessCount 1
# DefaultMinClassProcessCount is the number of persistent processes, unaffected by ProcessLifeTime.
# Set this to 0 if you want to keep your idle RAM usage very low:
DefaultMinClassProcessCount 1
# The life time of a burstable process. Keep this at 200 so you don't use too much RAM:
ProcessLifeTime 200
# Keep IdleScanInterval at 30. It sets the frequency that ProcessLifeTime is checked:
IdleScanInterval 30
# BusyTimeout is the number of seconds a single request can take.
# If you are serving large downloads from PHP, you may want a high value:
# 21600 = six hours. 300 = five minute.
BusyTimeout 21600
# IPC* are connection settings between the fcgid module and your fcgi app:
IPCCommTimeout 240
IPCConnectTimeout 30
# Set the maximum number of requests a process should handle before it is killed.
# MaxRequestsPerProcess must be lower than 'PHP_FCGI_MAX_REQUESTS':
MaxRequestsPerProcess 500
DefaultInitEnv PHP_FCGI_MAX_REQUESTS 500
# Apache Process Settings
StartServers 4
MinSpareThreads 10
MaxSpareThreads 25
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 1000
<VirtualHost 127.0.0.1:41414>
ServerName yourdomain.com
ServerAlias www.yourdomain.com
# Standard CGI:
AddHandler cgi-script .py .cgi
# Force CGI processing of .phpcgi files with the php-cgi binary:
ScriptAlias /php-exe-path/ /home/soncnv/webapps/phpstack/php_fcgi/bin/php-cgi
AddType application/x-httpd-php .phpcgi
Action application/x-httpd-php "/php-exe-path/php-cgi"
AddHandler fcgid-script .php .fcgi
DefaultInitEnv PHPRC /home/soncnv/webapps/phpstack/php_fcgi/lib/php.ini
FCGIWrapper /home/soncnv/webapps/phpstack/php_fcgi/bin/php-cgi .php*** Our php.ini looks like this but I've played around a lot with apc settings:
memory_limit="128M"
magic_quotes_gpc = 0
register_globals = 0
session.auto_start = 0
mbstring.http_output = pass
mbstring.http_input = pass
mbstring.encoding_translation = 0
memcache.hash_strategy = consistent
upload_max_filesize = 10M
post_max_size = 10M
extension="igbinary.so"
;session.serialize_handler="igbinary"
extension="memcached.so"
; ; ::: miko note- this was in here twice :::
;memcache.hash_strategy="consistent"
extension="apc.so"
apc.enabled="1"
apc.shm_segments="1"
;commenting this out allows you to set it in each fastcgi process
apc.shm_size="80M"
apc.num_files_hint="2000"
apc.user_entries_hint="0"
; ; ::: miko optimization :::
;apc.ttl="7200"
;apc.user_ttl="7200"
apc.ttl="0"
apc.user_ttl="0"
apc.gc_ttl="600"
apc.cache_by_default="1"
;apc.filters=""
apc.mmap_file_mask="/tmp/apcphp5.XXXXXX"
apc.slam_defense="0"
apc.file_update_protection="2"
apc.enable_cli="0"
apc.max_file_size="1M"
apc.stat="1"
apc.write_lock="1"
apc.report_autofilter="0"
apc.include_once_override="0"
apc.rfc1867="0"
apc.rfc1867_prefix="upload_"
apc.rfc1867_name="APC_UPLOAD_PROGRESS"
apc.rfc1867_freq="0"
apc.localcache="0"
apc.localcache.size="1000"
apc.coredump_unmap="0"
apc.serializer="igbinary"
; ; ::: eAccelerator Configuration :::
; ; Disabled by default because it increases RAM usage.
; ; If you want eAccelerator to have a positive effect on your site's
; ; performance, then you should set DefaultMinClassProcessCount to a
; ; value higher than 0 (probably 2 or higher) in your httpd.conf file.
; ; You can enable eAccelerator by removing the ';' characters from
; ; the beginning of the following lines:
;extension="eaccelerator.so"
;eaccelerator.shm_size="16"
;eaccelerator.cache_dir="/home/soncnv/webapps/phpstack/php_fcgi/eaccelerator"
;eaccelerator.enable="1"
;eaccelerator.optimizer="1"
;eaccelerator.check_mtime="1"
; eaccelerator.debug="0"
; eaccelerator.filter=""
; eaccelerator.shm_max="0"
; eaccelerator.shm_ttl="0"
; eaccelerator.shm_prune_period="0"
; eaccelerator.shm_only="0"
; eaccelerator.compress="1"
; eaccelerator.compress_level="9"
; ; ::: ionCube Configuration :::
zend_extension="/home/soncnv/webapps/phpstack/ioncube/ioncube_loader_lin_5.2.so"Any advice appreciated. right now only 4-5 people are working on the site daily, but we're probably giving it a fair number of hits. I'd let you see the site if I could.
Comments
Follow-up
We have had the APC memory set lower and we HAVE had PHP_FCGI_MAX_REQUESTS and MaxRequestsPerProcess set to 1000 but I wasn't clear whether those changes did much. Obviously with apc at 50M the cache fills faster and so fragmentation happens faster. When apc fragments, it is probably working less efficiently, but still seems to help.
MIKO, Creative Technical Support and Developer
Miko's Support and Design Services, Seattle
http://mikosweb.com
While I don't have the time
While I don't have the time to look into the configuration, I can give you some general hints:
1) Memcache should "just" work. If the faster, more optimized module gives errors, just go with the stable version for now, Also use PECL memcache.
2) Consider your testing strategy:
From your post you don't know what really makes your sites slow loading. You can easily fix this by installing YSlow (Firefox Firebug Plugin) and Pagespeed (Firefox + Chrome.) Also going to Safari or Chrome and enabling Developer Toolbar and you can easily see what time is spent on what. (Resources Tab or Network Tab)
The timeline shows you how much document loading is (wait for the server), how much transferring is, what items are cached, which give 304s, etc.
Tell us your pagespeed score both logged out and logged in. Drupal load times are one important thing, but not all for perceived site speed.
3) Make sure your caching works properly (good expiration times) and use a great admin theme like rubik. If on each admin page, it has to reload all the JS and CSS you also loose performance.
4) Add a reverse proxy like Varnish in front of Apache. While it won't help with logged in users, it will help to cache static assets like JS, CSS and images, which is already a lot.
5) Enable APC statistics to get a feel of how internal page creation times are. (devel module)
And then of course:
Optimize the code, queries and use XDebug Profiler to find the real functions, where the time is spent.
I hope that helps a little.
Best Wishes,
Fabian
Thanks and...
Thanks Fabian.
Currently if we use memcache, we need to use 2.0.0b1 because of the need to use a memcached.sock with no port. However, I do realize that on a dedicated server, we could safely or more safely use TCP/port connections.
I have been using YSlow and we get an A on everything except:
Since it's Drupal I don't know how to resolve 2 or 3. I'm working on the CDN but we've had trouble hooking into local nginx because Drupal simply didn't allow the webhost manner of doing it (it caused recursion), so I need to install the CDN module.
Regarding #3- whether caching is working properly- that's all I've been doing lately! LOL. Immediately on the first day I set the apc.ttl and user.ttl or 0 because we immediately got memory errors with and notable ttl. I read somewhere that with ttl set to zero, it causes a clumsy cache flush, which I seem to be seeing.
We've been considering Varnish AND BOOST but we haven't even gotten to the point or worrying about anonymous users yet.
I'll look into APC stats (I didn't see them on the devel page before) and XDebug Profiler.
Thanks again.
MIKO, Creative Technical Support and Developer
Miko's Support and Design Services, Seattle
http://mikosweb.com
First; Also give APC a little
First; Also give APC a little more RAM. Your working set ("hot pages") should well fit into the APC cache.
Okay, that is the point here for using a Unix Domain Socket. Well, you could possibly pay some c/c++ coder to just change the bind() in the stable memcached server to use a unix domain socket instead. Should take not more than 1 hour if hardcoded. And maybe another hour for hardcoding the same path in the PECL solution for connect(). Have patches for that ready and go with a stable version. Thats what I would do.
Either use a professional CDN (AWS for example) or setup just a cookieless subdomain, but then you don't need nginx, but Apache + Varnish is enough.
In my opinion you are overcomplicating matters a little instead of going with one proven and stable solution. (LAMP Stack + Pressflow + Memcache + Varnish for example.)
Here is another case-study + guide, which might to easier to follow:
http://oshyn.com/landingpages/drupal-performance-tuning
How is the system performing on for example an EC2 instance? How is it on your dev servers? Perhaps the CPU power you get from your shared hosting experience is just too less and everything is fine with your site.
Best Wishes,
Fabian
In my opinion you are
I completely agree!
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Thanks but I'm not as clueless as you think :)
The problem is that no large group of high performance users have listed ONE solution as the perfect one. In fact there are lots of people using APC and Memcache, and lots of others using Memcache and Boost. To get one answer about the exact right solution has not been easy.
Also regarding Pressflow I've been considerably worried that it will have more bugs than stndard Drupal since I don't know enough about the maintainers. I seem to remember lots of little comment like "that doesn't work with Pressflow."
I was led down the apc and boost direction. Then only in the past few days I'm getting all of this feedback about Varnish.
Thanks and I will try to take it under advisement.
I think that there's a bottleneck on our site much bigger than any of those, hidden somewhere, in some module.
MIKO, Creative Technical Support and Developer
Miko's Support and Design Services, Seattle
http://mikosweb.com
I am not saying you are
I am not saying you are clueless, but you make things much more complicated than they are and combine this with fear of what you heard.
Optimization is complicated and difficult yes. But using proven ways, testing that they work as expected, is much easier and then go from there.
It is right that there is no perfect solution.
But those tools are not mutually exclusive, but rather a stack of optimizations.
Basic rule for optimizations is: Start simple and as near to proven solutions than you can.
Every change you make, can also change things for the worse - especially if you don't know what you are doing.
Optimization is a step-by-step process and normally will start really simple or with "standards" or proven solutions, then measure how they hold up and then just optimize remaining edge-cases individually.
It is not helpful to try to opzimize APC to perform like a ninja - if you don't have Memcache, Pressflow, Varnish, etc. yet.
That said:
APC: You always want to use it. Most don't tweak it at all besides memory size and it works great out of the box. - Are you sure your APC is effective with fast_fcgi? I've heard just recently that it needs a special module to share the memory between the processes ... So if in doubt, go back to mod_php first and then slowly optimize from there ...
Memcache: You want to use it as DB locking can be quite heavy on a site.
Pressflow: If you run PHP5, you want to use it, as its faster and allows Reverse-Proxies like Varnish to function out of the box with standard configurations.
Varnish: If you run Apache, you want Varnish - alone for reliability and static assets.
Boost: You want to use it, if you don't use Pressflow or run into SESSION errors. (Note there is Boosted Varnish soon to be released that integrates boost into the above stack out of the box).
( Drupal Optimization: You want to possibly use javascript_aggregator to minify filesize even more if JS is the problem. You want to disable statistics module. etc. )
That is one proven stack. Of course others use nginx with great success, of course there are other solutions, but those are always heavily customized by someone who has the basic setup done and working.
But there is not just one direction: "Use APC + Boost, no use Varnish, no use memcache + boost", but a general stack of solutions, which each have advantages and disadvantages. And as I said, boost can also be combined with Varnish to achieve fantastic results, but you need to make a plan, outline it and go for it.
Not sure what stage you are on with the project, but why not test out Pressflow for a week on staging or development servers? If no bug comes up, push to live ...
To give you some more knowledge, there is an excellent presentation on caching and why it matters from DrupalCon Chicago here:
It will explain all the basics of Memcache, Varnish, APC, etc. and why it matters.
Best Wishes,
Fabian
Pressflow
You can be assured there will be no concerns there. Pressflow's maintainers are some of the top minds in the community. Considering David Strauss is one of the admins for this very group and the fact that Narayan Newton is trusted with root on drupal.org, I think you can safely rely on the quality of the code they and the rest of the PF team maintains. :-)
PF bugs are the likely least of your worries. I've switched live sites between PF and core Drupal multiple times with no issues whatsoever. I have not run into a single "that doesn't work with Pressflow" issue on any of the sites I run. Usually, it's more like a, "X module unnecessarily sets a session cookie on anonymous users and prevents caching" issue. With PF and a properly-tuned Varnish it's eminently possible to serve a site to anonymous users entirely from the Varnish cache - bypassing the bottleneck on your backend at least for anonymous users and buying you time to sort out the problem.
The advice here from Fabianx, dalin, and the others is right in line with the generally accepted best practices in the Drupal high performance community. Sure, we all do things a bit differently, but that has more to do with our various environments and needs than anything. There is no "One Right Answer (tm)". If you came here looking for a magic silver bullet that will immediately solve all your problems, I'm afraid you may be a bit disappointed. :-) I think the Pantheon stack is about as close as you'll get, it's very good (I've modeled my systems on a lot of what it does) but it's certainly not universally accepted as some penultimate, ideal solution.
My stack looks a lot like what some of the others here describe:
Apache 2.2.16 (using mod_php)
PHP 5.2.14
MySQL 5.0.90
APC 3.1.6
memcached 2.1.3
Pressflow 6.20 (updating to 6.22 as we speak)
Varnish 2.0.6 (soon to be updated to 2.1.x)
nginx 0.7.65 (for SSL proxy/routing)
I run around a hundred Drupal sites on this stack (along with several hundred Moodle sites) using pretty modest hardware. It just plain works. It's capable of serving much, if not all, of the sites completely from the Varnish cache and has been hard to benchmark because I haven't found a testing system that can throw enough parallel clients at it to actually saturate the Varnish frontend.
Can what I need be done with other tools? Absolutely - you could very likely accomplish the same results swapping nginx in for Apache and use Boost instead of Varnish. I prefer the Pantheon-style Apache/Varnish stack, others here prefer using other tools. You use what works for you in your particular environment and what you're comfortable with, which will likely not be identical to what anyone else uses.
-Greg
Totally agree
Totally agree. To the point that I track the activity of the Pressflow maintainers both on Twitter and on drupal.org to see what they are up to. Whatever problems they are solving and contributing back to the community are generally things I'll run into myself at some point, and they have better solutions than I could come up with. Very smart group of people, and they care about both scalability as well as stability.
Pressflow, in my opinion, is extremely solid and stable. It's a no brainer if you are using a D6 site, you can drop it in with no concerns about compatibility or bugs.
I've actually been confused
I've actually been confused just by which fast cgi to use!
MIKO, Creative Technical Support and Developer
Miko's Support and Design Services, Seattle
http://mikosweb.com
I recommend
php-fpm. You'll need a fairly recent PHP version, 5.3, to get it prepackaged in a distro. To get it on 5.2.17 you'll need to patch PHP and compile it.
I advise against this last route since PHP has so many dependencies that you're bound to overlook some issue (already fixed in the distro) that will turn your php-fpm life not very pleasant.
Summarizing: on PHP 5.3 use php-fpm, ou PHP 5.2 use php-cgi.
Why?
I'm not clear why you're worrying about fastcgi to begin with - mod_php with APC configured with enough RAM works just great out of the box. Unless you're using nginx instead of Apache, I don't see a lot of advantages to it. In fact, I'd say there are more likely to be issues with fastcgi with APC, anyway.
-Greg
What about
for a start, separating the application from the server? Don't you see it as benefit? At least in a security sense?
That's the least of my (and,
That's the least of my (and, it sounds like, the original poster's) concerns. I certainly don't lose any sleep over using mod_php. As I said above: Apache+mod_php+APC just plain works. As far as I'm concerned, the separation inherent in fastcgi/php-fpm/whatever introduces annoying barriers (like the fact that each PHP thread needs its own separate opcode cache) that far outweigh any theoretical security benefits.
-Greg
Typical php-fpm
Typical php-fpm configurations use a shared memory opcode cache. Just wanted to clarify.
Thanks for the clarification.
Thanks for the clarification. I stand corrected. Since I'm still on PHP 5.2.x and I'm not having any problems with mod_php, I haven't looked closely enough at the various fastcgi options recently to notice when sharing the opcode cache became possible.
Regardless, I think the fastcgi/mod_php discussion is tangential to the original poster's issue - either will work reasonably well. I think he has bigger fish to fry. :-)
All of our hosting clients
All of our hosting clients are on VPSes so that's a non-issue for us.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Well
not quite. In fact you're isolated from the bare metal and each VM is isolated from the remaining running on that host. But it remains that each of those VM instances is hackable. And ultimately having the server separate from the app is a plus, IMHO.
OMG Rubik/Admin Tools was a disaster!
Well, I tried the Rubik + Tao + Admin 2.x Tools and as far as I could tell- it altered permissions in my system, OR I got hacked right after I installed it. Also, menus changed so completely that I could no longer work in CiviCRM. Apparently it removes too much from menus, and it needed to not remove anything from the CiviCRM menus.
It DID seem a little faster, but I had to uninstall it because my work got disrupted.
MIKO, Creative Technical Support and Developer
Miko's Support and Design Services, Seattle
http://mikosweb.com
To make this short: Rubik +
To make this short:
Rubik + Tao do not change any menus, but are not a suitable CiviCRM theme either, if you need sidebar blocks. Look into civik module for that.
Admin did work fine with CiviCRM, but I prefer admin_menu (stable) + civik or go with plain garland for civicrm admin theme.
Use themekey to put up different themes for different paths. Like civicrm/ -> garland.
Thanks again- themekey?
Is themekey better than Sections? Faster I mean?
We are using Sections but it hadn't occurred to me to use it with civicrm and I'd not heard of Themekey.
MIKO, Creative Technical Support and Developer
Miko's Support and Design Services, Seattle
http://mikosweb.com
Consider varnish + and tweeking mysql
sorry for my bad english. But...
I can recomend using the Varnish for a overlay on Apache. It speeds up things.
From experience can drupal be really heavy on the db so do alot of tweeking on the connection.
Also consider using css-sprites to reduce the http requests.
MARTIN SKÖLD
Web developer
073 - 50 66 521
www.twitter.com/martinskld
www.facebook.com/martin.skold
se.linkedin.com/in/martinskold
WILSON CREATIVE
Bredbandet 1
392 30 Kalmar
http://www.wilsoncreative.se
0480 - 36 37 70
If you have long page
If you have long page execution times that can't be explained by query time via devel, it's time to get familiar with profiling using either xdebug or xhprof - you're best of setting up a local development environment and profiling on that. Those will show you exactly where the time is spent, and can help with identifying memory issues as well.
If your issue is with individual pages loading slowly, and the time is not in the database, then whether you use mod_php or php fast_cgi doesn't really matter (although whether you have APC set up properly does) - PHP time is in CPU and calls to other things (database, network requests sometimes), and apart from an opcode cache which helps a lot, you need to look at the actual application performance to see what's going on. Badly behaved custom code or contrib modules can completely hammer response times.
Thanks Everyone
Thanks everyone for your advice. We test moved the whole site and I got some performance experts testing out scenarios, and so far, even on a dedicated test server with tons of CPU and memory- there's still a mystery page execution lag. It is indeed a mystery but these performance people will do some systematic tests and if they find the core issue, I'll report back here just in case anyone is interested.
MIKO, Creative Technical Support and Developer
Miko's Support and Design Services, Seattle
http://mikosweb.com
I would love to hear your
I would love to hear your results, high CPU usage is definitely a problem on many of my sites.
Marios
μεταχειρισμενα αυτοκινητα
strange interactions can lead to slowdowns
You may have already seen this, but if you haven't you might find it interesting:
http://groups.drupal.org/node/135989 (see the PDF)
In particular:
• Do not use Memcache with CacheRouter - the module performs very poorly with that
caching backend.
• Avoid using APC turned on by default (global) in conjuction with APC enabled on Drupal
(CacheRouter, APC).
From what I can tell from the numbers presented, it appears some of these combinations may be up to 2-3 times SLOWER than default Drupal caching with no additional help. So definitely try disabling all you additional caching, then re-enabling them one at a time and measuring the performance gain for each. Then try combinations of two, etc.
(I've seen similar conclusions elsewhere, including similarly odd "counterproductive" results for certain combinations of WordPress caching plug-ins.)