Hi everyone, I'm looking for some advice optimizing a site for authenticated users.
VPS with 2GB RAM
Pressflow 6.19
80 modules (including OG, Views, CCK & GMap)
3,500 nodes
Averaging 115,000 page views per month
MySQL using InnoDB tables
Memcache (84% hit rate)
APC (set to 64 M)
MySQL setup is:
key_buffer = 32M
sort_buffer_size = 16M
read_buffer_size = 16M
read_rnd_buffer_size = 16M
myisam_sort_buffer_size = 16M
query_cache_size = 32M
innodb_buffer_pool_size = 1G
innodb_additional_mem_pool_size = 8M
innodb_log_file_size = 256M
innodb_log_files_in_group = 3
innodb_log_buffer_size = 32M
innodb_flush_log_at_trx_commit = 1Using Devel I'm seeing pretty inconsistent differences in page execution time. After the initial database hit, I'll get:
Page execution time was 632.28 ms. Executed 52 queries in 41.34 milliseconds.
Memory usage:
Memory used at: devel_init()=1 MB, devel_shutdown()=23.22 MB.After refreshing the same page a couple of times, I'll get:
Page execution time was 2158.98 ms. Executed 52 queries in 194.32 milliseconds.
Memory usage:
Memory used at: devel_init()=1 MB, devel_shutdown()=23.22 MB.The following is typical of top output at the moment:
top - 14:25:04 up 19 days, 16:12, 1 user, load average: 0.23, 0.27, 0.17
Tasks: 103 total, 1 running, 102 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.0%us, 1.5%sy, 0.0%ni, 83.6%id, 0.1%wa, 0.0%hi, 0.1%si, 8.7%st
Mem: 2068848k total, 1725540k used, 343308k free, 105992k buffers
Swap: 262136k total, 3780k used, 258356k free, 1146220k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10671 www-data 20 0 63668 30m 23m S 50 1.5 0:02.56 apache2
10346 mysql 20 0 1301m 200m 5592 S 4 9.9 0:15.21 mysqld
10675 root 20 0 2260 1132 880 R 1 0.1 0:00.09 top
30023 nobody 20 0 62332 58m 672 S 1 2.9 19:46.61 memcached
1 root 20 0 1984 576 548 S 0 0.0 0:22.98 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd MySQL slow query log isn't turning up much either. I added a few additional indexes to cck fields being used on 'WHERE' clauses, but I'm starting to wonder if this is a hardware issue as traffic is pretty low.
I'm not a performance guru, but this has landed in my lap now and I feel I've tried many of the obvious interventions with little success.
Comments
It looks like it's not a
It looks like it's not a MySQL issue:
2158.98 ms. Executed 52 queries in 194.32 milliseconds.
while 194ms is high for 52 queries, that leaves 2seconds of PHP time for one page which is over the top. Are there any external connections required to build the page (connections by the server to Digg, Facebook, anything else)?
A hardware issue is less likely as top shows a pretty idle server.
You might try to setup a PHP debugger so that you can profile PHP to find the bottleneck. Take a look into XDebug or XHProf. You can analyze the profiling results with webcachegrind or kcachegrind.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
XDebug
So I tried XDebug and used Webgrind to take a look at the results. The total inclusive cost for the top functions looks high, but I'm not sure how I would go about reducing those times.
The results are viewable here.
The site is calling Facebook using XFBML, and the Disqus module is being used for comments.
There's no profile files to
There's no profile files to grind there.
Learning how to work with a grind does take some practice. Basically you need to use the charts as a guide to try and find inefficient code. Perhaps there's a bug that needs fixing, perhaps a static variable is needed, or some caching. Perhaps a contributed module is doing something that simply will not perform well in your situation.
Note that by default webgrind is always analyzing the most recent page load. One thing to be careful of is to not accidentally review a grind for an imagecache process, or an image 404, but for the actual page in question.
If the server makes calls to Facebook or Disqus to generate the page (not to be confused with the client making JavaScript calls to those services), then they may be the source of your problem. The quick way to check would be to disable the module in question and see if the problem goes away. But with webgrind you will see the offending functions taking a lot of time.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
vzfs?
Is your VPS running on OpenVZ or Virtuozzo? If so, check if disk access is your bottleneck. The vzfs file system can be slow, and even with APC, PHP will access each script, and on a slow file system these add up.
If you type the df or mount command, it will show the file system type.
Try the site on your local computer and see if you are facing the same issue.
The 8.7% stolen time seems to point to that direction, or a busy physical server.
Also check if you have APC fragmentation, or not enough memory allocated, although I doubt that with 64MB and "just" 80 modules.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
APC
Both the dev and production sites are hosted with Linode. This is the df output.
Filesystem 1K-blocks Used Available Use% Mounted on/dev/xvda 65797900 7958580 55165452 13% /
tmpfs 1034424 0 1034424 0% /lib/init/rw
tmpfs 1034424 4 1034420 1% /dev/shm
APC show 0% fragmentation most of the time, but will occasionally jump to 100% whilst refreshing the results.
APC stats are here.
You have apc.shm_size=64M but
You have apc.shm_size=64M but you're only getting 30MB because you need to increase the system's shared memory maximum.
For temporary, on the fly:
sysctl -w kernel.shmmax=67108864And for persistent after reboot, add the following to /etc/sysctl.conf:
kernel.shmmax = 67108864I did this, but it seems that
I did this, but it seems that since APC was installed via apt-get then it wasn't using kernel shared memory so assigning more didn't make a difference. I altered the APC settings in php.ini and now I have 64M of mmap.
Would I be better to compile php from source to use kernel shared memory? I'm assuming this would probably be faster.
http://drupal.org/node/415348
My mistake. It does indeed
My mistake. It does indeed look like you're using mmap:
apc.mmap_file_mask /tmp/apc.GFiY9bI'm not sure that guy is correct about needing to recompile to do kernel shm. I looked and he's right that all the recent ubuntu and debian packages are compiled with --enable-apc-mmap but they don't seem to be using it by default. Check php.ini and /etc/php5/conf.d/apc.ini to see whether the file map is something that's been set by your host. (or grep -R "mmap" /etc/)
Anyways, APC is a pecl package, so it's compiled separately from PHP.
Authcache?
Since you have authenticated users, have you considered using the Authcache module?
Authenticated User Page Caching (Authcache)
http://drupal.org/project/authcache
Hello, I have been working on
Hello, I have been working on my VPS for about a week now, and I am a total noob (which is part of the problem) but if I could get page times posted like the ones above I would be stoked. I am using apc and have it set at 128mb.
My Page times look more like this:
Executed 199 queries in 632.97 milliseconds. Queries taking longer than 5 ms and queries executed more than once, are highlighted. Page execution time was 19713.22 ms.
Any ideas?