Hi,
Posting this here (rather than general performance group) because I know Mercury and APC are tightly integrated (though I'm still figuring out how).
I have a stock Mercury AWS install that's been running in production for several months. Nice.
Authenticated performance has been degrading, so I figured I'd take a look at the APC diagnostics screen (apc.php). God I wish there was a manual for dummies on this screen. Anyhow, I'm seeing some things that definitely don't look kosher and I'm hoping some of the Mercury Astronauts can help a poor man stuck back on earth.
The most seemingly worrisome diagnostic is the following:
Memory Usage
(multiple slices indicate fragments) Hits & Misses
Free: 16.1 MBytes (53.6%) Hits: 4891715 (43.0%)
Used: 13.9 MBytes (46.4%) Misses: 6483399 (57.0%)
Fragmentation: 100.00% ( 8.4 MBytes out of 8.4 MBytes in 1778 fragments)
If I refresh the data fragmentation bounces around on each refresh b/w 50% and 100%.
Is this bad? Some research suggests it is...
Here's some of the other diagnostics on the screen:
File Cache Information
Cached Files 149 ( 4.9 MBytes)
Hits 4940446
Misses 6515120
Request Rate (hits, misses) 37.07 cache requests/second
Hit Rate 15.99 cache requests/second
Miss Rate 21.08 cache requests/second
Insert Rate 20.56 cache requests/second
Cache full count 22143
User Cache Information
Cached Variables 0 ( 0.0 Bytes)
Hits 50
Misses 0
Request Rate (hits, misses) 0.00 cache requests/second
Hit Rate 0.00 cache requests/second
Miss Rate 0.00 cache requests/second
Insert Rate 0.02 cache requests/second
Cache full count 22143
Runtime Settings
apc.cache_by_default 1
apc.coredump_unmap 0
apc.enable_cli 0
apc.enabled 1
apc.file_update_protection 2
apc.filters
apc.gc_ttl 3600
apc.include_once_override 1
apc.max_file_size 1M
apc.mmap_file_mask
apc.num_files_hint 1000
apc.report_autofilter 0
apc.rfc1867 1
apc.rfc1867_freq 0
apc.rfc1867_name APC_UPLOAD_PROGRESS
apc.rfc1867_prefix upload_
apc.shm_segments 1
apc.shm_size 4096
apc.slam_defense 0
apc.stat 1
apc.stat_ctime 0
apc.ttl 0
apc.user_entries_hint 4096
apc.user_ttl 0
apc.write_lock 1
I guess it would help if I understood more about how & where Mercury and APC integrate. Does Mercury "control" APC settings?
Thanks to anyone willing to have a look. Hopefully this will help others wondering how APC and Mercury integrate and what an earthbound sysadmin should be looking for...
Mike
Comments
What's your apc.ini looks
What's your apc.ini looks like?
Benoit Borrel
oops too fast... it's already
oops too fast... it's already here...
Benoit Borrel
System limit for Shared memory block size?
what gives you the shell command: $ sysctl kernel.shmmax
Benoit Borrel
Yes, that's bad. Benoit's
Yes, that's bad.
Benoit's right. Huge apc.shm_size size of 4GB but only getting one 30MB segment cause you need to set kernel.shmmax higher. You can raise kernel.shmmax by putting "kernel.shmmax = 96000000" in /etc/sysctl.conf on debian/ubuntu systems and reboot. Different for other distros, I think. Or set higher number of apc.shm_segments in /etc/php5/conf.d/apc.ini but usually just set kernel.shmmax higher.
Depending on your number of sites and modules and stuff, increase size til you can keep Cache full count = 0.
Mmm...rather then multiple
Mmm...rather then multiple reboots, probably best to change the kernel.shmmax on the fly with
sysctl -w kernel.shmmax=96000000until you find the right number, and then edit the /etc/sysctl.conf when you find it so it will set itself on the next reboot.
thanks for the suggestions
thanks for the suggestions gentlemen. I'll give it a go.
Do you think apc.shm_size 4096 is unusual? Most of the discussions i see have it around 250 or less. However, this was the default set by Mercury, so I assume this is correct.
Also, what should the fragmentation % be at?
Thanks again,
Mike
Code probably fits within
Code probably fits within 30-96MB depending on how many modules you have. I'd try something like 96MB and see if it fills. Then increase/decrease it til it's about 75% full. Leaving some extra room helps prevent fragmentation. I'm not sure why Mercury would set it at 4096. I think it's just an upper limit and doesn't actually allocate that much memory, so probably no harm done, but still, I dunno why it would be set to 4GB. Are you on a small 1.7GB AMI?
My understanding is that Mercury uses memcached for user data caching, so APC should be caching code only. There's no reason there should be any fragmentation, but sometimes a few percent creep in, and you can flush it til it goes away. More disturbing is the 22143 times it tried to cache something, found the cache full, had to flush something, then reload the code into the cache, and all the cache misses involved in that. Cache hit rate for code only should be 99.99% excepting only the first time the code is loaded and when you update. Cache full count should be zero.
If you have multiple copies of Drupal in different directories, then APC will cache multiple copies of the code. However, this should not be an issue with the Drupal mulitsite feature. Are you running any other php stuff besides Drupal?
The 30MB default kernel setting for Ubuntu is low, but Mercury should know that it needs to be raised.
Brian, This is great
Brian,
This is great information. Thanks for the help. I really appreciate it.
I'm on a medium instance. I'm not sure why it's using 4 GB either, but as I mentioned that's how it came out of the box and have gotten into the habit of not messing with the base mercury settings. Perhaps Greg or Josh will chime in. Nevertheless, I think I will start moving levers as you suggest.
How would you go about doing that?
Would love to know too. This appears to be my big question now.
Just Munin monitoring software. But that's probably not PHP. I do run several copies of core as multisite for dev & staging. Might put those in another instance.
The 30MB default kernel setting for Ubuntu is low, but Mercury should know that it needs to be raised.
apc.php has a button for
apc.php has a button for "Clear opcode cache". Also, if you check out the "System Cache Entries" tab and sort by size you should be able to see if you're doubling or tripling up on the same files.
You have to set a password
You have to set a password and log in for those.
I fired up a 1.0 version of mercury on a medium instance and it set a 256MB apc cache. Not sure how you got 4GB
I fired up a 1.0 version of
Dang. Gremlins....
OK, so the plot thickens... I
OK, so the plot thickens...
I also fired up a fresh medium instance and like you it set the apc shm size to 256mb. But if you look at the Mercury installation guide there's the following in step 17:
If you look in config_mem.sh you'll see the following:
elif (($RAM>=15000000)); thenAPC_SIZE="4096"
PHP_SIZE="2048M"
TOMCAT_MAX_THREADS="1600"
VARNISH_SIZE="8192M"
elif (($RAM>=8000000)); then
I'm assuming this is how my APC size got set to 4096.
Anybody on the Mercury team know what's going on? And best practices in this situation? Can I lower my APC_SIZE to 256?
How much RAM do you have?
How much RAM do you have on this instance? That script should only set APC_SIZE to 4096 if you have more than 15GB of RAM installed.
Bitten by a bug
I think I know what happened here.
Back on 5/26, there was a bug fixed in the config_mem.sh script:
http://bazaar.launchpad.net/~pantheon-developers/pantheon/1.0/revision/114
Basically, any Mercury install that runs that file will be configured as though it had 16GB of RAM.
There's a few things you can do here.
1) There's a script somewhere you can run that will update mercury (update_mercury.sh IIRC). I have no experience with running this script, but it should get you up to date with all the bugfixes in Mercury 1.0 since you installed your instance.
2) If you're otherwise happy with performance, just download the latest config_mem.sh script from http://bazaar.launchpad.net/~pantheon-developers/pantheon/1.0/download/h... and drop that in /etc/mercury/. Then run the script. Note this script will restart varnish, apache, tomcat, and memcached, so plan on a very brief outage. This is the approach I would take personally, but it's completely up to you.
This explains your poor performance over time issue when you think about it. You're allocating more memory than what you have, so when you first start things up, things are okay. As these services start using more of the memory they've been allocated, then the box starts scrambling to provide it, likely going to swap at times.
Or, I could be completely wrong :)
Can I lower my APC_SIZE to 256?
Yes, you can change your size to 256.
You must have launched before we fixed the bug in the math there. ;)
https://pantheon.io | http://www.chapterthree.com | https://www.outlandishjosh.com