Dear Drupal community,
We would like some advice and maybe further collaborations regarding a Drupal-based project, specifically concerning performance issues.
First a few words about the project: www.greatfirewall.biz. We are the only website which continously monitors the firewall of China - probably the worlds most sophisticated online censorship and certainly the one affecting the most people. Our aim is to provide transparent information on trends and changes of the censorship, as well as a basic tool to check whether a certain website or Google search is blocked or not. If you help us out on some issues regarding our Drupal-based website, you are also assisting in working for an free internet. Thank you.
Now, the issues. Our website is very slow and we don't know why or how to fix it. Many pages take more than 20 seconds to load which is both irritating and hurting our search engine ranking.
If you visit the website - http://www.greatfirewall.biz - you may initially find it fast enough. That's because we've enabled the Boost module meaning that you're often hitting static pages. These have to be regenerated of course and that takes time.
Some info on the website.
Core: Drupal 7.0.
Contrib modules: adsense devel metatags_quick styles views_php wysiwyg ajax_poll boost ctools entitycache globalredirect mollom references token xmlsitemap apc customfilter google_analytics media pathauto smtp views (plus some self-made modules).
PHP Info: http://www.greatfirewall.biz/phpinfo.php.
Web host: EC2. m1.small instance-store. 1.7GB memory. 10GB disk.
TOP snapshot:
top - 09:15:42 up 60 days, 6:35, 2 users, load average: 5.36, 5.42, 5.70
Tasks: 74 total, 2 running, 72 sleeping, 0 stopped, 0 zombie
Cpu(s): 41.3%us, 1.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 57.1%st
Mem: 1741016k total, 1671312k used, 69704k free, 49224k buffers
Swap: 917496k total, 7564k used, 909932k free, 1320304k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28737 www-data 20 0 80116 39m 23m R 13.2 2.3 0:02.16 apache2
28750 www-data 20 0 74488 34m 23m S 11.6 2.1 0:01.86 apache2
28675 mysql 20 0 391m 65m 6232 S 6.6 3.8 0:12.20 mysqld
28743 www-data 20 0 74988 35m 23m S 4.3 2.1 0:02.03 apache2
28651 www-data 20 0 74676 35m 23m S 2.6 2.1 0:04.57 apache2
28754 www-data 20 0 74476 34m 23m S 2.6 2.0 0:01.83 apache2
28576 www-data 20 0 74140 34m 23m S 2.0 2.0 0:05.16 apache2
1 root 20 0 2812 988 660 S 0.0 0.1 0:08.53 init
Actions taken so far:
- Enabled APC. This seemed to speed up the website initially but it appears that something happens after a while which brings performance down again.
- Increased Mysql's key_buffer from 16M to 256M.
- Enabled the Entitycache module.
- Ran a lot of Apache benchmarking (ab).
Do you have other ideas for what we could do? All advice very welcome! We would also be interested in more significant contributions if you want to collaborate. You can reply here or email us at greatfirewall.biz@gmail.com.
By the way, the website is blocked in China, but we have a mirror accessible at https://122.248.216.0. If you are in China you can use that to access the website.
Thank you!
Comments
Sample page loads from
Sample page loads from Devel:
Executed 166 queries in 1001.03 ms. Queries exceeding 5 ms are highlighted. Page execution time was 10192.28 ms.
Executed 283 queries in 4631.46 ms. Queries exceeding 5 ms are highlighted. Page execution time was 24397.29 ms.
Executed 209 queries in 3146.2 ms. Queries exceeding 5 ms are highlighted. Page execution time was 12697.86 ms.
looks like
quick look tells me that your web is spending a lot of time on external javascript files like google syndication and twitter etc. Where are you accessing your website from? Which country?
Yes, there are external
Yes, there are external javascript but I'm not concerned with those right now. I'm measuring the time to download the html only using Apache Benchmark and that frequently takes 20s+.
I don't see any significant
I don't see any significant JavaScript issues. It appears that when doing a search the search portion completes in ~ 1s, but the time to load the results is ~8.5s. The result page is just a full node view. I would enable the devel query output to see if there are any really bad queries being executed on the full node view page.
As for APC you are probably not giving enough RAM to APC. Find the appropriate version of apc.php here:
http://svn.php.net/viewvc/pecl/apc/tags/RELEASE_3_0_19/apc.php?revision=...
Put it somewhere on your site and see if the Cache full count is > 0, or if there is significant fragmentation. If so you must increase apc.shm_size and reboot Apache. I would disable the APC Drupal module until you find the source of your woes. Using an alternative cache backend will not help to cover a gaping wound. To elaborate, APC can be used in two ways: It stores a compiled version of your PHP code in memory and shares this among all Apache processes. This way the PHP code doesn't need to be recompiled on every page load. Secondly it can be used as a key-value store. The APC Drupal module uses this as an alternative place to store Drupal's cache records (as opposed to the database).
Increasing the key buffer probably did nothing for you. Drupal 7 uses InnoDB and not MyISAM. You can try the dbtuner module to get some helpful suggestions:
http://drupal.org/project/dbtuner
But take care to increase the buffers to no more than 4x the default until you have a strong understanding of what each does.
Where are you located? I'm in Hong Kong.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Hi Dave, Thanks a lot for
Hi Dave,
Thanks a lot for lots of useful info! I've downloaded apc.php and placed it here: http://www.greatfirewall.biz/apc.php. Right now Cache full count is 364 (very bad?) and there seems to be a lot of fragmentation. I had already increased the memory size in php.ini to apc.shm_size 512M. However, in apc.php it says that shared memory is 1 Segment(s) with 30.0 MBytes (mmap memory, pthread mutex locking). Any idea why the two values are different?
The APC lines I've added to my settings.php are these:
$conf['cache_backends'] = array('sites/all/modules/apc/drupal_apc_cache.inc');
$conf['cache_class_cache'] = 'DrupalAPCCache';
$conf['cache_class_cache_bootstrap'] = 'DrupalAPCCache';
I ran a search on the website and this is the output from Devel: Executed 234 queries in 15557.12 ms. Queries exceeding 5 ms are highlighted. Page execution time was 18919.42 ms.
The slowest query by far was:
6402.771
DrupalDatabaseCache::getMultiple
P A E
SELECT cache_entity_node.cid AS cid, cache_entity_node.data AS data, cache_entity_node.created AS created, cache_entity_node.expire AS expire, cache_entity_node.serialized AS serialized FROM cache_entity_node cache_entity_node WHERE (cache_entity_node.cid IN (:db_condition_placeholder_0, :db_condition_placeholder_1, :db_condition_placeholder_2, :db_condition_placeholder_3, :db_condition_placeholder_4, :db_condition_placeholder_5, :db_condition_placeholder_6, :db_condition_placeholder_7, :db_condition_placeholder_8, :db_condition_placeholder_9, :db_condition_placeholder_10, :db_condition_placeholder_11, :db_condition_placeholder_12, :db_condition_placeholder_13, :db_condition_placeholder_14, :db_condition_placeholder_15, :db_condition_placeholder_16, :db_condition_placeholder_17, :db_condition_placeholder_18, :db_condition_placeholder_19, :db_condition_placeholder_20, :db_condition_placeholder_21, :db_condition_placeholder_22, :db_condition_placeholder_23, :db_condition_placeholder_24, :db_condition_placeholder_25, :db_condition_placeholder_26, :db_condition_placeholder_27, :db_condition_placeholder_28, :db_condition_placeholder_29, :db_condition_placeholder_30, :db_condition_placeholder_31, :db_condition_placeholder_32, :db_condition_placeholder_33, :db_condition_placeholder_34, :db_condition_placeholder_35, :db_condition_placeholder_36, :db_condition_placeholder_37, :db_condition_placeholder_38, :db_condition_placeholder_39, :db_condition_placeholder_40, :db_condition_placeholder_41, :db_condition_placeholder_42, :db_condition_placeholder_43, :db_condition_placeholder_44, :db_condition_placeholder_45, :db_condition_placeholder_46, :db_condition_placeholder_47, :db_condition_placeholder_48, :db_condition_placeholder_49, :db_condition_placeholder_50, :db_condition_placeholder_51, :db_condition_placeholder_52, :db_condition_placeholder_53, :db_condition_placeholder_54, :db_condition_placeholder_55, :db_condition_placeholder_56, :db_condition_placeholder_57, :db_condition_placeholder_58, :db_condition_placeholder_59, :db_condition_placeholder_60, :db_condition_placeholder_61, :db_condition_placeholder_62, :db_condition_placeholder_63, :db_condition_placeholder_64, :db_condition_placeholder_65, :db_condition_placeholder_66, :db_condition_placeholder_67, :db_condition_placeholder_68, :db_condition_placeholder_69, :db_condition_placeholder_70, :db_condition_placeholder_71, :db_condition_placeholder_72, :db_condition_placeholder_73, :db_condition_placeholder_74, :db_condition_placeholder_75, :db_condition_placeholder_76, :db_condition_placeholder_77, :db_condition_placeholder_78, :db_condition_placeholder_79, :db_condition_placeholder_80, :db_condition_placeholder_81, :db_condition_placeholder_82, :db_condition_placeholder_83, :db_condition_placeholder_84, :db_condition_placeholder_85, :db_condition_placeholder_86, :db_condition_placeholder_87, :db_condition_placeholder_88, :db_condition_placeholder_89, :db_condition_placeholder_90, :db_condition_placeholder_91, :db_condition_placeholder_92, :db_condition_placeholder_93, :db_condition_placeholder_94, :db_condition_placeholder_95, :db_condition_placeholder_96, :db_condition_placeholder_97, :db_condition_placeholder_98, :db_condition_placeholder_99))
Thanks a lot for your help! If you want to get more involved with the project you're welcome to. There's a lot to work on.
I figured out the APC memory
I figured out the APC memory limit issue. I had it formatted wrongly in php.ini as "512M". It should be a number only. Changed it now to 256.
Just like it's difficult to
Just like it's difficult to say but perhaps can you post all sql query who are very slow. Personally, it's the first things I'll check. After, I'll check, PHP code problem, Cache problem...
Julien Didelet
Founder
Weblaa.com
Thanks Julien. It seems APC
Thanks Julien. It seems APC was the main problem. The queries aren't so slow, mostly, but the page execution time was very high. Any other advice for what we should do with our website?
The site seems a lot faster
The site seems a lot faster now that we managed to increase the APC memory limit. Thanks for all your advice! More help is always welcome. Another issue we have is that the Boost module doesn't seem to flush its cache when nodes are changed or commented on. Any idea on how to solve that?
More problems
The website stopped responding completely. When I restarted Apache, I got these errors:
[apc-error] apc_mmap: mmap failed: Cannot allocate memory
[apc-warning] Potential cache slam averted for key 'www.greatfirewall.biz|cache_bootstrap|variables' in sites/all/modules/apc/drupal_apc_cache.inc on line 173.
I then decreased apc.shm_size to 128 and added apc.slam_defense = Off. Right now the website seems to be working ok. Am I on the right track? Any idea why this happened?
Out of memory?
See if you have something similar in syslog. It seems like you've run out of memory. Are you using virtual machine with limited amout of memory? In my vhost environment shell top-command does NOT show vhost real memory limit. It needs to be checked within control panel.
Try this:
grep memory /var/log/messagesIn your case (post #1) it seems like you do have enought memory. But if you get too much of those 'cannot allocate memory' -errors you may not have enough of memory for all processes you're running.
--
Perttu Ehn
Thanks Ropsu, You were right,
Thanks Ropsu,
You were right, the machine was out of memory. I realized that a background process was using up most of it. Now that its cleared up the website is fast.
Thanks for all your great advice!