I am working on a web application which will primarily be used by authenticated users. The web application imports records from CSV files using the Feeds module, MongoDB was chosen to store this data because it is much faster to do the writes rather than in MySQL. The application is importing thousands of records every day really fast. The records are referenced to the users and when they log in, they can consult their information. The application is also used as an API for the mobile app.
I have a 17gb AWS EC2 Instance as the server.
Varnish is install working perfectly serving content for anonymous users. I'm getting around 3000-4000 requests per second which is more than enough.
My problem is when benchmarking for authenticated users, on API pages that are only json content I get only 24 requests per second. And other pages I get like 9 - 11 requests per second. Here is some apache benchmark output.
Concurrency Level: 100
Time taken for tests: 9.154 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 1790900 bytes
HTML transferred: 1731600 bytes
Requests per second: 10.92 [#/sec] (mean)
Time per request: 9154.469 [ms] (mean)
Time per request: 91.545 [ms] (mean, across all concurrent requests)
Transfer rate: 191.05 [Kbytes/sec] receivedMemcache is installed, using memcache_storage module, for some reason I got slightly higher requests per second using this module rather than memcache. 4Gb are set as Memcache cache. Here is some Memcache statistics output.
Cache Information
Current Items(total) 10730 (95109)
Hits 644064
Misses 74324
Request Rate (hits, misses) 1.44 cache requests/second
Hit Rate 1.29 cache requests/second
Miss Rate 0.15 cache requests/second
Set Rate 0.19 cache requests/secondAPC is also installed, working perfectly.
I don't have many modules installed:
CORE
block, field, field sql storage, field UI, file, filter, image, list, locale, menu, node, number, options, path, system, taxonomy, text, user,
CONTRIB
content access, varnish, ctools, eck, feeds, feeds ui, feeds tamper, feeds tamper ui, email, field permissions, hsm field, node reference, references, user reference, jquery update, mongodb, mongodb storage, mongodb watchdog, entity api, disable messages, image url formatter, job scheduler, libraries, quicktabs, quicktabs styles, apc, eck entitycache, entitycache, memcache storage, services, rest server, entityfieldquery views, views, views json, and views ui.
As for the Devel output, usually I get something like this. (This output was after I refresh the page)
Executed 57 queries in 10.79 ms. Queries exceeding 5 ms are highlighted. Page execution time was 167.27 ms. Memory used at: devel_boot()=1.36 MB, devel_shutdown()=13.96 MB, PHP peak=14.25 MB._drupal_session_write query is always the longest, it never executes in less than 8 ms, and sometimes it reaches 500 ms like in the output below.
Executed 57 queries in 607.49 ms. Queries exceeding 5 ms are highlighted. Page execution time was 2395.87 ms. Memory used at: devel_boot()=1.36 MB, devel_shutdown()=13.96 MB, PHP peak=14.25 MB.These are my PHP settings in settings.php
# MongDB configuration
$conf['mongodb_connections'] = array(
// Connection name/alias
'default' => array(
// Omit USER:PASS@ if Mongo isn't configured to use authentication.
'host' => 'mongodb://localhost',
// Database name
'db' => 'mydb',
),
);
$conf['mongodb_watchdog'] = 'mydb_logs';
$conf['field_storage_default'] = 'mongodb_field_storage';
# Memcached
$conf['cache_backends'][] = 'sites/all/modules/memcache_storage/memcache_storage.inc';
$conf['cache_default_class'] = 'MemcacheStorage';
$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
$conf['session_inc'] = 'sites/all/modules/memcache_storage/includes/session.inc';
$conf['lock_inc'] = 'sites/all/modules/memcache_storage/includes/lock.inc';
# Varnish
$conf['cache_backends'][] = 'sites/all/modules/varnish/varnish.cache.inc';
$conf['cache_class_cache_page'] = 'VarnishCache';
# Add APC Caching.
$conf['cache_backends'][] = 'sites/all/modules/apc/drupal_apc_cache.inc';
$conf['cache_class_cache'] = 'DrupalAPCCache';
$conf['cache_class_cache_bootstrap'] = 'DrupalAPCCache';Configurations in my.cnf file.
key_buffer_size = 12M
query_cache_size = 32M
query_cache_limit = 2M
query_cache_type = 1
# InnoDB caches.
innodb_buffer_pool_size = 500M
innodb_flush_method = O_DIRECT
max_connections = 75
table_cache = 96
sort_buffer_size = 12M
tmp_table_size = 12MI really feel lost at this moment, I have read numerous articles about performance and scalability and I feel like I've done most of the things that are mentioned, but don't seem to work at all. I'm stressed because I feel like the server will crash really fast with this numbers. I hope someone can give me advices.
Kind regards.
Comments
Some loose ideas
Drop Apache + mod_php (I assume you're using that).
Use Nginx + php-fpm. Well configured should buy you a good deal of concurrency.
Why are you using APC for caching data? AFAIK that's not very wise due to cache
fragmentation.
Your InnoDB buffer pool is way too small regarding the memory of the machine
you have. You should increase it. It depends on how hammered is your machine
but at least 50% of the total memory, ideally 75%. Using Nginx + php-fpm will
liberate resources from PHP handling instead of using Apache.
Try the Nginx cache with micro caching. It will free the shared memory
taken by Varnish. You could also use the file layer instead of memory for
Varnish but I venture the Nginx cache is more performant. Try it.
Start there and see how it fares.
PS: Ah yes if using Nginx then don't use ab for testing the performance. You will hit APR related limits. Use a modern tool like wrk to get the real server limits. It's scriptable in Lua so you can test everything you want.
Great start
1, 2. I don't have much experience with ngnix, but I have read that is really good, so I'll try it, and if its going to help with concurrency sounds awesome.
3. APC helped reducing the page execution by 50% - 70%.
4. If I increase the memory in the InnoDB buffer, does it mean I have to reduce it for Memcache? I have allocated 4GB in Memcache.
5. I'm using the file layer instead of memory for Varnish, since the application is primarily used by authenticated users I prefer use memory resources for this purpose.
Thanks for your advices António.
Well APC
for opcode or data caching? I meant data caching. Let's say that you put the variables there and you don't touch at the settings that often. In that case it works ok. It's only when you update the data to be cached frequently that it poses problems.
As for sessions, it's true that most lambda developers just use it and abuse it. I'm currently working on a client project replacing session variables by a secure data cookie. No storage backend intervention. If you need to update the session often use something like that. I have plans to publish the module for D7. Let's see.
Here's the project
for securely storing data on a cookie: https://drupal.org/project/secure_cookie_data.
The confidentiality/crypto part is TBD.
You already identified the
You already identified the problem, I'll try to pinpoint the cause.
You need to make sure that
$_SESSIONis only changed when absolutely necessary. If it is modified on every page request, the session has to be stored in the database and retrieved on a subsequent request.It is also important to know that messages are stored in the session. Given that you have the Disable Messages module, installed, I assume that the culprit is some code constantly emitting messages.
Therefore my advice is:
Obviously if your messages are caused by PHP errors, you should fix the code causing them in the first place.
I'm currently an external
I'm currently an external contractor on a site where the internal developers sure love to abuse globals, including $_SESSION. They're also using Oracle, which has a really buggy PDO driver which causes reads and writes of "large" fields (over 2K or so… like, say, a serialized $_SESSION variable if the devs have been abusing it) very slow. Refactoring their custom code to not use $_SESSION so much was very juicy low-hanging fruit.
The Boise Drupal Guy!
I had no idea that messages
I had no idea that messages were store in session. I have disabled the module along with locale which was also generating long queries. Still _drupal_session_write query is taking its time to execute, I have to find out which other module is causing it, I have the feeling it might be content access related.
After I uninstall those I reached 13.5 requests per second, not significant, but it increased.
Thanks.
If you still see frequent
If you still see frequent calls to
_drupal_session_writethen you still have code tampering with the$_SESSION. It is always a warning-sign when an application writes to the database in the scope of a HTTP GET request.