Memcached Optimization

Posted by mbopp on July 2, 2013 at 3:00pm

I have a large website that uses memcached. All users of the website are authenticated (no anonymous users). In addition to users always being authenticated, the content they see is very customized based on user (profile) fields and properties; like social media sites including facebook, linkedin, etc. Also, this site has several thousand users.

I also have New Relic Service set up for the site. Which alerted me to how much time is being spent in memcached. > 800 ms. Total page load times are consistently above 1 sec. Disabling memcached improves performance a little instead spending extra time communicating with the database and PHP processing.

My site is configured as follows...

<?php
$conf['cache_backends'][] = 'sites/all/modules/contrib/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['memcache_key_prefix'] = 'ccprod';
$conf['memcache_servers'] = array(
  '127.0.0.1:11211' => 'default',
);
$conf['memcache_bins'] = array(
  'cache' => 'default',
);
?>

I have a Gigabyte configured for memcached (but the site typically uses about 250mb of that).

Much of the contextualized data displayed on the site (which is just about everything) is filtered using a custom views handler/filter. After writing this custom filter I then just add it to each view that needs it.

I'm wondering if memcached is ineffective because the queries vary so much due to contextualization. Or if I'm not taking something into account that I should be. Perhaps the way memcached is configured, or the way my queries are constructed (as a custom view filter).

Does anyone have thoughts on ways I can improve my usage of memcached?

Thanks so much!

Attachment	Size
Screenshot_7_2_13_9_18_AM.png	153.97 KB

Comments

You may move your lock system

Posted by Spleshka on July 2, 2013 at 3:12pm

You may move your lock system and user sessions to memcached. I'd suggest you to use Memcache Storage instead of Memcache API for numerous reasons.

The second advice is to use sockets for memcached connections instead of tcp.

The third advice is to use configured PECL Memcached instead of PECL Memcache.

And the last but not least - configure settings for memcached daemon properly. It is very important to pay attenction on memory fragmentation.

Good configuration for you

Posted by Spleshka on July 2, 2013 at 3:20pm

Good configuration for you may look like that:

<?php
# Move cached data into memcached.
$conf['cache_backends'][] = 'sites/all/modules/memcache_storage/memcache_storage.inc';
$conf['cache_default_class'] = 'MemcacheStorage';
$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
$conf['cache_class_cache_update'] = 'DrupalDatabaseCache';

# Configure Memcache Storage module.
$conf['memcache_storage_debug'] = FALSE;
$conf['memcache_storage_wildcard_invalidate'] = 60 * 60 * 24 * 5; // 5 days.

# Configure memcached instance.
$conf['memcache_servers'] = array(
  'unix:///var/run/memcached/memcached.socket0' => 'default',
);

# Set current extension.
$conf['memcache_extension'] = 'Memcached';

# Configure memcached extenstion.
$conf['memcache_options'] = array(
  Memcached::OPT_TCP_NODELAY => TRUE,
  Memcached::OPT_NO_BLOCK => TRUE,
  Memcached::OPT_BINARY_PROTOCOL => TRUE,
);

# Move storage for lock system to memcached.
$conf['lock_inc'] = 'sites/all/modules/memcache_storage/includes/lock.inc';

# Move storage for users' sessions to memcached.
$conf['session_inc'] = 'sites/all/modules/memcache_storage/includes/session.inc';
?>

memcached daemon settings

Posted by mbopp on July 2, 2013 at 3:21pm

Thanks Spleshka, when you refer to configuring things properly to prevent fragmentation, do you just mean the amount of memory you allocate to the daemon? Or are there other settings I should pay attention to?

Mike Bopp

Nice

Posted by mbopp on July 2, 2013 at 3:22pm

Thank You! for the sample configuration!

Mike Bopp

do you just mean the amount

Posted by Spleshka on July 2, 2013 at 3:27pm

do you just mean the amount of memory you allocate to the daemon>

No. I mean smart configuration, including some fragmentation settings. See example of my memcached configuration for CentOS:

#! /bin/sh
#
# chkconfig: - 55 45
# description:    The memcached daemon is a network memory cache service.
# processname: memcached
# config: /etc/sysconfig/memcached
# pidfile: /var/run/memcached/memcached.pid
 
# Standard LSB functions
#. /lib/lsb/init-functions
 
# Source function library.
. /etc/init.d/functions
 
USER=memcached
OPTIONS="-a 0766 -c 2048 -f 1.07 -n 512 -L"
RETVAL=0
PROG="memcached"
 
start_instance() {
    echo -n $"Starting $PROG ($1): "       
    daemon --pidfile /var/run/memcached/memcached.$1.pid memcached -d -s $3 -u $USER -m $2 -P /var/run/memcached/memcached.$1.pid -v 2>> /var/log/memcached/memcached.$1.log $OPTIONS
 
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/memcached.$1
}
 
stop_instance() {
    echo -n $"Stopping $PROG ($1): "
    killproc -p /var/run/memcached/memcached.$1.pid /usr/bin/memcached
    RETVAL=$?
    echo
    if [ $RETVAL -eq 0 ] ; then
        rm -f /var/lock/subsys/memcached.$1
        rm -f /var/run/memcached.$1.pid
    fi
}
 
start () {
    # insure that /var/run/memcached has proper permissions
    if [ "stat -c %U /var/run/memcached" != "$USER" ]; then
        chown $USER /var/run/memcached
    fi
 
    # start 3 socket streams for memcached.
    start_instance default 1024 /var/run/memcached/memcached.socket0;
}
 
stop () {
    stop_instance default;
}
 
restart () {
    stop
    start
}
 
 
# See how we were called.
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  status)
    status memcached
    ;;
  restart|reload|force-reload)
    restart
    ;;
  condrestart)
    [ -f /var/lock/subsys/memcached ] && restart || :
    ;;
  *)
    echo $"Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
    exit 1
esac
 
exit $?

And Debian startup daemon

Posted by chilic on July 2, 2013 at 3:30pm

And Debian startup daemon script :)
http://www.chilic.ru/blog/debian-memcached-socket-start-script

5 days. - Wooow!

Posted by chilic on July 2, 2013 at 3:30pm

5 days. - Wooow!

Beware memcached extension

Posted by fgm on July 3, 2013 at 1:38pm

On various systems I have found it be consistently not only slower than the "old" memcache extension, but even slower than a (tuned) MySQL. So the most important advice I would add after this is: "Measure".

Memcached Extension

Posted by mbopp on July 3, 2013 at 2:08pm

Thanks, I'll be keeping a close eye on performance as i change things. It's good to know that in installing certain processes, things will not automatically be faster.

Mike Bopp

PECL MemcacheD Extension

Posted by mbopp on July 8, 2013 at 4:17pm

I'm having a heck of a time getting this pecl extension installed on ubuntu 10.04. I know this extenstion relies on the libmemcached library. But the incompatibilities of what versions are provided by aptitude is a real pain. I'd like to benchmark them both, but for now I guess I'll stick with the existing memcache pecl extension. Especially since it is up in the air as to whether the memcacheD extension is really any better.

Anyone have similar issues with the memcached extension? I tried manually building different versions to no avail. I'm sure I can get it figured out, but would rather not sink the time into it now.

Mike Bopp

libmemcached issue

Posted by kjala on July 17, 2013 at 4:07pm

I had similar issues with installing libmemcached. I went with the memcache pecl(and its working) instead of the memcached extension which needs libmemcached.

I concur

Posted by mbopp on July 17, 2013 at 4:14pm

Turns out that my production server was using the libmemcached version. (test was already using memcache) Switching production to the PECL memcache version resolved the issue and improved performance a great deal.

Mike Bopp

Correct Memcached packages

Posted by nikhileshpaul on June 8, 2017 at 10:36am

@Mike Bopp, I would appreciate if you can elaborate your comment here. Currently I can see the below 2 packages installed in my servers:
libmemcached-0.31-1.1.el6.x86_64
memcached-1.4.4-5.el6.x86_64

So, are you telling that installing php-pecl-memcache.x86_64 or php-pecl-memcached.x86_64, instead of libmemcached-0.31-1.1.el6.x86_64 will improve the performance

content cache does not clear

Posted by kjala on July 18, 2013 at 2:15pm

Not to hijack this thread but on the same line of memcache module, after installing and running the memcache, content cache is not clearing up on publishing content with modification for anonymous users even after the 5min cache time set in Drupal. Any ideas or does anything else to be tweaked or a new module to be installed along with Memcache to deal with cache expiration.

I believe you need to

Posted by mbopp on July 18, 2013 at 2:26pm

I believe you need to configure your memcache expiration outside of the regular drupal settings. As shown in examples above you can set the 'memcache_storage_wildcard_invalidate' property if you are using the memcache_storage module.

<?php
$conf['memcache_storage_wildcard_invalidate'] = 60 * 60 * 24 * 5; 
?>

for 5 days of cache.

I also believe that cache is not eliminated, but rather invalidated. So the next time the php is requested it will be re-"compiled". I just mention this in case you are seeing that the memory usage remains high.

Mike Bopp

Thanks mbopp. I'm using

Posted by kjala on July 18, 2013 at 2:36pm

Thanks mbopp. I'm using memcache and on testing it in Dev the usage is around 60%. Due to the frequency of content updates by multiple editors we have set drupal cache expire to 5mins. Is there a similar wildcard invalidate setting for memcache module too?

@kiranjala, use Memcache

Posted by Spleshka on July 18, 2013 at 2:21pm

@kiranjala, use Memcache Storage module instead of Memcache.

@Spleshka, I have seen your

Posted by kjala on July 18, 2013 at 2:41pm

@Spleshka, I have seen your posts recently in other places and liked the progress of your project. The memcache namespace has some weight in the team and so I went with that. I would sure consider Memcache Storage if memcache module does not work out well.

I wish the Memcache module page posts links to alternate or similar modules like yours so we have options in the first place.

There is an issue with a

Posted by jason.fisher on July 18, 2013 at 4:25pm

There is an issue with a specific version of either memcache/memcached being enabled in php.d that is known to cause large slowdowns with memcached access. I would look there..

If you haven't tried xhprof yet, it will give you good insight into where you are getting stuck and whether it is actual CPU time or being blocked by an external source (memcached/mysql).

In my experience, the biggest general performance hit in D7 to large sites (100,000 nodes/2,000 fields) is time spent in unserialize. I have seen several cases where 1/3 of the execution time is spent there, so keep your $forms tidy.

Thanks for recommending

Posted by loudpixels on February 12, 2016 at 4:52am

Thanks for recommending memcache storage ... I was using memcache but had many issues with user sessions ... now everything seem to be working great.

Quick question I used the config file you posted in the 3rd post but had to disable the following:

Configure memcached instance.

$conf['memcache_servers'] = array(
'unix:///var/run/memcached/memcached.socket0' => 'default',
);

Configure memcached extenstion.

$conf['memcache_options'] = array(
Memcached::OPT_TCP_NODELAY => TRUE,
Memcached::OPT_NO_BLOCK => TRUE,
Memcached::OPT_BINARY_PROTOCOL => TRUE,
);

Would love to know more about what these settings do in order to get most of my server:

Here are my specs:
Intel® Xeon® Processor E5-2650 w/ 32GB RAM
Centos 6
Drupal 4.71
Memcache Storage: PECL Memcache extension Version: 3.0.8

The website using this is currently doing about 75k views per day with peaks reaching 1000 concurrent users.

Any help improving speed would be appreciated.

Thanks again for the great module and for your help.

Not really sure on those

Posted by Jamie Holly on February 12, 2016 at 5:09am

Not really sure on those settings, but I would suggest upgrading Drupal as caching improvements have been made over the years. I handle a site for a client on the same server specs, except only 16gb of RAM, running on Drupal 6. We also use memcache. We got over 150,000 nodes, with 20+ being added per day, a heavy authenticated user base and average over 350,000 page views per day and never have any problems. During one peek time we had over 60,000 page views in one hour and our server load never went above 3.

HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

$conf['memcache_servers'] =

Posted by Spleshka on February 12, 2016 at 10:06am

$conf['memcache_servers'] = array(
'unix:///var/run/memcached/memcached.socket0' => 'default',
);

In this setting we switch to use to sockets instead of tcp connection. It's usually 10-30% faster than tcp connection. Although sockets can be used only for local server set up and don't support shared memcached instance for a bunch of servers. You should adjust your memcached config on the server in order to enable sockets. Just google this stuff, the examples will pop in the search results.

$conf['memcache_options'] = array(
Memcached::OPT_TCP_NODELAY => TRUE,
Memcached::OPT_NO_BLOCK => TRUE,
Memcached::OPT_BINARY_PROTOCOL => TRUE,
);

Well, it's sort of "claimed-to-improve-memcached-performance" configs. All these settings work only with pecl memcached. Just google for each of them and you'll get a detailed description for each of those.

One more thing to mention here is even though "Memcached::OPT_NO_BLOCK => TRUE" is proven to give a good performance boost by myself, I've experienced very weird and not reproducable issues when turned this setting on.

Pretty impressive numbers

Posted by loudpixels on February 12, 2016 at 5:20am

Pretty impressive numbers Jamie and very re-comforting since our numbers are exponentially growing on a monthly basis.

Sorry I meant to say Drupal 7.41 ... my dyslexia got the better of me :)

Thanks!

Thanks Spleshka for this

Posted by loudpixels on February 12, 2016 at 1:52pm

Thanks Spleshka for this explanation ... I will google it and see if it's something I can setup.

Another quick weird behavior is that now the "Who's online" block is reporting 0 users althought I currently have 500 active users on the site.

Any thoughts on what could be causing this?

Thanks!

Yeah, "Who's online" block

Posted by Spleshka on February 12, 2016 at 2:03pm

Yeah, "Who's online" block counts row in database {session} table. There is no way to count how many active sessions using memcached. It's architecture (memcached architecture, not the module's one) doesn't allow any counting operation like that.

I guess I spoke too soon ...

Posted by loudpixels on February 12, 2016 at 4:05pm

I guess I spoke too soon ... still having problems with user sessions.

If I login via Firefox for instance ... the system ask me for my login credentials ... cool but if I switch to another browser and try to login with the same credentials it refuses and asks me to reset my password ... once I do this and go back to the Firefox, same thing.

Any ideas how to fix this?

I did set this config setting:

$conf['memcache_storage_wildcard_invalidate'] = 180; / 3 minutes

Could this setting have caused this?

Thanks

Could this setting have

Posted by Spleshka on February 12, 2016 at 4:36pm

Could this setting have caused this?

Nope, please leave this variable to its defaults. It's quite internal and usually should not be changed.

One more thing ... the system

Posted by loudpixels on February 12, 2016 at 4:19pm

One more thing ... the system reports too many tries from your IP. IP blocked ... request a new password. (even tho it's the first attempt)

Requesting a new password lets me back in.

This is driving me bunkers!

Unlikely this behavior is

Posted by Spleshka on February 12, 2016 at 4:39pm

Unlikely this behavior is related to memcached. You should search for issue with authentication.

Thanks Spleshka ... that's

Posted by loudpixels on February 12, 2016 at 5:47pm

Thanks Spleshka ... that's what I was starting to think ... it might be related to varnish :(

I'm going to look around ... thanks again! and great module!!!! Thank you very much!

Yeap ... the problem was with

Posted by loudpixels on February 12, 2016 at 6:50pm

Yeap ... the problem was with Varnish ... had to disable it for now :(

Thanks for all your help!

Fatal error:Out of memory (allocated 17039360) (tried to allocat

Posted by ameykinalekar on August 1, 2017 at 8:44am

Hi All

I have Installed memcache on windows, And also installed the memcache php extension. Enabled memcache and memcache admin module.

Done the following configuration in setting.php.

     $conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';
     $conf['cache_default_class'] = 'MemCacheDrupal';
     $conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
 $conf['page_cache_without_database'] = TRUE;
     $conf['page_cache_invoke_hooks'] = FALSE;

I started jmeter to check the load for 30 users it was working fine but once i crossed 60 it generated error "Fatal error: Out of memory (allocated 17039360) (tried to allocate 7680 bytes) in "

Can anyone knows anything about this.

How does one figure our the optimal setting for memcached

Posted by doyle@iwanamaker.com (not verified) on February 18, 2018 at 5:25am

This is my first post and have never worked with Drupal before. I started my company in 2011 and have left it up to my developers to do whatever they wanted until last fall when I took over. Now feel a bit overwhelmed to say the least. I'm an old Microsoft Windows guy and now feel like I am now drinking through a fire hose learning everything I can about Drupal 7, Amazon Linux, Apache, MySQL and all the custom code that has been written since 2011.
Many times over the past 7 years I have heard my developers talk a lot about the importance of memcached so I have been trying to learn as much as I can and am at the point of trying to figure out if the settings are right for my site. What is the best approach in determining the optimal settings?

Comments

You may move your lock system

Good configuration for you

memcached daemon settings

Nice

do you just mean the amount

And Debian startup daemon

5 days. - Wooow!

Beware memcached extension

Memcached Extension

PECL MemcacheD Extension

libmemcached issue

I concur

Correct Memcached packages

content cache does not clear

I believe you need to

Thanks mbopp. I'm using

@kiranjala, use Memcache

@Spleshka, I have seen your

There is an issue with a

Thanks for recommending

Not really sure on those

$conf['memcache_servers'] =

Pretty impressive numbers

Thanks Spleshka for this

Yeah, "Who's online" block

I guess I spoke too soon ...

Could this setting have

One more thing ... the system

Unlikely this behavior is

Thanks Spleshka ... that's

Yeap ... the problem was with

Fatal error:Out of memory (allocated 17039360) (tried to allocat

How does one figure our the optimal setting for memcached

High performance

Group organizers

New groups

Group notifications

Hot content this week