Memcached Optimization

mbopp's picture

I have a large website that uses memcached. All users of the website are authenticated (no anonymous users). In addition to users always being authenticated, the content they see is very customized based on user (profile) fields and properties; like social media sites including facebook, linkedin, etc. Also, this site has several thousand users.

I also have New Relic Service set up for the site. Which alerted me to how much time is being spent in memcached. > 800 ms. Total page load times are consistently above 1 sec. Disabling memcached improves performance a little instead spending extra time communicating with the database and PHP processing.

My site is configured as follows...

<?php
$conf
['cache_backends'][] = 'sites/all/modules/contrib/memcache/memcache.inc';
$conf['cache_default_class'] = 'MemCacheDrupal';
$conf['memcache_key_prefix'] = 'ccprod';
$conf['memcache_servers'] = array(
 
'127.0.0.1:11211' => 'default',
);
$conf['memcache_bins'] = array(
 
'cache' => 'default',
);
?>

I have a Gigabyte configured for memcached (but the site typically uses about 250mb of that).

Much of the contextualized data displayed on the site (which is just about everything) is filtered using a custom views handler/filter. After writing this custom filter I then just add it to each view that needs it.

I'm wondering if memcached is ineffective because the queries vary so much due to contextualization. Or if I'm not taking something into account that I should be. Perhaps the way memcached is configured, or the way my queries are constructed (as a custom view filter).

Does anyone have thoughts on ways I can improve my usage of memcached?

Thanks so much!

AttachmentSize
Screenshot_7_2_13_9_18_AM.png153.97 KB

Comments

You may move your lock system

Spleshka's picture

You may move your lock system and user sessions to memcached. I'd suggest you to use Memcache Storage instead of Memcache API for numerous reasons.

The second advice is to use sockets for memcached connections instead of tcp.

The third advice is to use configured PECL Memcached instead of PECL Memcache.

And the last but not least - configure settings for memcached daemon properly. It is very important to pay attenction on memory fragmentation.

Good configuration for you

Spleshka's picture

Good configuration for you may look like that:

<?php
# Move cached data into memcached.
$conf['cache_backends'][] = 'sites/all/modules/memcache_storage/memcache_storage.inc';
$conf['cache_default_class'] = 'MemcacheStorage';
$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
$conf['cache_class_cache_update'] = 'DrupalDatabaseCache';

# Configure Memcache Storage module.
$conf['memcache_storage_debug'] = FALSE;
$conf['memcache_storage_wildcard_invalidate'] = 60 * 60 * 24 * 5; // 5 days.

# Configure memcached instance.
$conf['memcache_servers'] = array(
 
'unix:///var/run/memcached/memcached.socket0' => 'default',
);

# Set current extension.
$conf['memcache_extension'] = 'Memcached';

# Configure memcached extenstion.
$conf['memcache_options'] = array(
 
Memcached::OPT_TCP_NODELAY => TRUE,
 
Memcached::OPT_NO_BLOCK => TRUE,
 
Memcached::OPT_BINARY_PROTOCOL => TRUE,
);

# Move storage for lock system to memcached.
$conf['lock_inc'] = 'sites/all/modules/memcache_storage/includes/lock.inc';

# Move storage for users' sessions to memcached.
$conf['session_inc'] = 'sites/all/modules/memcache_storage/includes/session.inc';
?>

memcached daemon settings

mbopp's picture

Thanks Spleshka, when you refer to configuring things properly to prevent fragmentation, do you just mean the amount of memory you allocate to the daemon? Or are there other settings I should pay attention to?

Mike Bopp

Nice

mbopp's picture

Thank You! for the sample configuration!

Mike Bopp

do you just mean the amount

Spleshka's picture

do you just mean the amount of memory you allocate to the daemon>

No. I mean smart configuration, including some fragmentation settings. See example of my memcached configuration for CentOS:

#! /bin/sh
#
# chkconfig: - 55 45
# description:    The memcached daemon is a network memory cache service.
# processname: memcached
# config: /etc/sysconfig/memcached
# pidfile: /var/run/memcached/memcached.pid

# Standard LSB functions
#. /lib/lsb/init-functions

# Source function library.
. /etc/init.d/functions

USER=memcached
OPTIONS="-a 0766 -c 2048 -f 1.07 -n 512 -L"
RETVAL=0
PROG="memcached"

start_instance() {
    echo -n $"Starting $PROG ($1): "      
    daemon --pidfile /var/run/memcached/memcached.$1.pid memcached -d -s $3 -u $USER -m $2 -P /var/run/memcached/memcached.$1.pid -v 2>> /var/log/memcached/memcached.$1.log $OPTIONS

    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/memcached.$1
}

stop_instance() {
    echo -n $"Stopping $PROG ($1): "
    killproc -p /var/run/memcached/memcached.$1.pid /usr/bin/memcached
    RETVAL=$?
    echo
    if [ $RETVAL -eq 0 ] ; then
        rm -f /var/lock/subsys/memcached.$1
        rm -f /var/run/memcached.$1.pid
    fi
}

start () {
    # insure that /var/run/memcached has proper permissions
    if [ "stat -c %U /var/run/memcached" != "$USER" ]; then
        chown $USER /var/run/memcached
    fi

    # start 3 socket streams for memcached.
    start_instance default 1024 /var/run/memcached/memcached.socket0;
}

stop () {
    stop_instance default;
}

restart () {
    stop
    start
}


# See how we were called.
case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  status)
    status memcached
    ;;
  restart|reload|force-reload)
    restart
    ;;
  condrestart)
    [ -f /var/lock/subsys/memcached ] && restart || :
    ;;
  *)
    echo $"Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
    exit 1
esac

exit $?

And Debian startup daemon

5 days. - Wooow!

chilic's picture

5 days. - Wooow!

Beware memcached extension

fgm's picture

On various systems I have found it be consistently not only slower than the "old" memcache extension, but even slower than a (tuned) MySQL. So the most important advice I would add after this is: "Measure".

Memcached Extension

mbopp's picture

Thanks, I'll be keeping a close eye on performance as i change things. It's good to know that in installing certain processes, things will not automatically be faster.

Mike Bopp

PECL MemcacheD Extension

mbopp's picture

I'm having a heck of a time getting this pecl extension installed on ubuntu 10.04. I know this extenstion relies on the libmemcached library. But the incompatibilities of what versions are provided by aptitude is a real pain. I'd like to benchmark them both, but for now I guess I'll stick with the existing memcache pecl extension. Especially since it is up in the air as to whether the memcacheD extension is really any better.

Anyone have similar issues with the memcached extension? I tried manually building different versions to no avail. I'm sure I can get it figured out, but would rather not sink the time into it now.

Mike Bopp

libmemcached issue

kjala's picture

I had similar issues with installing libmemcached. I went with the memcache pecl(and its working) instead of the memcached extension which needs libmemcached.

I concur

mbopp's picture

Turns out that my production server was using the libmemcached version. (test was already using memcache) Switching production to the PECL memcache version resolved the issue and improved performance a great deal.

Mike Bopp

content cache does not clear

kjala's picture

Not to hijack this thread but on the same line of memcache module, after installing and running the memcache, content cache is not clearing up on publishing content with modification for anonymous users even after the 5min cache time set in Drupal. Any ideas or does anything else to be tweaked or a new module to be installed along with Memcache to deal with cache expiration.

I believe you need to

mbopp's picture

I believe you need to configure your memcache expiration outside of the regular drupal settings. As shown in examples above you can set the 'memcache_storage_wildcard_invalidate' property if you are using the memcache_storage module.

<?php
$conf
['memcache_storage_wildcard_invalidate'] = 60 * 60 * 24 * 5;
?>

for 5 days of cache.

I also believe that cache is not eliminated, but rather invalidated. So the next time the php is requested it will be re-"compiled". I just mention this in case you are seeing that the memory usage remains high.

Mike Bopp

Thanks mbopp. I'm using

kjala's picture

Thanks mbopp. I'm using memcache and on testing it in Dev the usage is around 60%. Due to the frequency of content updates by multiple editors we have set drupal cache expire to 5mins. Is there a similar wildcard invalidate setting for memcache module too?

@kiranjala, use Memcache

Spleshka's picture

@kiranjala, use Memcache Storage module instead of Memcache.

@Spleshka, I have seen your

kjala's picture

@Spleshka, I have seen your posts recently in other places and liked the progress of your project. The memcache namespace has some weight in the team and so I went with that. I would sure consider Memcache Storage if memcache module does not work out well.

I wish the Memcache module page posts links to alternate or similar modules like yours so we have options in the first place.

There is an issue with a

jason.fisher's picture

There is an issue with a specific version of either memcache/memcached being enabled in php.d that is known to cause large slowdowns with memcached access. I would look there..

If you haven't tried xhprof yet, it will give you good insight into where you are getting stuck and whether it is actual CPU time or being blocked by an external source (memcached/mysql).

In my experience, the biggest general performance hit in D7 to large sites (100,000 nodes/2,000 fields) is time spent in unserialize. I have seen several cases where 1/3 of the execution time is spent there, so keep your $forms tidy.

Thanks for recommending

loudpixels's picture

Thanks for recommending memcache storage ... I was using memcache but had many issues with user sessions ... now everything seem to be working great.

Quick question I used the config file you posted in the 3rd post but had to disable the following:

Configure memcached instance.

$conf['memcache_servers'] = array(
'unix:///var/run/memcached/memcached.socket0' => 'default',
);

Configure memcached extenstion.

$conf['memcache_options'] = array(
Memcached::OPT_TCP_NODELAY => TRUE,
Memcached::OPT_NO_BLOCK => TRUE,
Memcached::OPT_BINARY_PROTOCOL => TRUE,
);

Would love to know more about what these settings do in order to get most of my server:

Here are my specs:
Intel® Xeon® Processor E5-2650 w/ 32GB RAM
Centos 6
Drupal 4.71
Memcache Storage: PECL Memcache extension Version: 3.0.8

The website using this is currently doing about 75k views per day with peaks reaching 1000 concurrent users.

Any help improving speed would be appreciated.

Thanks again for the great module and for your help.

Not really sure on those

Jamie Holly's picture

Not really sure on those settings, but I would suggest upgrading Drupal as caching improvements have been made over the years. I handle a site for a client on the same server specs, except only 16gb of RAM, running on Drupal 6. We also use memcache. We got over 150,000 nodes, with 20+ being added per day, a heavy authenticated user base and average over 350,000 page views per day and never have any problems. During one peek time we had over 60,000 page views in one hour and our server load never went above 3.


HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

$conf['memcache_servers'] =

Spleshka's picture

$conf['memcache_servers'] = array(
'unix:///var/run/memcached/memcached.socket0' => 'default',
);

In this setting we switch to use to sockets instead of tcp connection. It's usually 10-30% faster than tcp connection. Although sockets can be used only for local server set up and don't support shared memcached instance for a bunch of servers. You should adjust your memcached config on the server in order to enable sockets. Just google this stuff, the examples will pop in the search results.

$conf['memcache_options'] = array(
Memcached::OPT_TCP_NODELAY => TRUE,
Memcached::OPT_NO_BLOCK => TRUE,
Memcached::OPT_BINARY_PROTOCOL => TRUE,
);

Well, it's sort of "claimed-to-improve-memcached-performance" configs. All these settings work only with pecl memcached. Just google for each of them and you'll get a detailed description for each of those.

One more thing to mention here is even though "Memcached::OPT_NO_BLOCK => TRUE" is proven to give a good performance boost by myself, I've experienced very weird and not reproducable issues when turned this setting on.

Pretty impressive numbers

loudpixels's picture

Pretty impressive numbers Jamie and very re-comforting since our numbers are exponentially growing on a monthly basis.

Sorry I meant to say Drupal 7.41 ... my dyslexia got the better of me :)

Thanks!

Thanks Spleshka for this

loudpixels's picture

Thanks Spleshka for this explanation ... I will google it and see if it's something I can setup.

Another quick weird behavior is that now the "Who's online" block is reporting 0 users althought I currently have 500 active users on the site.

Any thoughts on what could be causing this?

Thanks!

Yeah, "Who's online" block

Spleshka's picture

Yeah, "Who's online" block counts row in database {session} table. There is no way to count how many active sessions using memcached. It's architecture (memcached architecture, not the module's one) doesn't allow any counting operation like that.

I guess I spoke too soon ...

loudpixels's picture

I guess I spoke too soon ... still having problems with user sessions.

If I login via Firefox for instance ... the system ask me for my login credentials ... cool but if I switch to another browser and try to login with the same credentials it refuses and asks me to reset my password ... once I do this and go back to the Firefox, same thing.

Any ideas how to fix this?

I did set this config setting:

$conf['memcache_storage_wildcard_invalidate'] = 180; / 3 minutes

Could this setting have caused this?

Thanks

Could this setting have

Spleshka's picture

Could this setting have caused this?

Nope, please leave this variable to its defaults. It's quite internal and usually should not be changed.

One more thing ... the system

loudpixels's picture

One more thing ... the system reports too many tries from your IP. IP blocked ... request a new password. (even tho it's the first attempt)

Requesting a new password lets me back in.

This is driving me bunkers!

Unlikely this behavior is

Spleshka's picture

Unlikely this behavior is related to memcached. You should search for issue with authentication.

Thanks Spleshka ... that's

loudpixels's picture

Thanks Spleshka ... that's what I was starting to think ... it might be related to varnish :(

I'm going to look around ... thanks again! and great module!!!! Thank you very much!

Yeap ... the problem was with

loudpixels's picture

Yeap ... the problem was with Varnish ... had to disable it for now :(

Thanks for all your help!

High performance

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week