Best Caching Option for Site Growing from Anonymous 1m to Authenticated 10m users?

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
kandrupaler's picture

Hi High Performance Gurus! Need your help since I'm kinda stuck...

I admin a Drupal 7 site on which the authenticated user base is growing rapidly.

We get about 1 million pageviews per month, and we have a target to grow to 10 million in a year and most of those hits will come from authenticated users. We're building several user-specific features such as "favorite posts", "read later", "posts from people i follow", etc. etc.

He's how we're caching things right now:

(1) Boost (for anonymous users)
(2) Memcache (as db cache)
(3) Opcache (built into PHP)
(4) Authcache (for authenticated users)
(5) Drupal core page caching for anonymous users - turned off (required for both Boost and Authcache)
(6) Block caching - turned off
(7) Views caching - turned off

Question #1: Am I doing the right thing in #6, and #7? As far as I understand, these are not useful because we're caching the whole page using either Boost or Authcache. Why take the trouble of going into each view and turning the cache on?

Question #2: Is Authcache the right choice? I'm facing several problems working with authcache as we add on user-specific features. Even the module page says "This module is probably not suitable for Facebook-type sites where each page contains heavy user-centric content." Okay, we're not becoming another Facebook, but we're going to have a good deal of user-centric content in future. What's my best option if it's not Authcache?

Question #3: Varnish? We're testing out Varnish right now and we'll most likely deploy it. However, there's a ton of customization required for authenticated users. Is it worth the effort or should we continue to use Boost and handle authenticated users using Authcache or the other option you suggest (:-)?

In case it helps, our hardware is a VPS with 2.5 GHz 2-core Intel processor and 4 GB RAM. We're running Centos 6 and the usual LAMP stack.

Comments

I would separate out your

Greg Boggs's picture

I would separate out your servers. At a minimum, I'd use 1 web server and 1 database server.

For software on the server, I'd also use PHP7. And, I'd make sure you have a CDN because serving up images will crash your web server. I'd consider using Nginx in place of Apache, and MariaDB in place of the hobby/community version of mySQL.

Next, I'd make sure to disable dblog, and I would use block cache and views cache on everything you can. You'll want to use Views Content Cache: https://www.drupal.org/project/views_content_cache to expire views caches. The advantage here is that a block can be cached correctly across thousands of pages even if the auth cache is expired.

You are correct in assuming that auth cache will be a big investment to be able to do dynamic, per user content. But, it's certainly powerful.

Thanks a lot for your inputs,

kandrupaler's picture

Thanks a lot for your inputs, Greg. PHP7 + Nginx + MariaDB will be a huge change from what we have, but we'll look into this.

dblog is disabled, but we have syslog. Does that kill? I'm not seeing any impact, to be frank. Also, the statistics module is disabled too. We just use GA.

Regarding block cache and views cache - does it make sense when the page on which the block or view is rendered is fully cached by authcache?

What's your take on whether or not to use Varnish in place of Boost?

Auth cache is per page. But,

Greg Boggs's picture

Auth cache is per page. But, blocks are used across pages. So, a single uncached auth cache page build will likely be built up from many pre-cached blocks and views.

I haven't had a lot of success with Varnish for logged in users. But, ESI+hash cache could work in theory.

Improving your software stack will have a big impact on the performance of your application.

Thanks for the explanation, Greg.

kandrupaler's picture

Thanks for explaining how views and block caching helps! Till now I wasn't sure if it helps at all...

BTW drupal.org seems to be using only Varnish and Memcache even though there are million(s) of authenticated users. Any idea how they can manage? Money spent on hardware?

I don't know the specifics of

Greg Boggs's picture

I don't know the specifics of Drupal.org, but they certainly have a collection of servers, and they offload downloads and static files to Fastly CDN.

Q1: No, even with pages being

Beanjammin's picture

Q1: No, even with pages being cached by boost you want to turn on views and block caching.
Q2: This depends... What problems are you running into with authcache? How much user specific content will there be? Will it appear on all pages or a smaller subset (like a dashboard)? Are you using memcache as your backend or something else?
Q3: If your traffic continues to grow as you expect, you will, at some point, need more than one web server. At that point you'll need a proxy sitting in front of them. Varnish is a good option, especially if your site doesn't use SSL, however nginx should also be considered.

Other things to think about would include whether your server is properly tuned to balance the RAM requirements of apache/PHP, mysql, memcache, opcache, and the OS? Mysql tuning can make a huge performance difference. Traffic patterns are also important to consider. If you have a few pages that get most of the traffic then page caching is great, but if your traffic is spread over a large number of different pages then it's less effective. Have you load tested your site to approximate the load from 10M page views using typical traffic patterns? If so, did your server become CPU or RAM bound (or I/O)?

kandrupaler's picture

Hi Beanjammin,

Q1 - Yes, that's a great learning for me on this thread. Thanks a lot to you and @Greg Boggs for making me realize this fundamental fact.

Q2 - After I started this discussion thread I went back and tried to think clearly about my problems with authcache. I'm quite relieved after realizing that it's just one problem: Getting Ajax Comments to work with Authcache. But my fear comes from the unknown... :-)

Q3 - Right now I'm being very stingy and trying to extract the most out of our existing server. But truth be told, I'm a noob when it comes to all things server, so I don't even know how to do load testing. I've heard of things like Xhprof, but haven't had the bandwidth to learn. Any quick video/documentation pointers? Also, can I load test on the production site? I don't have another setup with the same h/w and s/w stack.

One thing I know for sure is that we were being memory limited before we upgraded our VPS. I used to get tons of email about services getting stopped because of memory issues. Once we upgraded and added more RAM, I'm able to sleep at night :-)

We're a cash-starved startup and I'm the only software guy. When I'm not writing software, I'm writing and editing (let's say half of) the content that's bringing in the 1M pageviews right now!

Some feedback / general

btopro's picture

Some feedback / general things to look into based on our experience tuning auth'ed Drupal in a crazy project / scope of needs. Squeaking performance out of the stack is definitely the first place to start.

PHP
Opcache - https://github.com/elmsln/elmsln/blob/master/scripts/server/opcache.txt
PHP.ini - https://github.com/elmsln/elmsln/blob/master/scripts/server/php.txt

MySQL
Sane my.cnf additions -- https://github.com/elmsln/elmsln/blob/master/scripts/server/my.txt

Apache
add to conf to deliver assets better -- https://github.com/elmsln/elmsln/blob/master/scripts/server/zzz_performa...

Drupal modules (beyond authcache as mentioned):
mysql optimization recommendations / eliminates DB deadlocks -- https://www.drupal.org/project/apdqc
cache path references -- https://www.drupal.org/project/pathcache
Block cache alter -- https://www.drupal.org/project/blockcache_alter
Advagg to compress / modify / deliver CSS/JS faster -- https://www.drupal.org/project/advagg
Mentioned earlier, views content cache to only update on content being changed -- https://www.drupal.org/project/views_content_cache
Request things asynchronously, useful developer library, works w/ advagg -- https://www.drupal.org/project/httprl

apdqc is a must have in my mind and not brought up previously in this thread. When touching the database it should be as fast as possible.

Some feedback / general

btopro's picture

Some feedback / general things to look into based on our experience tuning auth'ed Drupal in a crazy project / scope of needs. Squeaking performance out of the stack is definitely the first place to start.

PHP
Opcache - https://github.com/elmsln/elmsln/blob/master/scripts/server/opcache.txt
PHP.ini - https://github.com/elmsln/elmsln/blob/master/scripts/server/php.txt

MySQL
Sane my.cnf additions -- https://github.com/elmsln/elmsln/blob/master/scripts/server/my.txt

Apache
add to conf to deliver assets better -- https://github.com/elmsln/elmsln/blob/master/scripts/server/zzz_performa...

Drupal modules (beyond authcache as mentioned):
mysql optimization recommendations / eliminates DB deadlocks -- https://www.drupal.org/project/apdqc
cache path references -- https://www.drupal.org/project/pathcache
Block cache alter -- https://www.drupal.org/project/blockcache_alter
Advagg to compress / modify / deliver CSS/JS faster -- https://www.drupal.org/project/advagg
Mentioned earlier, views content cache to only update on content being changed -- https://www.drupal.org/project/views_content_cache
Request things asynchronously, useful developer library, works w/ advagg -- https://www.drupal.org/project/httprl

apdqc is a must have in my mind and not brought up previously in this thread. When touching the database it should be as fast as possible.

Thanks a lot! One quick

kandrupaler's picture

Thanks a lot!

One quick question: how did you arrive at 256 MB for PHP? That's what I'm using, too.

I've looked into all the modules you've mentioned but I'm not using any of those right now.

I'm afraid I don't have

Beanjammin's picture

I'm afraid I don't have experience with Authcache and Ajax Comments so can't help you there. That said, asking a question on a closed issue probably won't get you an answer, I'd suggest opening a new issue.

Server tuning can be very involved and what works best will vary from site to site. Big thick books have been written on the subject... That said, here are some quick things to look at:

Use the Devel module to find out how much RAM PHP is using per page load. It'll vary from page to page. You want a rough average, rounding up.

Use a script like https://github.com/RootService/tuning-primer to evaluate your mysql configuration, it'll suggest changes that are noob friendly and let you know how much RAM mysql will consume based on your configuration.

Get the total file size of all your PHP files, be sure to include .inc files and exclude images, CSS, JS, etc. This will tell you how much memory your opcache needs. Round it up a bit, its important all your PHP files fit in the opcache or it'll actually slow down performance.

Memcache (the pecl module not the drupal module) comes with a bundled php script called memcache.php that will give you information about how much RAM memcache is consuming. If you can't find a copy on your server you can get it from the source here http://pecl.php.net/package/memcache. You'll want to make sure memcache isn't consuming all of the RAM you have allotted while at the same time not wasting memory by giving it too much.

Once you have adjusted mysql, memcache, and opcache to be using the right amount of RAM based on the steps above, take the configured amount of RAM available to them and total it up. Now add at least 128MB for the OS and file system cache. Subtract the total from the 4096MB of RAM your server has, this is how much RAM you have left over for apache and PHP. Take that number and divide it by the average amount of RAM used per page load (taken from the Devel module step above) and round the result down. This is how many apache processes you can run given the available RAM, set the MaxClients config in apache to this number. Now in your mysql config set the max_connections value to this same number + 5. You now have a "quick and dirtily" tuned server.

For load testing, no you probably don't want to do that on your production server. I would recommend cloning your VPS and using that. There are cloud services that you can use for load testing or you can use something like JMeter (http://jmeter.apache.org/). Whatever you use, you'll want to make sure the traffic you generate for testing is approximately the same as the traffic your site typically gets, including the percentage of anonymous vs logged in users and what they are doing on your site.

Yes, I did create a new issue

kandrupaler's picture

Yes, I did create a new issue and I'm getting good support on the authcache + ajax_comments issue. I'm pretty close to the solution.

Thanks a ton for educating me on optimizing server config. My approach till now has been to wait for something to break before adjusting things.

I also need to use devel seriously...

BTW, how do you calculate RAM allocation for PHP? We started getting issues with the default 64 MB, so I moved it to 256 MB. Everything's fine but maybe not ideal...

Opcache and Memcache RAM allocations are right now at 25% and 50% usage, so I think there may be place to optimize even there.

Load testing... yes, I need to do it...

good question

rajibmp's picture

BTW, how do you calculate RAM allocation for PHP?

Actually this is very good question with no definitive answer for it.

Beanjammin already mentioned here:

Take that number and divide it by the average amount of RAM used per page load (taken from the Devel module step above) and round the result down.

when you use Devel module, turn on the display memory and it shows memory used and peak memory for the page load. eg:
Memory used at: devel_boot()=2.03 MB, devel_shutdown()=41.5 MB, PHP peak=44 MB.

That's one approach and it also depends on which module you are using. Are you using apache's mod php or fastcgi like php-fpm?

In case of mod php, I'd multiply the peak memory usage by 10 times as there are concurrency issues which affects the memory usage too.

so in above case my php memory would be 512MB (44 x 10 + some safe margin)

while using FPM, there is Process Manager (pm) which spins processes. There are tools (ps) to measure the consumption of each processes, lets say one processes use 20Mb of memory and you have pm.max_children of 20 then 20mb x 20 child processes = 400Mb i.e. 512 Mb with some margin for error as for large page I've seen some processes use up to 52Mb before being killed. How to limit memory for each child process is yet not possible in php.

Thanks rajibmp! So PHP memory

kandrupaler's picture

Thanks rajibmp! So PHP memory utilization and Opcache limit are one and the same?

The aren't quite the same.

Beanjammin's picture

The aren't quite the same. When PHP runs it first interprets the PHP script and turns it into opcode, this is what the opcache holds and is why it's size is based on the filesize of all your php scripts, after the opcode has been created it is run and the memory consumed while the opcode is actually running is the PHP memory utilization (all the objects, variables, etc that are created during the page load)

The idea behind opcache is that it removes the script interpretation step from each page load.

Question #1) If it's

Lee-'s picture

Question #1) If it's cachable, then cache it in a way that makes sense. You could get in to quite complex cache invalidation rules, which would be unique to your site architecture. The end goal is to cache things as long as possible, but have proper invalidation logic so that stale data doesn't get served beyond a reasonable time depending on the types of content and so forth.

Questions #2 and #3) Use ESI (you don't have to use Varnish as there are other alternatives, but Varnish is probably most known). We also use ESI on our sites to deal with authenticated user caching and have a number of elements that are specific to an individual user and to users' roles. It works great once configured properly. It allows you to serve entirely cached pages, even if they contain user specific data. For example, if you have "Hello, kandrupaler" in a block on your page, this is specific to a user. If this block shows up on more than 1 page, then the first time you go to a page, this block will be a cache miss (but the rest of the page will be a cache hit, so only this block needs to be generated). Then on subsequent page requests, provided the page itself is already cached (because another user visited it), and your user specific block was cached from your previous page visit, then the entire new page request can be served without running Drupal at all.

ESI should be the go to method of dealing with authenticated user caching, especially when it comes to serving customized elements on a per user or per role basis. Don't bother with boost, let Varnish or whatever you're using to deal with ESI also take care of your anonymous user caching.

You should be using a CDN and if you don't want to deal with running your own Varnish (or similar) server, you can use a CDN provider that can serve your static content as well as deal with ESI processing.

What proper Varnish + ESI support requires is properly controlling cache related headers (have Varnish respect the headers output by Drupal for the most part), and having ways of purging cache at the Varnish (and Drupal) side when associated actions take place.

Varnish with anonymous users is very easy to set up. ESI is a little more work and it can certainly seem daunting at first, but once you understand how it functions, you'll see it's not nearly as complicated as it may seem.

There are alternatives to Varnish (we're currently using Varnish, but I'm considering switching because of our architecture I have to do some clunky things to make it all play well together).

In summary
1) cache everything for as long as possible, but make sure you have proper cache invalidation logic
2) use a CDN
3) ditch boost + authcache in favor of Varnish+ESI (or an alternative to Varnish that supports ESI or a CDN provider that supports ESI)

End of stream of consciousness. Good luck!

Amazing insights, thanks a lot!

kandrupaler's picture

I'm glad I've found someone who advocates for Varnish for authenticated users. Do you use the varnish module (https://www.drupal.org/project/varnish)? What about esi (https://www.drupal.org/project/esi)?

I have Varnish working for anonymous users on my localhost. I thought of using Authcache with Varnish backend but it started getting too complicated. I knew I could just use Varnish + ESI but didn't venture into it.

Right now Authcache + Boost is what we have. My plan was to slowly replace Boost with Varnish for anonymous users and pass on all authenticated traffic to Authcache with Varnish backend. But yes, if we don't have to use Authcache at all, I'll prefer that.

But as of now, I shudder to even think of those three letters: VCL! Is there a good book or video which can bring a noob up to speed on Varnish?

A few years ago I tried the

Lee-'s picture

A few years ago I tried the authcache module, but couldn't get it to work right. I ended up using Varnish and ESI to achieve my goals. This was on a D6 site and perhaps it would have worked out on a D7 site, but I have no experience with authcache on D7.

The varnish and esi modules will get you off to a good start. You may want to throw in the expire module if it will help to meet the cache expiration needs of your site. You'll probably need to write some custom expiration logic as well, but this may save you some time when getting started.

When it comes to how the data for the esi fragment is generated, the esi module basically uses a path prefix of /esi/ and executes the related block. There can be some pitfalls with this. Some functions won't work as expected if called within such a block. Additionally, it requires a drupal bootstrap. Open your mind when it comes to esi processing -- those esi paths can be whatever you want and handled by any means you want on the backend (doesn't even have to point to Drupal). ;)

As far as VCL, my best recommendation is to stare at this flow chart while reading the VCL documentation:
https://www.varnish-cache.org/docs/4.1/reference/states.html

The most important pieces being vcl_recv, vcl_hash, vcl_backend_fetch, vcl_deliver. Read the documentation on these and stare at that flow chart until you fully grasp it. I also added debugging code in to a header, so I could print out the flow of my code and various values so I made sure I understood what was happening and what those variables contained.

If you need to do anything more advanced than a minimal configuration using the varnish and esi modules, you'll need to know VCL anyway. It will also help you to understand how to utilize Varnish and ESI more effectively in your site, add special cases where you may want to override a default behavior, and so on. Not trying to scare you away. The basic varnish + esi module will get you a huge improvement, but to take it to the next level will require you to learn VCL. While VCL may be intimidating at first, it's really not bad at all.

I guess there's been a lot of

kandrupaler's picture

I guess there's been a lot of improvements to Authcache in D7...

I've got Authcache to work, but I had to be very careful to avoid data leaks from one auth user to another.

I have this feeling, however, that Authcache requires every piece of custom content to be in a block. That makes life rather difficult.

Am I right in understanding that with Varnish + ESI, you can just add the <esi:....> tag anywhere that that stuff will get delivered via ESI? That is powerful.

Yes, you can add a tag

Lee-'s picture

Yes, you can add a tag wherever you like with any path you like. The path used is where Varnish makes the request to get the data, which doesn't have to even be a path to Drupal -- ie you could write some script in ruby to generate the data if you wanted.

That said, if you want to do it the easy way with the esi module, then it would be similar to authcache -- you have to put your personalized data in to either a block, pane, or context and then change the esi cache settings for those blocks/panes/contexts that contain personalized data.

What the esi module is doing behind the scenes is replacing those blocks/panes/contexts with esi tags and setting up the /esi/ URLs and appropriate cache values, but you can manually add esi tags wherever you want and write your own scripts to generate that content. I suspect the authcache module does similar with respect to the tag replacement and paths for the ajax requests.

Something to consider if your

Beanjammin's picture

Something to consider if your site uses SSL, or may at some point in the future, is that Varnish doesn't terminate SSL. The usual work around to this is to put nginx in front of Varnish,have it terminate SSL and then pass the request on to Varnish. This works really well, and may be what you want to do, but adds a fair bit of complexity. If you have plans to use SSL consider using nginx instead of Apache and Varnish. nginx supports SSI, which the Drupal ESI module also supports, that way you get SSL termination, caching, and web server all in one piece.

Thanks for this. Indeed, with

kandrupaler's picture

Thanks for this. Indeed, with a growing number of authenticated users, SSL is on the cards. I like the idea of keeping the software stack simple.

High performance

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week