High Ram usage

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
Ruriko's picture

Hi I just switched from apache to nginx so I'm quite new to it. I am running ubuntu 10.04 with 1GB ram I've installed APC & Boost. Currently I'm getting high ram usage and I don't know what's causing it.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  578 www-data  20   0  180m  69m  40m S    0  6.9   0:04.31 php-fpm
  577 www-data  20   0  165m  57m  43m S    0  5.7   0:04.93 php-fpm
  579 www-data  20   0  166m  54m  40m S    0  5.5   0:03.74 php-fpm
  581 www-data  20   0  159m  50m  42m S    5  5.1   0:04.24 php-fpm
  499 mysql     20   0  157m  50m 6448 S    0  5.0   0:13.93 mysqld
  586 www-data  20   0  159m  49m  41m S    0  5.0   0:04.44 php-fpm
  584 www-data  20   0  159m  48m  41m S    0  4.9   0:04.24 php-fpm
  587 www-data  20   0  158m  47m  40m S    0  4.8   0:03.88 php-fpm
  583 www-data  20   0  158m  46m  40m S    0  4.7   0:03.94 php-fpm
  580 www-data  20   0  158m  46m  39m S    0  4.6   0:04.39 php-fpm
  582 www-data  20   0  158m  45m  38m S    0  4.6   0:04.66 php-fpm
  576 root      20   0  155m 4072 1308 S    0  0.4   0:00.06 php-fpm
  685 snmp      20   0  8736 3808 2204 S    0  0.4   0:00.19 snmpd
  940 root      20   0  8792 3172 2480 S    0  0.3   0:00.04 sshd
1048 root      20   0  8792 3156 2492 S    0  0.3   0:00.08 sshd
  943 root      20   0 20584 3084 2208 S    0  0.3   0:00.04 console-kit-dae
  553 www-data  20   0  5644 2160  888 S    0  0.2   0:00.21 nginx
  552 www-data  20   0  5620 2148  888 S    0  0.2   0:00.13 nginx
  555 www-data  20   0  5612 2144  888 S    0  0.2   0:00.16 nginx
  467 root      20   0  5552 2140 1728 S    0  0.2   0:00.01 sshd
  554 www-data  20   0  5624 2120  888 S    0  0.2   0:00.12 nginx
  670 root      20   0  5820 1804 1440 S    0  0.2   0:00.01 master
  678 postfix   20   0  5880 1760 1404 S    0  0.2   0:00.00 qmgr
1090 root      20   0  3084 1732 1364 S    0  0.2   0:00.01 bash
  677 postfix   20   0  5836 1712 1372 S    0  0.2   0:00.01 pickup
    1 root      20   0  2772 1604 1204 S    0  0.2   0:00.78 init
  362 syslog    20   0 34680 1548 1044 S    0  0.2   0:00.03 rsyslogd
  574 ntp       20   0  4412 1376 1036 S    0  0.1   0:00.05 ntpd
1101 root      20   0  2548 1220  928 R    0  0.1   0:01.51 top
  549 root      20   0  5160 1036  292 S    0  0.1   0:00.00 nginx
  229 root      20   0  2320  884  668 S    0  0.1   0:00.03 upstart-udev-br
  397 messageb  20   0  2664  848  624 S    0  0.1   0:00.01 dbus-daemon
1047 root      20   0  2012  840  656 S    0  0.1   0:00.00 sftp-server
  503 root      20   0  2380  800  632 S    0  0.1   0:00.00 cron
  231 root      16  -4  2276  624  324 S    0  0.1   0:00.03 udevd
  480 root      20   0  1792  572  488 S    0  0.1   0:00.00 getty
  471 root      20   0  1792  568  484 S    0  0.1   0:00.00 getty
  472 root      20   0  1792  568  488 S    0  0.1   0:00.00 getty
  486 root      20   0  1792  568  488 S    0  0.1   0:00.00 getty
  729 root      20   0  1792  568  488 S    0  0.1   0:00.00 getty
  731 root      20   0  1792  568  488 S    0  0.1   0:00.00 getty
  479 root      20   0  1792  564  488 S    0  0.1   0:00.00 getty
  270 root      18  -2  2184  440  208 S    0  0.0   0:00.00 udevd
  271 root      18  -2  2184  440  208 S    0  0.0   0:00.00 udevd
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd

This is my php-fpm settings:
pm.max_children = 20
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 10
pm.max_requests = 500

Comments

What is your nginx config?

chilic's picture

What is your nginx config?

It's not Nginx

perusio's picture

but rather php-fpm. Drupal is a I/O bound application so what determines the settings for php-fpm is the available memory and traffic patterns of the site.

Rule of thumb for I/O bound php-fpm:

number_of_children = 1.2 * total_memory / average_space_per_process

This is for a site with a moderate to high traffic. If your site is low traffic you can use the ondemand governor:
pm = ondemand
pm.process_idle_timeout = 60 ; kill child process if no request comes in during this time
pm.max_children = 1

This is will increase the number of workers ondemand. This is a brand new feature, I haven't played with it yet. I'll try to give it a spin this weekend and report back.

What makes you think that's

brianmercer's picture

What makes you think that's high ram usage? What is the output of "free -m"? Are you swapping? And not just if you have some swap memory used, but are you swapping regularly, i.e. swap io. You WANT to "use" all your memory. You just want to use it all in the best way.

Maybe you want that memory as mysql cache or maybe as php child processes, or maybe not tied up in applications at all but rather free for the OS to use as buffer and file cache. Your ps output looks like a pretty good usage to me if you're using 5 or 6 active child processes most of the time. and not emptying your mysql qcache too often.

The number of fpm children should be the number of children that you need. As a starting point you want generally at least as many as CPUs as you have, so maybe 1 or 2 or 4 depending on your computer, plus 2 or 3 more for when a child is waiting on something like a database backend. But that is only a general rule. If your child processes are blocking for long periods of time for something, like your php script is retrieving something offsite, you might want more. With just Drupal accessing a database, you don't need that many extra.

You could install collectd and add a bit of configuration and then have a handy graph of how many child processes are active or idle over a period of time. If you find that the most you're ever using at once is 2 or 3 child processes at a time, and that 7-8 are idle 99% of the time, then maybe reduce to 4 or 5 minimum and 20 maximum. Maximum of course is dictated by available RAM like the formula that perusio stated. But if you're only ever using 2 or 3 at a time, then set the minimum to 5 and free up the rest of the ram for os file cache use. Or increase your qcache by 16MB if you need it. If you get a spike then fpm will spawn those new processes up to the maximum, but the other 99% of the time you'll be serving your static files a bit faster because more of them fit in the file OS cache. Or if you're constantly using 13 active children, then raise your minimum up to 16 if that's what you need.

Stat monitoring will also tell you your swap IO. And can also tell you your qcache efficiency. Or you can check qcache evictions with mysqltuner.pl and tuningprimer.sh. Maybe you want to kill off two of those php-fpm children and use that 25MB for a memcached instance for your variables table and views and ctools cache tables.

If this machine was devoted exclusively to php then you could just set children to use 90% of the memory and be done. But if you're doing mysql and static files from the same machine, you have to allocate between them.

The ondemand option is really for a different use case. If you were hosting 40 different clients and wanted each to have their own pool for security reasons, and their traffic was low enough, then you wouldn't want to spend your RAM by giving each even 1 minimum. You might want to give each 0 minimum. That is the use case for ondemand and it's just being implemented. Until ondemand, fpm hasn't been practical for shared hosts to replace suexec.

For a single fpm pool, it's hard to justify using ondemand. You'd have to have an unusual use case where you wouldn't want to spare the memory for at least 1 or 2 children available at all times.

(No subject)

Ruriko's picture

My site receives roughly 1k+ daily traffic. The vps is xen and has 1GB to swamp. When I had my site on apache I was only using 500mb ram but when on nginx it uses up 1gb
free - m

             total       used       free     shared    buffers     cached
Mem:           999        864        135          0        100        597
-/+ buffers/cache:        166        833
Swap:         1023          0       1023

nginx.conf

user www-data;
worker_processes     4;
error_log  /var/log/nginx/nginx.log info;
pid        /var/run/nginx.pid;

events {
  worker_connections  1024;
  multi_accept on;
}

http {
  include           fastcgi.conf;
  include           mime.types;
  default_type      application/octet-stream;
  set_real_ip_from  127.0.0.1;
  real_ip_header    X-Forwarded-For;
 
  ## Proxy
  proxy_redirect          off;
  proxy_set_header        Host  $host;
  proxy_set_header        X-Real-IP $remote_addr;
  proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
  client_max_body_size    10m;
  client_body_buffer_size 128k;
  proxy_connect_timeout   90;
  proxy_send_timeout      90;
  proxy_read_timeout      90;
  proxy_buffers           32 4k;

  ## Compression
  gzip             on;
  gzip_types       text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;
  gzip_disable     "MSIE [1-6]\.(?!.*SV1)";

  ### TCP options
  tcp_nodelay         on;
  tcp_nopush          on;
  keepalive_timeout   65;
  sendfile            on;

  include /etc/nginx/sites-enabled/*;
 
}

sites nginx.conf

server {
    server_name www.gorgeousanime.com gorgeousanime.com;
    access_log /srv/www/gorgeousanime.com/logs/access.log;
    error_log /srv/www/gorgeousanime.com/logs/error.log;
    root /srv/www/gorgeousanime.com/public_html;

        location ~ \.php$ {
                include fastcgi_params;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_intercept_errors on;
                fastcgi_pass unix:/var/run/php5-fpm.sock;
        }

location ~ /forum
{
        root /srv/www/gorgeousanime.com/public_html/;
        index index.php index.html;

        if (!-e $request_filename)
        {
                # actions
rewrite ^/(activate|admin|announce|ban|boardrecount|buddy|calendar|cleanperms)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(collapse|convertentities|convertutf8|coppa|deletemsg|detailedversion|display|dlattach)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(dumpdb|editpoll|editpoll2|featuresettings|featuresettings2|findmember|help|helpadmin)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(im|jsoption|jsmodify|lock|lockVoting|login|login2|logout)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(maintain|manageattachments|manageboards|managecalendar|managesearch|markasread|membergroups|mergetopics)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(mlist|modifycat|modifykarma|modlog|movetopic|movetopic2|news|notify)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(notifyboard|optimizetables|packageget|packages|permissions|pgdownload|pm|post)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(post2|postsettings|printpage|profile|profile2|quotefast|quickmod|quickmod2)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(recent|regcenter|register|register2|reminder|removetopic2|removeoldtopics2|removepoll)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(repairboards|reporttm|reports|requestmembers|search|search2|sendtopic|serversettings)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(serversettings2|smileys|smstats|spellcheck|splittopics|stats|sticky|theme)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(trackip|about:mozilla|about:unknown|unread|unreadreplies|viewErrorLog|viewmembers|viewprofile)/?$ "/index.php?pretty;action=$1" last;
rewrite ^/(verificationcode|vote|viewquery|who|\.xml)/?$ "/index.php?pretty;action=$1" last;

# boards + topics
rewrite ^/([-_!~*'()$a-zA-Z0-9]+)/[0-9]?/?$ /index.php?pretty%3Bboard=$1.0 last;
rewrite ^/([-_!~*'()$a-zA-Z0-9]+)/([0-9]*)/[0-9]?/?$ /index.php?pretty%3Bboard=$1.$2 last;
rewrite ^/([-_!~*'()$a-zA-Z0-9]+)/([-_!~*'()$a-zA-Z0-9]+)/[0-9]?/?$ /index.php?pretty%3Bboard=$1%3Btopic=$2.0 last;
rewrite ^/([-_!~*'()$a-zA-Z0-9]+)/([-_!~*'()$a-zA-Z0-9]+)/([0-9]*|msg[0-9]*|new)/[0-9]?/?$ /index.php?pretty%3Bboard=$1%3Btopic=$2.$3 last;
        }
}

#######################################################
### nginx.conf catch-all
#######################################################

client_max_body_size 75M;

gzip_static on;
gzip on;
gzip_comp_level 9;
gzip_types application/x-javascript text/css text/plain text/xml application/xml application/xml+rss text/javascript;

location ~ \.(gif|jpg|jpeg|png|tif|bmp|jpe)$ {

valid_referers server_names none blocked ~(pcwintech.com|google.|simpleportforwarding.com);

if ($invalid_referer) {

rewrite ^ http://$host/showimage?file=${uri} permanent;
}

}

location ~ \.(exe|zip|reg|vbs)$ {

valid_referers server_names none blocked ~(pcwintech.com|google.|simpleportforwarding.com);

if ($invalid_referer) {

rewrite ^ http://$host? permanent;
}

}


location = /favicon.ico {
log_not_found off;
access_log off;
}

location = /robots.txt {
allow all;
log_not_found off;
access_log off;
}

# This is mostly based on Drupal's stock .htaccess
location ~* ^.+(\.(txt|engine|inc|info|install|module|profile|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)|code-style\.pl|/Entries.*|/Repository|/Root|/Tag|/Template)$ {
return 404;
}

# serve static files directly
location ~* ^.+\.(jpg|jpeg|gif|png|ico|swf|flv)$ {
access_log off;
expires 30d;
}

# Very rarely should these ever be accessed outside of your lan
location ~* \.(txt|log)$ {
allow 192.168.0.0/16;
deny all;
}

location ~ \..*/.*\.php$ {
return 403;
}

## Deny some crawlers
if ($http_user_agent ~* (HTTrack|HTMLParser|libwww) ) {
return 444;
}
## Deny certain Referers (case insensitive)
if ($http_referer ~* (poker|sex|girl) ) {
return 444;
}
## www. redirect
if ($host = 'pcwintech.com' ) {
rewrite ^/(.*)$ http://www.pcwintech.com/$1 permanent;
}


##
## required only when using purl, spaces & og for modules: ajax_comments, watcher and fasttoggle
## the /og path should be modified to match your default for og/purl URL for organic groups
##
location ~* ^/og {
rewrite ^/og\-(.*)/ajax_comments/(.*)$ /index.php?q=ajax_comments/$2 last;
rewrite ^/og\-(.*)/context/ajax-block-view$ /index.php?q=context/ajax-block-view last;
rewrite ^/og\-(.*)/comment/reply/(.*)\?reload=1$ /index.php?q=comment/reply/$2&reload=1 last;
rewrite ^/og\-(.*)/node/([0-9]+)/toggle/(.*)$ /index.php?q=node/$2/toggle/$3 last;
rewrite ^/og\-(.*)/node/([0-9]+)/edit\?(.*)$ /index.php?q=node/$2/edit?$3 last;
rewrite ^/og\-(.*)/user/([0-9]+)/watcher/toggle/(.*)$ /index.php?q=user/$2/watcher/toggle/$3 last;
rewrite ^/(.*)$ /index.php?q=$1 last;
}

## 6.x starts
location / {
deny 1.2.3.4;
deny 83.228.199.228;
deny 194.8.75.141;
deny 84.122.26.246;
deny 194.8.75.0/24;
deny 194.8.74.0/24;
#rewrite ^/(.*)/$ /$1 permanent; # remove trailing slashes - disabled
try_files $uri @cache;
}

location @cache {
if ( $request_method !~ ^(GET|HEAD)$ ) {
return 405;
}
if ($http_cookie ~ "DRUPAL_UID") {
return 405;
}
error_page 405 = @drupal;
add_header Expires "Tue, 24 Jan 1984 08:00:00 GMT";
add_header Cache-Control "must-revalidate, post-check=0, pre-check=0";
add_header X-Header "Boost Citrus 1.9";
charset utf-8;
try_files /cache/normal/$host${uri}_$args.html /cache/$host${uri}_$args.html @drupal;
}

location @drupal {
###
### now simplified to reduce rewrites
###
rewrite ^/(.*)$ /index.php?q=$1 last;
}

location ~* (/\..*|settings\.php$|\.(htaccess|engine|inc|info|install|module|profile|pl|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)$|^(Entries.*|Repository|Root|Tag|Template))$ {
deny all;
}

location ~* /files/.*\.php$ {
return 444;
}
location ~* /themes/.*\.php$ {
return 444;
}

location ~ \.css$ {
if ( $request_method !~ ^(GET|HEAD)$ ) {
return 405;
}
if ($http_cookie ~ "DRUPAL_UID") {
return 405;
}
error_page 405 = @uncached;
access_log off;
expires max; #if using aggregator
add_header X-Header "Boost Citrus 2.1";
try_files /cache/perm/$host${uri}_.css /cache/$host${uri}_.css $uri =404;
}

location ~ \.js$ {
if ( $request_method !~ ^(GET|HEAD)$ ) {
return 405;
}
if ($http_cookie ~ "DRUPAL_UID") {
return 405;
}
error_page 405 = @uncached;
access_log off;
expires max; # if using aggregator
add_header X-Header "Boost Citrus 2.2";
try_files /cache/perm/$host${uri}_.js /cache/$host${uri}_.js $uri =404;
}

location ~ \.json$ {
if ( $request_method !~ ^(GET|HEAD)$ ) {
return 405;
}
if ($http_cookie ~ "DRUPAL_UID") {
return 405;
}
error_page 405 = @uncached;
access_log off;
expires max; # if using aggregator
add_header X-Header "Boost Citrus 2.3";
try_files /cache/normal/$host${uri}_.json /cache/$host${uri}_.json $uri =404;
}

location @uncached {
access_log off;
expires max; # max if using aggregator, otherwise sane expire time
}

location ~* /files/imagecache/ {
access_log off;
try_files $uri @drupal; #imagecache support - now it works
}

location ~* ^.+\.(jpg|jpeg|gif|png|ico)$ {
access_log off;
expires 30d;
try_files $uri =404;
}

location ~* \.xml$ {
if ( $request_method !~ ^(GET|HEAD)$ ) {
return 405;
}
if ($http_cookie ~ "DRUPAL_UID") {
return 405;
}
error_page 405 = @drupal;
add_header Expires "Tue, 24 Jan 1984 08:00:00 GMT";
add_header Cache-Control "must-revalidate, post-check=0, pre-check=0";
add_header X-Header "Boost Citrus 2.4";
charset utf-8;
types { }
default_type application/rss+xml;
try_files /cache/normal/$host${uri}_.xml /cache/normal/$host${uri}_.html /cache/$host${uri}_.xml $uri @drupal;
}

location ~* /feed$ {
if ( $request_method !~ ^(GET|HEAD)$ ) {
return 405;
}
if ($http_cookie ~ "DRUPAL_UID") {
return 405;
}
error_page 405 = @drupal;
add_header Expires "Tue, 24 Jan 1984 08:00:00 GMT";
add_header Cache-Control "must-revalidate, post-check=0, pre-check=0";
add_header X-Header "Boost Citrus 2.5";
charset utf-8;
types { }
default_type application/rss+xml;
try_files /cache/normal/$host${uri}_.xml /cache/normal/$host${uri}_.html /cache/$host${uri}_.xml $uri @drupal;
}

#######################################################
### nginx.conf catch-all
#######################################################
# imagecache needs to have php read any files that it's planning to manipulate
location ^~ /sites/default/files/imagecache/ {
     index  index.php index.html;
     # assume a clean URL is requested, and rewrite to index.php                                                                
      if (!-e $request_filename) {
          rewrite  ^/(.*)$  /index.php?q=$1  last;
          break;
      }
}

# serve static files directly
location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico)$ {
    access_log        off;
    expires           30d;
}
}

Looks to me like you are only

wipeout_dude's picture

Looks to me like you are only using 166MB of memory and no swap which I would say is pretty good.. The rest is buffers and cache which is fine..

For your php-fpm pool config you can try these settings as a starting point for low memory usage and let php manage the processes..

pm = dynamic
pm.max_children = 10
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 2
pm.max_requests = 500

Then enable the php-fpm status interface and check it after a couple of days to see if the "max children reached:" value is incrementing to quickly..

So I read my ram usage the

Ruriko's picture

So I read my ram usage the opposite?

             total       used       free     shared    buffers     cached
Mem:           999        864        135          0        100        597
-/+ buffers/cache:        166        833
Swap:         1023          0       1023

cause I thought it used 864 instead of 166 ram

The 864 is INCLUDING buffers

wipeout_dude's picture

The 864 is INCLUDING buffers and cache..

So from those numbers you aren't actually using all of your memory at this stage which means you have spare RAM that you could allocate to optimising MySQL buffer sizes or query cache (use tuning-primer or mysqltuner for some guidance, and some common sense obviously).. You could look at allocating more apc cache if needed (use apc.php to see how its running) or even look at using some nginx caching..

Of course I would say leave your server running for a few days to see where these numbers settle after a standard loading..

Also if you are using a version 3.0 linux kernel you may find the server using a little swap.. this isn't a problem and doesn't mean you are running out of memory.. Its simple moving tjings that don't need to be in RAM to the hard drive to make more memory available for applications and files that will make better use of it.. Of course if you don't like the thought it is a configurable parameter..

Well

perusio's picture

your config is pretty unNginxy. The -e operator is to be used only and only if you're really testing for a file presence. Imagine a site where the presence of a maintenaice.html file should triger the server to return 503.

There are so many things that could be improved that would be a gigantic post. Just a few notes.

This is not the way to do a server level redirect. With an if. Nginx is not Apache that uses a Yoda style of logic. Create another server block that just does the redirect.

server {
    server_name pcwintech.com;
    return 301 http://www.pcwintech.com;
}

If you want to block HTTP methods do it at the server level:

## This needs to be at the http level.
map $request_method $not_allowed_method {
    default 1;
    GET 0;
    HEAD 0;
    POST 0;
}

and

## At the server level.
if ($not_allowed_method) {
    return 405;
}

The rewrites regexes could be much improved and in fact there's no need for any rewrite.

Also

perusio's picture

that's not the way to block unauthorized PHP scripts execution.

Enumerate all PHP files that are allowed and add at the bottom of your config this:

location ~ \.php$ {
    return 404;
}

Yep. Memory looks good. Boost

brianmercer's picture

Yep. Memory looks good. Boost working nicely.

You might want more than 1 minimum fpm child depending on how many of those visitors are logging into the site or the forums and are not just hitting the Boost pages. If they hit the Boost page then it doesn't involve php at all, so it's only the logged in users that will affect that.

The front page has an image on this subdomain that is 404ing: ns1.gorgeousanime.com

Usually ns1 is a dns nameserver, so dunno why that'd be in there.

And remember, we're all just guessing until you get some actual stats.