Nginx, Drupal and Xsend Support

brianmercer's picture

I've been looking at a project where I'm going to use "private files" to make sure that only certain people can download certain files. I'd heard that this method was slower, since the web server can't serve the file directly. Instead it has to go through php.

X-Sendfile support is intended to speed up private file transfers. Your backend (drupal through php-cgi in our case) sends a special header to the web server to transfer a file directly once the backend has authenticated the user and request. For Apache there is an x-sendfile module, and lighty also supports this feature. There is a fairly new Drupal module that implements the feature by sending the proper header. http://drupal.org/project/xsend

I had read that nginx supports the feature, though it renames the header. I decided to take a look at the module to see if it could be adapted to nginx use.

Luckily it's a very small module and there wasn't much for me to screw up. First I went about disabling the checks for existence of the Apache module in xsend.install:

/**
function xsend_install() {
  if(!in_array('mod_xsendfile', apache_get_modules())) {
    drupal_set_message('X-send file module for apache2 is not installed in your system. See the readme for instructions.', 'error');
  }
  else {
    drupal_set_message('X-send file module is installed in your system. Follow the instructions carefully to success rest of isntalling.');
  }
}
*/

and also in xsend.module:
//  if(in_array('mod_xsendfile', apache_get_modules())) {
//   drupal_set_message('X-send file module for apache2 is not installed in your system. Untill you correctly install it do not enable this$
//  }
//  else {
    $form['settings_general'] = array(
        '#type' => 'fieldset',
        '#title' => 'Fast private file transfer settings',
        '#collapsible' => TRUE,
    );
    $form['settings_general']['file_transfer_handler'] = array(
        '#type' => 'radios',
        '#title' => 'Xsend private file transfer support',
        '#default_value' => variable_get('file_transfer_handler', "file_transfer_default"),
        '#options' => array("file_transfer_default"=>t('Disabled'), "file_transfer_xsendfile"=>t('Enabled - files are transferred by x-send$
        '#description' => 'Enable/Disable X-send file transfer support.'
    );
    return system_settings_form($form);
//  }

Then I changed the header that it was sending:
    if (count($headers)) {
//      drupal_set_header('X-Sendfile: '. realpath(file_create_path($filepath)));
      drupal_set_header('X-Accel-Redirect: /protected/'.$filepath);
      foreach ($headers as $header) {

to send the X-Accel-Redirect header instead of X-Sendfile and to change the path from dynamically created /system/files/ to my hardcoded chosen path of /protected/.

Then I added the following to my nginx configuration file for the given server

  location /protected/ {
    internal;
    alias /var/www/dev.secure.example.com/private/;
    error_page 404 /index.php?q=protected;
  }

The "internal" directive keeps that location from being accessed directly. It works only on an internal redirect. If someone goes to /protected/filename.pdf directly, they get a 404. The error page redirects the page back to drupal to simulate a regular 404 since they'd get just a plain 404 with my current setup. If you have something in the server scope like "error_page 404 @drupal" you wouldn't need that part. Also note that in my setup the document root is /var/www/dev.secure.example.com/public/ so the /private/ directory is not in the document root and cannot be accessed directly with /private/filename.pdf since that folder doesn't exist in the document root.

Then I increased some file size limits. In /etc/nginx/nginx.conf:

    client_max_body_size      500m;

and in /etc/php5/cgi/php.ini:
    post_max_size = 500M
    upload_max_filesize = 500M

and also the "Default maximum file size per upload" to 500 at admin/settings/uploads. I suppose you don't need to change all those upload limits if you're just going to upload large files through ftp or something and let users download them. If you're going to let users (or yourself) upload huge files through Drupal, then you need to increase those.

Finally I went to admin/settings/file-system/x-send to enable the module. It is under the File Settings menu item so not shown on the normal two level admin page unless you promote it manually. Took me a minute to find it.

And what was the result? Well I tested it on my local home network which is 10mbps and I uploaded and then downloaded a 200MB file. Locally it was pretty fast anyways. However, I noticed that with the xsend module disabled it started out slowly and then worked its way up to max speed. With xsend enabled it went to the max speed almost instantly.

But more interesting was what I saw when I watched top on the server while downloading with the xsend module enabled and disabled.

With xsend disabled, an nginx process used up a large amount of cpu processing with a small amount of one php-cgi process while sending the file. The nginx process spiked to 99% and hovered between 40 and 70% while the php-cgi process was at about 6%.

With the xsend module enabled, the nginx process used 3% of cpu while sending the file and no php-cgi load at all.

I'll link this thread back to the xsend issue queue and see what the maintainer thinks. Has anyone here tried using the nginx upload progress module for pretty bars while uploading?

Comments

I found this discussion on

brianmercer's picture

I found this discussion on the filefield issue queue regarding upload progress and noted why APC and PECL uploadprogress aren't going to be of any help. http://drupal.org/node/437254#comment-2306892

It looks to me like we'd need to recompile nginx with the uploadprogress add-on module and then someone with some javascript skill will need to adapt the core progress.js to call the location specified by the nginx uploadprogress module for the status updates.

Quite good explain on xsend

heshan.lk's picture

Quite good explain on xsend and Nginx, we can use this in xsend module

Senior Drupal Developer at DrupalConnect

This was one piece I was

mike503's picture

This was one piece I was going to look into when I got into private file serving under nginx, to make sure that Drupal is passing the file off to the webserver after it's done with the PHP work. Sounds like you may have done all the legwork!

Great explanation! Will try

404's picture

Great explanation! Will try it out!

I follow brianmercer steps

spacereactor's picture

I follow brianmercer steps and it work great with doc, pdf, txt, all the doc file is in protected folder and required drupal role to access but it doesn't work on media files, example mp4, mov, avi,png, gif and jpg and etc. it just link me to 404 page if i click on media related files. Did i miss out anything, I really need help here.

Post your config. And

mike503's picture

Post your config.

And anything else relevant. On my tests it worked just fine with images. Although I wound up hacking my own version.

I really should see if I can help out. I wanted to repurpose this same module but make it recognize nginx vs. apache inside of the same module, instead of starting a new module.

server { server_name

spacereactor's picture

server {
server_name www.domainname;
rewrite ^ $scheme://domainname$request_uri permanent;
}

server {
server_name domainname;
root /var/www/domainname/public_html;
access_log /var/log/nginx/domainname.access.log;
error_log /var/log/nginx/domainname.error.log notice;

location = /favicon.ico {
log_not_found off;
access_log off;
}

location @rewrite
rewrite ^/(.*)$ /index.php?q=$1 last;
}

location @uncached {
access_log off;
expires 45d;
}

location /protected/ {
internal;
alias /var/www/domainname/private/;
error_page 404 /index.php?q=protected;
}

location ^./files/.+.(txt|pdf|doc|zip|rar|7z|mpeg|avi|mov|flv|mp4)$ {}
location ~
^.+.(jpg|jpeg|gif|png|ico)$ {
access_log off;
expires 30d;
try_files $uri =404;
}

location = /robots.txt {}

location ~ (.)/x-progress-id:(\w) {
rewrite ^(.)/x-progress-id:(\w) $1?X-Progress-ID=$2;
}
location ^~ /progress {
report_uploads uploads;
}

location ~ .php$ {
fastcgi_pass php;
track_uploads uploads 30s;
}
}

this my nginx config user

spacereactor's picture

this my nginx config

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
}

http {
include fastcgi.conf;
include mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

limit_zone                   conns $binary_remote_addr 1m;
limit_req_zone               $binary_remote_addr zone=reqs:10m rate=12r/s;
upload_progress uploads      1m;
ignore_invalid_headers       on;
recursive_error_pages        on;

set_real_ip_from 127.0.0.1;
real_ip_header X-Forwarded-For;

sendfile                     on;
server_name_in_redirect      off;
server_tokens                off;
autoindex                    off;

client_body_buffer_size      16k;
client_header_buffer_size    4k;
client_max_body_size         5m;
large_client_header_buffers  4 16k;

keepalive_timeout            5 5;
client_body_timeout          15;
client_header_timeout        15;
send_timeout                 15;

tcp_nodelay                  on;
tcp_nopush                   on;

gzip_static                  on;
gzip                         on;
gzip_buffers                 16 8k;
gzip_comp_level              6;
gzip_http_version            1.0;
gzip_min_length              0;
gzip_types                   text/plain text/css image/x-icon i$
gzip_vary                    on;
gzip_disable                 "MSIE [1-6]\.(?!.*SV1)";
gzip_proxied                 any;

log_format        main '"$http_x_forwarded_for" $host [$time_local] '
                     '"$request" $status $body_bytes_sent '
                     '$request_length $bytes_sent "$http_referer" '
                     '"$http_user_agent" $request_time "$gzip_ratio"';

index    index.php index.html index.htm;

upstream php {
server 127.0.0.1:9000;
}

include /etc/nginx/sites-enabled/*;

}

I'm thinking you should turn

mike503's picture

I'm thinking you should turn on nginx debug logging - and see what happens.

That config is a mess - looks like you are using an upload progress module too, which might mix things up a little bit.

An empty location for the files inside of /files seems a bit unnecessary to me.
I don't see anything passing to @rewrite (?)
There are conflicts, or at least too much noise for me to concentrate on what is going on.

Anyway - try a simpler config - remove everything but the basics for a Drupal site.

For example you can get by without any possible location conflicts with this for a Drupal 6 or 7 site. try_files doesn't work properly still I think (I might be wrong, it might work now in D7, I forget to re-check myself) so I just keep the if() in there still.

This assumes your fastcgi.conf defines SCRIPT_FILENAME as $document_root$fastcgi_script_name

This should work for you:

server {
   server_name domainname;
   root /var/www/domainname/public_html;
   access_log /var/log/nginx/domainname.access.log;
   error_log /var/log/nginx/domainname.error.log notice;
   if (!-e $request_filename) {
      rewrite ^/(.*)$ /index.php?q=$1 last;
   }
   location ~ .php$ {
      fastcgi_pass 127.0.0.1:9000;
   }
}

try the simple conf also

spacereactor's picture

try the simple conf also getting same result 404 for media files. Sorry, I in a rush for a project i move to drupal 7 for now, the drupal 7 have private and public file system path that is easier to work with.

Whatever works!

mike503's picture

Whatever works! However...

you won't get the benefit of X-Accel-Redirect in D7 (without something like the xsend module.) I was working on a site going back and forth between D6 and D7, at one point I had D6 working perfectly with all files going through X-Accel-Redirect it seems. D7 totally changed up how files are served, and for X-Accel-Redirect to work, the path has to be normalized to the URI from the document root in nginx - and I just got a headache trying to get that to work.

I should see if I can make some time to mess around with that again :)

ya, i understand that, i

spacereactor's picture

ya, i understand that, i choose drupal 6 instead of drupal 7 at the first place. but can't get pass the media problem and need a quick fix by moving to drupal 7. Really hope to get it resolve but the clock is ticking for my deadline.

Looks like you're talking

mike503's picture

Looks like you're talking about two different problems, or really ONE problem but making this module part of the requirement.

The only thing xsend do is passing off byte-serving files using readfile() or whatever Drupal uses which keeps a PHP engine busy spoon-feeding a browser bytes, and it can be offloaded to the webserver easily.

Sounds like you just care about public/private files, which isn't what this is designed for. This just accelerates the download of files that get pushed through PHP (which would be any private files since it requires PHP code to boostrap the user and check privileges, etc.)

D6 has no ability to do private and public at the same time. D7 does, so that could buy you time, but I don't recommend scaling a site through readfile() unless you can throw away a lot of resources on PHP engines staying busy for that. It's not a scalable mechanism in my mind. I would definitely look at offloading it as soon as possible.

Yes, that config is

brianmercer's picture

Yes, that config is incomplete in several ways.

Sorry about your deadline. If you want help (and have time) post your current config with drupal version, what types of files are private download, if you're using the upload tracker, and if you're using xsend.

my drupal conf and nginx conf

spacereactor's picture

my drupal conf and nginx conf is in the previous tread, about 7 to 8 tread before this, after that i change to the simple conf that mike503 advise, but doesn't work.

I still hope to get it to run on drupal6 as some module i need haven't port over to drupal 7 yet. I follow your guide and I'm able to use xsend and "protected" folder to download pdf and odt only but fail to display or download image file like gif & jpeg and media file like mov & mp4.

my drupal root /var/www/domainname/public_html
my private is outside drupal /var/www/domainname/private
protected folder /var/www/domainname/public_html/protected (generated by cron i think)

I trying to build a private drupal site with file doc archive(doc|odt|pdf), photo gallery archive(jpg|gif|png) and media archive(flv|mov|avi|mp4) for two work groups, est about 15 users only.

Give this a try:server { 

brianmercer's picture

Give this a try:

server {
  server_name www.domainname;
  rewrite ^ $scheme://domainname$request_uri? permanent;
}

server {
  server_name domainname;
  root /var/www/domainname/public_html;
  access_log /var/log/nginx/domainname.access.log;
  error_log /var/log/nginx/domainname.error.log notice;

  location / {
    rewrite ^/(.*)$ /index.php?q=$1 last;
  }

  location = /favicon.ico {
    try_files /favicon.ico =204;
  }

  location = /robots.txt {}

  ## Use this location if using private download method
  location ^~ /system/files/ {
    rewrite ^/(.*)$  /index.php?q=$1 last;
  }

  ## Use this location for xsend module
  location ^~ /protected/ {
    internal;
    alias /var/www/domainname/private/;
    error_page 404 @drupal;
  }

  ## Use this location for tracking uploads
  location ~ (.*)/x-progress-id:(\w+) {
    rewrite ^(.*)/x-progress-id:(\w+) $1?X-Progress-ID=$2;
  }

  ## Use this location for tracking uploads
  location ^~ /progress {
    report_uploads uploads;
  }

  location ~* ^.+\.xml$ {
    try_files $uri @drupal;
  }

  ## Disable this section if not using the imagecache module
  location ~* ^.*/imagecache/.+$ {
    expires 45d;
    try_files $uri @drupal;
  }

  location ~* ^.+.(jpg|jpeg|gif|png|ico|js|css)$ {
    access_log off;
    expires 45d;
  }

  ## Use this location to allow any type of file from the Drupal /files
  ## directory to be be downloaded normally. Be careful with modules like
  ## backup_migrate which put sensitive files here
  location ~* ^.*/files/.+$ {}

  location @drupal {
    rewrite ^/(.*)$ /index.php?q=$1 last;
  }

  ## Define "php" in an upstream directive to the proper port or socket
  location = /index.php {
    include /etc/nginx/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /var/www/domainname/public_html/index.php;
    fastcgi_pass php;
  }

  ##Disable after installation
#  location = /install.php {
#    include /etc/nginx/fastcgi_params;
#    fastcgi_param SCRIPT_FILENAME /var/www/domainname/public_html/install.php;
#    fastcgi_pass php;
#  }

  ##Disable if using Drush for updatedb
  location = /update.php {
    include /etc/nginx/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /var/www/domainname/public_html/update.php;
    fastcgi_pass php;
  }

  ##Disable if using Drush for cron
  location = /cron.php {
    include /etc/nginx/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /var/www/domainname/public_html/cron.php;
    fastcgi_pass php;
  }

  ##Disable if not using any xmlrpc services
#  location = /xmlrpc.php {
#    include /etc/nginx/fastcgi_params;
#    fastcgi_param SCRIPT_FILENAME /var/www/domainname/public_html/xmlrpc.php;
#    fastcgi_pass php;
#  }

}

Oh yeah, for tracking uploads

brianmercer's picture

Oh yeah, for tracking uploads you'd need:

  ## Define "php" in an upstream directive to the proper port or socket
  location = /index.php {
    include /etc/nginx/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /var/www/domainname/public_html/index.php;
    fastcgi_pass php;

    ## If tracking uploads
    track_uploads uploads 60s;
  }

and backslash missing from this one:
  location ~* ^.+\.(jpg|jpeg|gif|png|ico|js|css)$ {
    access_log off;
    expires 45d;
  }

brianmercer try you conf and

spacereactor's picture

brianmercer try you conf and it WORK!!!! THANK YOU!!!!!

Nginx

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: