After some discussion on the filefield issue queue, today I released a module that adds the much needed upload progress functionality to sites running on nginx servers. Brian suggested I also post here as it may be useful to others not following that issue.
The maintainer of FileField still has to decide whether this should be integrated into the FileField module itself or whether it should go on drupal.org as a separate module. So in the mean time, I have uploaded it into my github repository which also offers the ability to download the tarball:
http://github.com/smoothify/filefield_nginx_progress
To use it Nginx does need to be compiled with masterzen's upload progress module and there is some configuration required in nginx.conf, the details of which are listed in my github repository above.
Comments
Thanks again. For folks using
Thanks again.
For folks using Ubuntu who don't want to recompile nginx, there's a couple PPAs on Launchpad that have nginx with the upload_progress module already compiled in.
I tested this one and it worked fine:
https://launchpad.net/~chris-lea/+archive/nginx-extras
Custom Rollout
It's actually incredibly easy to make your own PPA build of Nginx.I do it for every single development release of nginx. I do an extremely customized roll out though so it'd be painful for anyone but my company to use.
Michael Lustfield
Ubuntu Member, Nginx Hacker
Thank you! I can confirm it
Thank you!
I can confirm it works also with latest nginx-0.8.41
http://skitch.com/omega8cc/de6iy/picture-7
Quick how-to example:
#
# install nginx
#
cd /var/opt
git clone git://github.com/masterzen/nginx-upload-progress-module.git
wget http://sysoev.ru/nginx/nginx-0.8.41.tar.gz
tar -xzf nginx-0.8.41.tar.gz &&
cd nginx-0.8.41
./configure --prefix=/opt \
--sbin-path=/opt/sbin/nginx \
--conf-path=/opt/etc/nginx/http-nginx.conf \
--error-log-path=/var/log/http-nginx/error.log \
--http-log-path=/var/log/http-nginx/access.log \
--pid-path=/var/run/http-nginx.pid \
--lock-path=/var/lock/http-nginx.lock \
--user=www-data \
--group=www-data \
--with-http_realip_module \
--with-rtsig_module \
--with-http_gzip_static_module \
--with-http_stub_status_module \
--with-http_ssl_module \
--without-mail_pop3_module \
--without-mail_imap_module \
--without-mail_smtp_module \
--http-client-body-temp-path=/var/cache/nginx/client_body_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--add-module=/var/opt/nginx-upload-progress-module/ \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp &&
make &&
make install
cd /var/opt
1. Download & enable module: git clone git://github.com/smoothify/filefield_nginx_progress.git
Thanks for the feedback
Thanks for the feedback Grace, its good to see it working for you!
BTW. I found the Nginx and Aegir configuration from yourself and Brian very useful, thanks for putting them on your github :)
The filefield_nginx_progress
The filefield_nginx_progress module has now been added to drupal.org
http://drupal.org/project/filefield_nginx_progress
Is there any posiblity that i
Is there any posiblity that i can use upload progress with my nginx that is a reverse proxy for apache?
my only not really clever solution is to pass the proxy but then i have to use another port besides 80 and thats annoying...
i compiled with uploadprogress module, can see that there a lot of GET req and when i try to open the page (example.com/progress/...) he just shows the "starting state" JSON Object...
Do i have to rewrite the ID from apache to the nginx? because on my apache server i can see the GET requests, may they never should get there because nginx is handling the upload (im right?)
also i get the id twice:
[code]
2a01:dontgetmyv6 - - [27/Jul/2010:23:18:16 +0000] "GET /progress?X-Progress-ID=2d8ae6f996fae1239c34df2329efa100 HTTP/1.0" 200 38 "-" "Drupal (+http://drupal.org/)"
2002:dontgetmyv6 - - [27/Jul/2010:23:18:16 +0000] "GET /filefield_nginx_progress/2d8ae6f996fae1239c34df2329efa100 HTTP/1.1" 200 64 "http://ssdf/node/add/type" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.7) Gecko/20100713 Firefox/3.6.7"
[/code]
but only the last is passing to the apache...
should work
I have not used it as a reverse proxy myself, but I believe it should work.
It sounds like your rewrites are set up correctly as GET /progress?X-Progress-ID=xxx should NOT be forwarded to apache.
If you have the module installed, a good way to test if nginx is tracking is to upload a large-ish file, and before submitting use firebug to retrieve the UPLOAD_IDENTIFIER field value from the form.
Then access this link in your browser yourdomain.com/progress?X-Progress-ID=UPLOAD_IDENTIFIER (pasting the real upload id in of course)
if you do this before you start uploading it should say starting - if you start the upload and keep refreshing the page, if its working you should get more data about the progress of the upload as it happens
ok so tried it: firebug says
ok so tried it:
firebug says that the ID is fb445608560d50c43c0dcd3125927b03, then i started the upload in the nginx log i can see the folowing output:
2a01:söldf - - [29/Jul/2010:17:58:47 +0000] "GET /progress?X-Progress-ID=fb445608560d50c43c0dcd3125927b03 HTTP/1.0" 200 38 "-" "Drupal (+http://drupal.org/)"
::ffff:xxx - - [29/Jul/2010:17:58:47 +0000] "GET /filefield_nginx_progress/fb445608560d50c43c0dcd3125927b03 HTTP/1.1" 200 64 "http://sdf.net/node/add/trainvideo" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8"
ok so the ID is the right one and /progress?X-Progress-ID=... is not forwared to to apache, but filefield_nginx_progress/... is.
another problem is that i cant see anything but new Object({ 'state' : 'starting' }) on the /progress?X-Progres-ID=.... page
so whats now broken? is it the upload progress module?
edit: i think the problem is the ID is set by the apache webserver and nginx generates a new one which is obviously not the one which is in the form... so my browser asks the nginx with wrong id and the apache has no notice about the upload status :(
this should probably go in the issue queue...
http://drupal.org/project/issues/filefield_nginx_progress
However, looking at your problem it seems like the upload progress module is working but so far is not tracking.
My thoughts are that it may be something in your nginx.conf file that isn't quite right - do you have the following lines?
location ~ (.)/x-progress-id:(\w) {rewrite ^(.)/x-progress-id:(\w) $1?X-Progress-ID=$2;
}
location ^~ /progress {
report_uploads uploads;
}
and do you have the following in the proxying section ?
proxy_redirect default;
track_uploads uploads 60s;
perhaps it might be a good idea if you post some of your configuration, hiding of course any private info in there.
mhh what configs do you
mhh what configs do you need?
I just played a little bit around with it so do i understand that right:
* drupal just generate a completly random UUID (X-Progress-ID) which not depend on nginx or apache (apache or nginx do not know them until i start a upload)
* If i start a upload the file is cached at nginx proxy and after complete upload put to the apache server.
* so nginx is completly managing the upload and have to keep track of the progress
* the drupal module accesses example.com/filefield/progress/, your module uses example.com/filefield_nginx_progress/ID
* nginx has to rewrite to example.com/progress?X-Progresss-ID=
* so why i cant rewrite /filefield/progress/ to something?
so this is a config for a domain:
server {
include snippets/http-common-listen-port;
server_name mydomain.com;
location ^~ /progress {
report_uploads upload;
}
location ~ (.)/x-progress-id:(\w) {
rewrite ^(.)/x-progress-id:(\w) $1?X-Progress-ID=$2;
}
location ^~ (.)/filefield/progress/(\w) {
rewrite ^(.)/filefield/progress/(\w) $1/progress?X-Progress-ID=$2;
}
location / {
proxy_pass http://10.42.42.42/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_read_timeout 300;
client_max_body_size 300M;
proxy_redirect default;
upload_progress_json_output;
track_uploads upload 60s;
}
i also tried to rewrite the filefield_nginx_progress (with enabled module) but these requests are also send to the apache! why? In my oppinion the apache should not get any progress query?!
thanks in advance!
You have got the gist of how
You have got the gist of how it works, except the filefield_nginx_progess should go to apache, so that part is expected. (because nginx's upload format is different, the request needs to be fired from drupal and then parsed, rather than straight to the progress bar javascript)
Looking at your configuration, do you have the following ABOVE the server directive?
# Upload Progress: reserve 1MB under the name 'uploads' to track uploads
upload_progress uploads 1m;
server {
...
Also you are using:
report_uploads upload; NOT report_uploads uploads;
and
track_uploads upload 60s; NOT track_uploads uploads 60s;
the plural is mentioned in the documentation, and must match the declaration above.
Finally you shouldn't have:
location ^~ (.)/filefield/progress/(\w) {rewrite ^(.)/filefield/progress/(\w) $1/progress?X-Progress-ID=$2;
}
or
upload_progress_json_output;Broken link at github
I get “That page doesn't exist!” at the link in the OP:
http://github.com/smoothify/filefield_nginx_progress
It is available now here:
It is available now here: http://drupal.org/project/filefield_nginx_progress
Links there are broken, too
There’s a module, but it’s impossible to use because the links to the information on how to configure it are broken.
There's a README.txt file
There's a README.txt file with the module with the instructions.
Incorporate in the nginx.conf file..??
I have got nginx compliled with the upolad progress etc now.. but I need to know how to add this to the nginx.conf file... The first bit is my current conf file.. but I need to add the second bit to the conf file correctly... any help is appreciated.
Current conf file:
cPanel Nginx Master configuration
user nobody;
error_log logs/error.log;
Number of worker you need
worker_processes 1;
How many connections a worker can handle maximum.
events {
worker_connections 50000;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
server_names_hash_max_size 10000;
server_names_hash_bucket_size 1024;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 10;
gzip on;
gzip_min_length 1100;
gzip_buffers 4 32k;
gzip_types text/plain application/x-javascript text/xml text/css;
ignore_invalid_headers on;
client_header_timeout 3m;
client_body_timeout 3m;
send_timeout 3m;
include "/usr/local/nginx/conf/vhost.conf";
include "/etc/cpnginx/cpanelproxy.conf";
}
And this is what I need to add to the above file:
Nginx Configuration
You must be running the nginx webserver compiled with MasterZen's Upload Progress module.
Then, you must configure nginx.conf to track upload progress correctly (the ... are your existing configuration) :
http {
Any ideas how to add this correctly?