Having problem trying to protect private file and image from unauthorised users. I not sure if this is the problem with drupal core or my nginx config. My goal is setup content type with image field, file field and set a role that can access the image and file fields.
Steps taken
1.) Install a fresh drupal 7
2.) Install field permission module (http://drupal.org/project/field_permissions)
3.) goto http://mydomain.com/admin/config/media/file-system and set my Private file system path "sites/default/files/private"
4.) Default download method = Private local files served by Drupal.
5.) Edit my Content type "Basic page" with photo and file fields.
6.) goto http://mydomain.com/admin/structure/types/manage/page/fields and Add new field for photo field and file field, both are Upload destination to private files.
Field permissions is for photo field
enable = Create field_photo (edit on content creation).
enable = Edit field_photo, regardless of content author.
enable = View field_photo, regardless of content author.
Field permissions is for file field
enable = Create field_file (edit on content creation).
enable = Edit field_file, regardless of content author.
enable = View field_file, regardless of content author.
7.) Create a new role "private view" http://mydomain.com/admin/people/permissions/roles
8.) For new "private view" role give field_photo and field_file with create, edit and view permission.
9.) Create new Basic content and upload image and file
This is where i get stuck. This image and file doesn't for those without the "private view" role but if i copy the image url and file url and logout, use anonymous user, I still can access the image url and file url even i not in "private view" role.
my domainname nginx conf follow most part of perusio (https://github.com/perusio/drupal-with-nginx)
location ~* /system/files/ {
try_files $uri /index.php?q=$uri&$args;
log_not_found off;
}
location ~* /files/private/ {
internal;
}
Anyone know how to resolve this?
Comments
nginx suggestion: 'protect private file and image area'
I use something like this:
Do not be alarmed about my use of the 'optional' etags (nginx add-on)
# imagecache and imagecache_external supportlocation ~* /(?:external|system|files/imagecache|files/styles)/ {
access_log off;
tcp_nodelay off;
gzip_static off; # EdgeCast CDN does not want gzip assets!
expires max;
# fix common problems with old paths after import from 'standalone' to 'multi-site'
rewrite ^/sites/(.)/files/imagecache/(.)/sites/default/files/(.)$ /sites/$host/files/imagecache/$2/$3 last;
rewrite ^/files/imagecache/(.)$ /sites/$host/files/imagecache/$1 last;
rewrite ^/files/styles/(.)$ /sites/$host/files/styles/$1 last;
etags on; # etags required
etag_hash on;
etag_hash_method md5;
# Unset unnecessary headers
if_modified_since off;
add_header Pragma ""; # Disable Pragma
add_header Last-Modified ""; # Disable Last-Modified (use ETag instead)
add_header Cache-Control "public, must-revalidate, proxy-revalidate";
add_header X-Header "IC Generator 1.0";
try_files $uri /index.php?q=$uri&$args; # Using D7 'new' request_path() method
}
# deny direct access to private downloadslocation ~ ^/sites/.*/private/ {
access_log off;
internal;
}
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
is there other setting. my
is there other setting. my current nginx doesn't include etags add-on module
Revised - without
Revised - without etags:
# imagecache and imagecache_external supportlocation ~* /(?:external|system|files/imagecache|files/styles)/ {
access_log off;
tcp_nodelay off;
gzip_static off; # EdgeCast CDN does not want gzip assets!
expires max;
# fix common problems with old paths after import from 'standalone' to 'multi-site'
rewrite ^/sites/(.)/files/imagecache/(.)/sites/default/files/(.)$ /sites/$host/files/imagecache/$2/$3 last;
rewrite ^/files/imagecache/(.)$ /sites/$host/files/imagecache/$1 last;
rewrite ^/files/styles/(.)$ /sites/$host/files/styles/$1 last;
# Unset unnecessary headers
if_modified_since off;
add_header Pragma ""; # Disable Pragma
add_header Cache-Control "public, must-revalidate, proxy-revalidate";
add_header X-Header "IC Generator 1.0";
try_files $uri /index.php?q=$uri&$args; # Using D7 'new' request_path() method
}
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
doesn't work for me.
doesn't work for me. anonymous user can still access the file or image example like http://mydomain.com/system/files/photo/sample.jpg or http://mydomain.com/system/files/download/sample.pdf
This is where i get stuck.
@spacereactor
Quoting:
1) That suggest Nginx caching -and/or-
2) Session / Cookies issues
I would need to see your entire listing for: nginx.conf + D7 settings.php
to help on that issue!
For the moment (via Nginx), I suggest you follow; http://drupal.org/project/barracuda
git @ http://drupalcode.org/project/barracuda.git
rather than https://github.com/perusio/drupal-with-nginx
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
(No subject)
my nginx.conf
user www-data;
worker_processes 4;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
worker_rlimit_nofile 2048;
events {
worker_connections 1024;
use epoll;
multi_accept on;
}
http {
include /etc/nginx/fastcgi.conf;
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
sendfile on;
set_real_ip_from 0.0.0.0/32; # all addresses get a real IP.
real_ip_header X-Forwarded-For; # the ip is forwarded from the load balancer/proxy
limit_zone arbeit $binary_remote_addr 1m;
client_body_timeout 60;
client_header_timeout 60;
keepalive_timeout 10 10;
send_timeout 60;
reset_timedout_connection on;
client_max_body_size 100m;
tcp_nodelay on;
tcp_nopush on;
gzip on;
gzip_buffers 16 8k;
gzip_comp_level 1;
gzip_http_version 1.1;
gzip_min_length 10;
gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon application/vnd.ms-fontobject font/opentype application/x-font-ttf;
gzip_vary on;
gzip_proxied any; # Compression for all requests.
gzip_disable "msie6";
gzip_static on;
server_tokens off;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
add_header X-Frame-Options sameorigin;
upstream phpcgi {
server unix:/var/run/php5-fpm.sock;
}
include /etc/nginx/sites-enabled/*;
}
my drupal setting.php <?php/
my drupal setting.php
i remove it, just a standard drupal setting.php
(No subject)
my domain.com.conf
i have remove it, take up too much space. see https://github.com/perusio/drupal-with-nginx
Please comment out (#) the
Please comment out (#) the following code you have left active (in nginx.conf):
location ~* /system/files/ {try_files $uri /index.php?q=$uri&$args;
log_not_found off;
}
Also, for the moment comment out the following (in nginx.conf):
## Use index.html whenever there's no index.php.location = / {
error_page 404 =200 /index.html;
}
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
What's the rationale for that?
Care to elaborate?
Do you understand what you're saying? You're saying that he should comment out the config stanza that handles the private files.
Also the '= /' location is only used if acessing '/' and furthermore the redirect only happens on a 404.
What do you get in
in your logs? A 404?
Also this is riddled with unnecessary rewrites IHMO:
location ~* /(?:external|system|files/imagecache|files/styles)/ {access_log off;
tcp_nodelay off;
gzip_static off; # EdgeCast CDN does not want gzip assets!
expires max;
# fix common problems with old paths after import from 'standalone' to 'multi-site'
rewrite ^/sites/(.)/files/imagecache/(.)/sites/default/files/(.)$ /sites/$host/files/imagecache/$2/$3 last;
rewrite ^/files/imagecache/(.)$ /sites/$host/files/imagecache/$1 last;
rewrite ^/files/styles/(.)$ /sites/$host/files/styles/$1 last;
# Unset unnecessary headers
if_modified_since off;
add_header Pragma ""; # Disable Pragma
add_header Cache-Control "public, must-revalidate, proxy-revalidate";
add_header X-Header "IC Generator 1.0";
try_files $uri /index.php?q=$uri&$args; # Using D7 'new' request_path() method
}
You should be extremely careful when using regex based locations. They introduce a procedural component in what is mostly a declarative language.
You shouldn't be able to access
the private files directly. Meaning that when you set it
internalit means that the access must be made through an internal redirect liketry_filesdoes, for example.If I understand correctly you want all authenticated users to be able to view the images? Or just those that have the private view role?
i want those users under
i want those users under "private view" role can access and view the private cck field and 404 to those other roles or anonymous users.
If you read my first post at the top, currently field permission work and doesn't display the private cck field but if anonymous user who know the url link for the private field can still access the private field. example like http://mydomain.com/system/files/photo/sample.jpg or http://mydomain.com/system/files/download/sample.pdf
Drupal 6 with your nginx setting work with the private
location ~* /system/files/ {try_files $uri /index.php?q=$uri&$args;
log_not_found off;
}
location ~* /files/private/ {
internal;
}
but drupal 7 doesn't.
So the issue is
that you want to protect the files on D7, but somehow they're acessible and they shouldn't. If so then what you need is to define a
locationwith the proper regex. What's the path to the image files that you want to block? I venture that the images are being written somewhere else.Can you show how do you do the requests as an anon user?
the private path is
the private path is "sites/default/file/private"
I want to block two fields, Image and File.
Image field for File directory set as "photo" so the website path for Image to block is http://mydomain.com/system/files/photo/sample.jpg
File field for File directory set as "download" so the website path for File to block is http://mydomain.com/system/files/download/sample.pdf
I'm guessing
that somehow the location being matched is the catch all regex based location for static files. That's why the sloppy way drupal spreads static files around is a PITA when trying to implement a clean Nginx config. That sloppiness derives from the reverse logic that Apache
.htaccessinspires, IMHO. Regex based locations are a Pandora's box :(Also you have probably files of the same name around as public files, otherwise you would get a 404.
Change the location to this:
location ~* /system/files/.*\.(?:pdf|jpe?g|png)$ {try_files $uri /index.php?q=$uri&$args;
}
Try it & report. Thanks.
Why would you want a
Why would you want a "try_files $uri" in the /system/files location? Why not:
location ^~ /system/files {fastcgi_pass phpcgi;
}
I know some folks disagree, but I'm in favor of placing private files outside the web root.
Well
that's one way to do it. Of course it forces you to replicate all the fastcgi or proxy_pass stuff on this location. The
try_filesdirective requires at least two arguments.Also
AFAIK your approach although working on D7 (on D6 you'll need a capture for the args) relies on
request_path()building the query string. Although the firsttry_filesarg is dummy I suspect that doing a gratuitous Cstat()is faster than relying on PHP string manipulation.For Drupal 6 I use:
For Drupal 6 I use:
location ^~ /system/files/ {rewrite ^/(.*)$ /index.php?q=$1 last;
}
and for Drupal 7 I use:
location ^~ /system/files/ {include /etc/nginx/fastcgi_params;
fastcgi_param SCRIPT_FILENAME /var/www/drupal_dir/index.php;
fastcgi_param SCRIPT_NAME /index.php;
fastcgi_pass php;
}
It seems like using try_files for D7 is adding a gratuitous stat and an unnecessary pass looking up
location = /index.php.OK
That's a nice solution. But I would prefer using a
fastcgi_private_files.conffile where the FastCGI parameters are redefined. This way is easier to switch to a reverse proxy setup. Hmm, thinking onmap. I'm going to try it.Great discussion :)
Mo' better :)
If using something like this:
location ~* /system/files/ {include fastcgi_conf;
fastcgi_param SCRIPT_FILENAME $document_root/index.php;
fastcgi_param SCRIPT_NAME /index.php;
fastcgi_param QUERY_STRING q=$uri;
fastcgi_pass phpcgi;
}
I'm able to avoid the
else ifinrequest_path(), meaning no stinkin' PHP string manipulation :)Now to the file solution.
Sounds good. Is there any
Sounds good. Is there any situation in which there might be a query string in the original request?
In request_path()
which is invoked by
drupal_environment_initialize. Going through the code you realize thatrequest_path()is Drupal 7 way of mimicking a query string. In Drupal 6 that was done by a rewrite at the server level. Now the purpose ofrequest_path()is to extract the value of the query string. So in fact unless you have a non-empty$_GET['q']you're going to have to do some string manipulation and take into consideration things like unescaping the unsafe chars in the query string like&.<?php// From request_path() code.
if (isset($_GET['q'])) {
// This is a request with a ?q=foo/bar query string. $_GET['q'] is
// overwritten in drupal_path_initialize(), but request_path() is called
// very early in the bootstrap process, so the original value is saved in
// $path and returned in later calls.
$path = $_GET['q'];
}
(...) // other stuff in a elseif
?>
That's my take on it. If there's someone more versed on drupal 7 internals, please correct any misunderstanding on my part.
Also it works with drupal 6
using the
QUERY_STRINGFCGI param. Commited the new private files handling with FCGI. Thanks Brian.thank.location ~*
thank.
location ~* /system/files/ {include /etc/nginx/fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root/index.php;
fastcgi_param SCRIPT_NAME /index.php;
fastcgi_param QUERY_STRING q=$uri;
fastcgi_pass phpcgi;
log_not_found off;
}
and i move my private path out of the website root.
my website root is set to "/var/www/mydomain.com/public_html"
From "sites/default/files/private" to "/var/www/mydomain.com/private/default"
I not sure how to set the private for nginx, Currently i just enter as
location ~* private {internal;
}
Once you move the private
Once you move the private files out of the webroot, there is no need to protect it.
However, you are probably going to need something similar to support perusio's handy http://drupal.org/project/nginx_accel_redirect module. To reach the private directory outside the web root you'll need an alias. Something like:
location ^~ /private/ {internal;
alias /var/www/mydomain.com/private/default/;
}
Also, try to use literal locations instead of regular expressions where you know the beginning of the string and don't need wildcards. i.e.
location ^~ /system/files/is better thanlocation ~* ^/system/files/.*$. Literal locations are faster and you don't have to worry about the order of the locations.Yes
If it's writable by the server then it needs to get protected. There's plenty of tools that can do fishing expeditions and discover all web accessible URLs.
I agree on the literal locations of the
^~sort. However than can get tricky if you have catch all regex based locations. If drupal had an organized way of placing static assets, then everything would be much simpler and we could use literal locations everywhere and nest all regex based locations.Unfortunately drupal spreads files around a lot of dirs and we need a catch all regex based location for serving them. Regex based locations are, I'm afraid, a necessary evil :(
We could
take advantage of the higher priority of negated regex based locations if we take in consideration two things:
The
^~locations need to come after the equivalent regex based. For example:location ~* /system/files/ { # must appear first
(...) # do whatever you want here. To be on the safe side better duplicate the directives of the "real" location
}
and
location ^~ /system/files/ {
(...) # this is the real location
}
All regex based location that we want to negate must be included in the config.
This way we can have both catch all regex based locations and literal locations. If your config has a lot regex based locations or starts with (regular) locations then you'll gain something in terms of speed. Since Nginx will exit the location find phase sooner than later. But if your config is simple — like most drupal configs are — then you'll stand to gain very little at the cost of making the config less clear and unnecessary complex IMHO.
If a request matches a ^~
If a request matches a ^~ location then no regex locations are checked.
What regex location would need to be prioritized higher than the /system/files location?
If having a
a catch all location with a bunch of file extensions and a file called
this_is_my_document_file.pdf. Like:## Keep a tab on the 'big' static files.location ~* ^.+\.(?:m4a|mp3|mp4|mov|ogg|flv|pdf|ppt[x]*)$ {
expires 30d;
## No need to bleed constant updates. Send the all shebang in one
## fell swoop.
tcp_nodelay off;
}
Back to the fray
When I tested the issue of literal locations I did it on a OpenAtrium site. The fact is that OA prepends the group name to the
system/files/path.So if I have a group called
testthe requested URI will be/test/system/files/my_document_file.pdffor example. In that situation the regex based catch all location will win. I didn't pursue the issue further.Now that I'm updating nginx_accel_redirect in the issue queue a question was raised that led me to test it in D7. It works. So the issue seems to be in the case of OA that there's some black magick being done that screws up the URIs and instead of ending up with a URI starting with
/system/files/we end up with the group name prepended.I'm updating my config with the new literal locations. They're faster and less prone to side effects. Unfortunately people using OA or ManagingNews must bear the cost of fiddling with URIs that the OA developers have done.
Conclusion: it works in most sites. But not in OA or ManagingNews.
In the above example:
location ^~ /system/files {(...)
}
doesn't work. But:
location ^~ /test/system/files {(...)
}
does.
I find it not very practical, to say the least, to force you to alter your Nginx configuration for accommodating a purely dubious "aesthetical" decision on the part of OA and MN developers regarding URIs. This is a case where we would have been better served by the classic worse is better philosophy.
That definitely makes sense.
That definitely makes sense. I didn't know about OA or MN. I noticed that omegacc's configs had special handling for the organic groups module, but it's not something that I use myself.
I don't think
that this is something that comes from organic groups but rather from the purl module, which is used in both OA and MN. At least here on g.d.o, the files reside in a standard location. If you upload a file you get a URI that is standard.
I haven't looked the code yet so take it with a grain of salt until code proof is given.
http://drupal.org/project/ngi
http://drupal.org/project/nginx_accel_redirect currently doesn't support drupal 7 yet but the good news is mattman did a drupal port and I edit from there. http://drupal.org/node/1083494#comment-4663982
To brianmercer, using your
To brianmercer, using your recommend setting
location ^~ /system/files/ {include /etc/nginx/fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root/index.php;
fastcgi_param SCRIPT_NAME /index.php;
fastcgi_param QUERY_STRING q=$uri;
fastcgi_pass phpcgi;
log_not_found off;
}
BUT can user goto 404 if they don't have view permission?
Currently for a ANONYMOUS USER access http://mydomain.com/system/files/photo/sample.jpg he/she can't see the image cause they have to login in order to view the image but he/she is seeing a bundle of funny characters like the once below.
!I'0��O�&z] 1:��˲0��P�;h�;�}rF���*�َG���K,|�_L��;��p�柊Ђ��2���b��kU�Tk�>� �뀼Q��ϿX�_���s��2n�Iu�s,&̈����e��I�<�RD�4�g��3D =�GEPj08��u�M_L=J�73�}�qh�ϸ
>q9|#�H���$a��7y��NYA�ˋ � ݡ�Y��^^$Q2Y�LiJj��d��{��ʭ �2�ų�RN�d�� E$N�You're absolutely right, but
You're absolutely right, but it doesn't have to do with nginx.
I set up a test site on my test server as you instructed in the first post. It does show the secret files. But it behaves exactly the same way on apache, which I also have installed on my test server.
Your steps look correct and they're the same way I'd do it with D6 cck field permissions. The files are hidden in the node view, but then are directly accessible through the /system/files address.
Dunno why, but it behaves that way under apache2, so I think we must be configuring something incorrectly.
This node leads to a patch
This node leads to a patch but the patch is inaccessible:
http://drupal.org/node/1104060
Such is the danger of using
Such is the danger of using alpha quality modules. :(
http://drupal.org/node/1194364
You could workaround it by making each file a node. Node level permissions should be working without any contrib modules.
@perusio Many thanks!
@perusio
Many thanks!
1) Might be good to jump-back to the original context on this thread; -> http://groups.drupal.org/node/158059#comment-530244
2) The 'extra' rewrites are categorized by;
We have been 'de-bugging' a nginx issue that does not occur for me (using typical nginx settings via http://drupalcode.org/project/barracuda.git), but does occur for @spacereactor using his current nginx via https://github.com/perusio/drupal-with-nginx
I have heavily followed, de-bugged, and implemented both platforms (omega8cc and perusio) over the last few weeks. So sorry to be a 'pain' :)
Your input is very welcome!
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
@spacereactor If your Nginx
@spacereactor
If your Nginx configuration works with D6 per your 'quote';
Then I suggest you either 'migrate' to http://drupalcode.org/project/barracuda.git (omega8cc's Nginx methods), or wait for Perusio (via https://github.com/perusio/drupal-with-nginx -via some updates).
I did try both methods (oemga8cc and perusio), and finally implemented parts of 'both'. I had some critical issues with the direct 'as-is' use of guides @ https://github.com/perusio/drupal-with-nginx.
Be patient, if you test enough 'options' you will find your own answers (I certainly did)!
--
Linux: Web Developer
Peter Bowey Computer Solutions
Australia: GMT+9:30
(¯`·..·[ Peter ]·..·´¯)
So far this is my conclusion,
So far this is my conclusion, THANK to perusio with fastcgi_private_files.conf is now able let drupal control who to view the cck field but drupal cck field itself is buggy. The Field permission only block access to view from /system/files/ but if you know what is the URL of the image/field is store, ANONYMOUS user can still access it. Either you block private folder with nginx or move your private folder outside webroot. Again THANK to brianmercer on how to set private path outside webroot and his tip to make http://drupal.org/project/nginx_accel_redirect work under drupal 7.
If AUTHENTICATED user access the private field and he/she doesn't have the access right he/she get 404, but when ANONYMOUS user try to access private field, they get funny characters. My guess that is problem is drupal field permission itself is buggy and nothing to do with nginx setting.
Again thank you for all your help.