Getting HTTPS to play nice with Varnish

am4's picture

I've gotten Varnish working in front of Drupal 7. I've also got Pound in front of Varnish, and can successfully browse the site via HTTPS. However, whenever I try to log in over HTTPS, I get a 403 forbidden error, and I'm not entirely sure why. My guess is Varnish-related... but I'm not positive.

Anyone have any ideas/pointers as to what I'm missing? I'm glad to post Firebug output showing the POST to /user and the resulting 403 if it helps.

Relevant snippets from settings.php:

<?php
$conf
['reverse_proxy'] = TRUE;
$conf['page_cache_invoke_hooks'] = false;
$conf['cache'] = 1;
$conf['cache_lifetime'] = 0;
$conf['page_cache_maximum_age'] = 300;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');
$conf['reverse_proxy_header'] = 'HTTP_X_FORWARDED_FOR';
$conf['https'] = TRUE;

if (isset(
$_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https') {
 
$_SERVER['HTTPS']='on';
}
?>

Relevant snippets from default.vcl:

sub vcl_recv {
    remove req.http.X-Forwarded-For;
    set req.http.X-Forwarded-For = client.ip;

    # Do not allow outside access to cron.php or install.php
    if (req.url ~ "^/(cron|install).php$" && client.ip !~ inside) {
        # Have Varnish throw the error directly.
        error 403;
    }

    # Do not allow outside access to /admin
    if (req.url ~ "/user/?" && client.ip !~ inside) {
        # Have Varnish throw the error directly.
        error 403;
    }

    if (req.url ~ "/admin/?" || req.url ~ "/user/?") {
        return (pass);
    }

    # Remove the "has_js" cookie
    set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");

    # Remove the "Drupal.toolbar.collapsed" cookie
    set req.http.Cookie = regsuball(req.http.Cookie, "Drupal.toolbar.collapsed=[^;]+(; )?", "");

    # Remove any Google Analytics based cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");

    # Remove the Quant Capital cookies (added by some plugin, all __qca)
    set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");

    # Are there cookies left with only spaces or that are empty?
    if (req.http.cookie ~ "^ *$") {
        unset req.http.cookie;
    }

    # Cache static content unique to the theme (so no user uploaded images)
    if (req.url ~ "^/themes/" && req.url ~ ".(css|js|png|gif|jp(e)?g)") {
        unset req.http.cookie;
    }

    # Normalize Accept-Encoding header
    if (req.http.Accept-Encoding) {
        if (req.url ~ ".(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
            # No point in compressing these
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            remove req.http.Accept-Encoding;
        }
    }

    # Don't cache the install, update or cron files in Drupal
    if (req.url ~ "install.php|update.php|cron.php") {
        return (pass);
    }

    # Anything else left?
    if (!req.http.cookie) {
        unset req.http.cookie;
    }

    if (req.http.Authorization || req.http.Cookie) {
        # Not cacheable by default
        return (pass);
    }
}

Comments

I would make sure that things

dalin's picture

I would make sure that things are working well over HTTP before you throw HTTPS into the mix. It may just be that this snippet is blocking you:

    # Do not allow outside access to /admin
    if (req.url ~ "/user/?" && client.ip !~ inside) {
        # Have Varnish throw the error directly.
        error 403;
    }

--
Dave Hansen-Lange
Technical Manager
Advomatic LLC
Great White North Office
Canada

My general rule when

mike booth's picture

My general rule when debugging something like this is to find the log entry. Either there is a log entry somewhere showing the 403 happening, or there's a log you haven't turned on that you can turn on, after which the 403 will generate a log entry.

Even if the log entry is otherwise unenlightening, it will tell you which layer of your stack is rejecting you.

Pound is terminating SSH, I presume?

Meanwhile, in the annals of premature guessing, my eye is drawn to these lines of your VCL:

if (req.url ~ "/user/?" && client.ip !~ inside) {
    # Have Varnish throw the error directly.
    error 403;
}

Does passing through Pound mess with client.ip such that it no longer matches "inside"? Also, where is "inside" defined? Is that some VCL builtin that I just don't know about or is it defined elsewhere in the file?

@dalin - yup, HTTP logins

am4's picture

@dalin - yup, HTTP logins through Varnish work fine.

@mike - Correct, Pound is terminating SSL. The acl is defined at the beginning of the .vcl and is only used to restrict access to /user. It is as follows:

acl inside {
    "10.10.128.0"/17;
    "127.0.0.1"/32;
}

Commenting out that block did nothing for HTTPS. 403 still thrown. (HTTPS logins directly through Apache work fine... but I want to be able to control clients using HTTPS via the use of Varnish.)

Here's the output from Firebug during an attempted login over HTTPS through Varnish/Pound:

POST user

Response Headers

Accept-Ranges bytes
Age   0
Cache-Control no-cache, must-revalidate, post-check=0, pre-check=0
Connection keep-alive
Content-Length   0
Content-Type  text/html; charset=UTF-8
Date   Tue, 09 Oct 2012 17:49:08 GMT
Etag  "1349804948"
Expires  Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified Tue, 09 Oct 2012 17:49:08 +0000
Location    https://drupal.domain.com/user/2
Server Apache
Set-Cookie   SESSe15687525d17b8ec181665a71c88775c=EttXyyPvqESdU4RapW7xZkrRagGNHgRH5I9P6x0yRRE; expires=Thu, 01-Nov-2012 21:22:28 GMT; path=/; domain=.drupal.domain.com; httponly SSESSe15687525d17b8ec181665a71c88775c=mAdWa_a_OvcIIoWBuiVbLqFJzwyHiukfd_xBOVz_eaQ; expires=Thu, 01-Nov-2012 21:22:28 GMT; path=/; domain=.drupal.domain.com; secure; HttpOnly
Via  1.1 varnish
X-Drupal-Cache  MISS
X-Varnish  2081364495

Request Headers
Accept    text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Encoding gzip, deflate
Accept-Language   en-us,en;q=0.5
Connection   keep-alive
Cookie   has_js=1; __utma=194497400.1529654640.1349804906.1349804906.1349804906.1; __utmb=194497400.3.10.1349804906; __utmc=194497400; __utmz=194497400.1349804906.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
DNT 1
Host  drupal.domain.com
Referer   https://drupal.domain.com/user
User-Agent   Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1

GET 2

Response Headers

HTTP/1.1 403 Forbidden
Server: Apache
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Tue, 09 Oct 2012 17:49:08 +0000
Cache-Control: public, max-age=300
Etag: "1349804948-1"
Content-Language: en
X-Generator: Drupal 7 (http://drupal.org)
Set-Cookie: SSESSe15687525d17b8ec181665a71c88775c=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.drupal.domain.com; secure; httponly
SESSe15687525d17b8ec181665a71c88775c=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.drupal.domain.com; httponly
Vary: Cookie,Accept-Encoding
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Content-Length: 8916
Accept-Ranges: bytes
Date: Tue, 09 Oct 2012 17:49:09 GMT
X-Varnish: 2081364496
Age: 0
Via: 1.1 varnish
Connection: keep-alive

Request Headers
GET /user/2 HTTP/1.1
Host: drupal.domain.com
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20100101 Firefox/15.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Referer: https://drupal.domain.com/user
Cookie: has_js=1; __utma=194497400.1529654640.1349804906.1349804906.1349804906.1; __utmb=194497400.3.10.1349804906; __utmc=194497400; __utmz=194497400.1349804906.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); SESSe15687525d17b8ec181665a71c88775c=EttXyyPvqESdU4RapW7xZkrRagGNHgRH5I9P6x0yRRE; SSESSe15687525d17b8ec181665a71c88775c=mAdWa_a_OvcIIoWBuiVbLqFJzwyHiukfd_xBOVz_eaQ

Further info - looks like

am4's picture

Further info - looks like those cookies are being deleted when trying to use Pound/Varnish. That doesn't happen when going to Apache directly for an HTTPS login.

VCL

Elliot Schlegelmilch's picture

Do you have vcl which removes cookies for certain state? You'll want to double check that.
Also, don't forget about varnishlog.

I've used NGINX in front of

technicalknockout's picture

I've used NGINX in front of varnish to handle https traffic before. Might be worth a look if pound is still giving you trouble. It took very little time to setup and has worked like a charm.

Google & https

mgifford's picture

Lots of folks are going to be looking at this due to Google's recent announcement about https & ranking http://googlewebmastercentral.blogspot.ca/2014/08/https-as-ranking-signa...

I'm still not sure how that gets around these concerns https://www.varnish-cache.org/docs/trunk/phk/ssl.html

Possibly using https://bensmann.no/seperate-varnish-caching-http-https/

For Varnish + HTTPS sites

dalin's picture

For Varnish + HTTPS sites that we've done in the past we have something in front of Varnish doing SSL termination (either a load balancer, or Nginx, or people used to use Squid for this too). Whatever does the termination should be passing some sort of HTTP header to indicate if the source was secure.

In this setup you don't get a secure connection between the terminator and the web server, but unless you're a bank you probably don't care.

--
Dave Hansen-Lange
Technical Manager
Advomatic LLC
Great White North Office
Canada

Tried CloudFlare? Does both HTTP and HTTPS

Agileware's picture

Have you tried using CloudFlare which handles both HTTP and HTTPS in front of your Drupal site. They also recently announced SSL support on the free plans too. Think of it like a hosted NGINX service, with other benefits.

Agileware
Australian Drupal Developers
http://agileware.com.au

Pound->Varnish->Drupal

adammalone's picture

I have Pound sitting in front of Varnish on my stack which allows all HTTPS pages to be cached. I've written about it over here and included full Drupal/Pound configurations as well as the relevant VCL configuration.

Pound terminates the SSL on port 443 and forwards to port 80 on Varnish. A header set in Pound tells Varnish that it was originally SSL traffic and Varnish is able to cache it separately to non-SSL. This is important if you're allowing mixed mode traffic as otherwise non-SSL resources can appear on SSL pages.

…and Varnish is able to cache

dalin's picture

…and Varnish is able to cache it separately to non-SSL. This is important if you're allowing mixed mode traffic…

It's important for pure HTTPS too because you probably want to redirect HTTP to HTTPS. So to avoid infinite redirects Varnish needs to be able to treat them separately.

--
Dave Hansen-Lange
Technical Manager
Advomatic LLC
Great White North Office
Canada

I totally agree, terminating

Fabianx's picture

I totally agree, terminating SSL before Varnish is key to a successful SSL deployment.

And with proper X-Forwarded-Proto headers, it is also simple to setup in Varnish / Apache using SetEnvIf.

Some ther SSL Varnish Links

mgifford's picture

I haven't done this yet, but looking into it again. Thinking about the EFF's efforts to encrypt the web. Anyways, here are some different links for consideration.

https://www.digitalocean.com/community/tutorials/how-to-configure-varnis...

http://blog.ajnicholls.com/varnish-apache-and-https/

http://edoceo.com/howto/nginx-varnish-ssl

http://mikkel.hoegh.org/2012/07/24/varnish-as-reverse-proxy-with-nginx-a...

I wish there were more details about Cloudflare's offerings:
https://www.cloudflare.com/plans

Has anyone used them?

Cloudflare SSL

gchaix's picture

I've played with Cloudflare's new "universal SSL" a bit. It works well enough, but requires a modern browser that supports SNI. So if you have legacy users (like we do) who are running XP or IE6 or old Android versions (≤ICS) it will throw certificate errors.

Overall, Cloudflare is pretty nice. However they seem to be overly-sensitive to backend load, returning "the site can't be reached, giving you an old cached page" more often than needed.

I know this is an old thread,

vegardx's picture

I know this is an old thread, but it's the first that comes up when you search for Drupal SSL and Varnish, so thought I'd try to clarify some things.

The 403 you're seeing is generated by Drupal, Varnish doesn't touch it since you've authenticated yourself. All cookies use the source address ("client ip") in combination with some other things I can't recall. But since we're going through reverse proxies we can't reliably use the source address anymore. This is what X-Forwarded-For is used for. For every reverse proxy the request goes through it adds the source address to it, creating a list. The first, or left-most IP-address will be the original source address.

Drupal seems to be configured correctly, you're telling it that it's behind a reverse proxy and to look for the X-Forwarded-For header. When you're connecting over TLS the connection first goes to Pound (Or Nginx, HAProxy, whatever) which terminates the connection and forwards the requests downstream. It adds a X-Forwarded-For header, as it should, with the correct source address.

But, and this is a gotcha, you can never unset the X-Forwarded-For header later and then find the original source address. The default logic in varnish is to do just as Pound or any other reverse proxy, by adding the source address to the list. But in this case you're overriding the default logic and setting the X-Forwarded-For header to client.ip, which in this case then means the Pound-server. So when you sign in the cookie won't be generated correctly, and you will be thrown out as soon as you try to do anything. The first requests goes through as normal since you authenticate in the same step.

In Varnish you also have to hash things based on X-Forwarded-Proto, to make sure that you differentiate the caches for TLS and non-TLS. Otherwise you're going to run into some interesting issues. And in Drupal you can also reliably just nuke all cookies that are not SESS or SSESS, since they are always present when needed (forms, authenticated users).

--
Vegard