Nginx and ESI

Posted by superfedya on May 26, 2012 at 4:51am

Hi,

I use Perusio's Nginx config (https://github.com/perusio/drupal-with-nginx), there is any way to use it with ESI module? (http://drupal.org/project/esi)

Or there is no real advantages to use ESI over http://drupal.org/project/cache_warmer? Installation of Cache Warmer looks too complicated, ESI must be easier.

Thanks!

Comments

Nginx

Posted by perusio on June 1, 2012 at 7:49am

in core has no support for ESI. It has however SSI . Check this thread: http://groups.drupal.org/node/197478

2.x branch of ESI

Posted by mikeytown2 on June 1, 2012 at 7:03pm

The 2.x branch of the ESI module supports SSI. If it doesn't open an issue and let's get that bug fixed!

I need to modify Perusio's

Posted by superfedya on June 1, 2012 at 7:18pm

I need to modify Perusio's great nginx config?If yes how?

Thanks

Just make sure you have SSI

Posted by omega8cc on June 1, 2012 at 9:02pm

Just make sure you have SSI compiled in and then add required location. See this example from BOA: http://drupalcode.org/project/barracuda.git/blob/HEAD:/aegir/conf/nginx_...

There's nothing complicated

Posted by perusio on June 1, 2012 at 10:08pm

with cache_warmer. It's quite simple. You just say what you want to have cached and the frequency of the crawling. Done!

ESI/SSI are different beasts. They are to be used when you want to take advantage of having a template that is created around different expiration times of the content.

You can get 80% of the benefits of SSI by caching static files. Which means that you need to proxy_pass your static content. In a nutsheel you move the complexity from the template layer to the server config layer. There's no free lunches, no matter what the sign outside the saloon says :)

EDIT: The advantage of proxy passing the static files is that you doesn't require to touch your code or install any drupal module.

Any idea how to install a

Posted by superfedya on June 1, 2012 at 10:36pm

Any idea how to install a crawler for cache warmer?
--crawler-service-uri=http://crawl.example.com/cache-warmer

I installed nginx with lua, what I need to do next to setup a crawler domain?

Thanks

You need to

Posted by perusio on June 2, 2012 at 12:03am

create a vhost that is bound to the loopback and that receives the requests from the drush command.

Note that you only need that if you're doing parallel requests.

Check this.

Thanks! About this part:

Posted by superfedya on June 2, 2012 at 2:02am

Thanks!

About this part:

     listen 127.0.0.1:8890;
     server_name cache_warmer.no-ip; # bogus TLD for safety

I must change cache_warmer.no-ip to crawl.mysite.com?

Or I can use something like:
drush cache-warmer --updated-last='-2 days' --parallel=20 --crawler-service-uri=http://127.0.0.1:8890/cache-warmer --timeout=15 --hub-pages-file=hub_pages.txt http://mysite.com

Post 8890 must be open in iptables?

Parallel requests is much faster?

Yes

Posted by perusio on June 2, 2012 at 10:26am

Parallel requests replenish the cache faster than single requests. They do however put a strain on your site. Investigate to know how many requests you can do in parallel to the site without any external caching.

You need to have that port open for your loopback.

So, I don't need to change

Posted by superfedya on June 2, 2012 at 12:59pm

So, I don't need to change this line?

     listen 127.0.0.1:8890;
     server_name cache_warmer.no-ip; # bogus TLD for safety

and I can use the command like this after?
drush cache-warmer --updated-last='-2 days' --parallel=20 --crawler-service-uri=http://127.0.0.1:8890/cache-warmer --timeout=15 --hub-pages-file=hub_pages.txt http://mysite.com

Thanks

no

Posted by perusio on June 2, 2012 at 3:16pm

you can even use:

drush cache-warmer --updated-last='-2 days' --parallel=20 --crawler-service-uri=http://cache_warmer.no-ip/cache-warmer --timeout=15 --hub-pages-file=hub_pages.txt http://mysite.com

You'll have to add:

127.0.0.1 cache_warmer.no-ip
::1 cache_warmer.no-ip # IPv6

to your /etc/hosts.

Yes: no changes required.

Posted by perusio on June 2, 2012 at 3:18pm

you can even use:

drush cache-warmer --updated-last='-2 days' --parallel=20 --crawler-service-uri=http://cache_warmer.no-ip/cache-warmer --timeout=15 --hub-pages-file=hub_pages.txt http://mysite.com

You'll have to add:

127.0.0.1 cache_warmer.no-ip #IPv4
::1 cache_warmer.no-ip # IPv6

to your /etc/hosts.

I can't vow for your site withstanding 20 parallel requests, though ;)

Thankslisten 127.0.0.1

Posted by superfedya on June 2, 2012 at 5:08pm

Single thead works fine:
drush cache-warmer --updated-last='-2 days' --hub-pages-file=hub_pages.txt http://mysite.com

But parallel no:
drush cache-warmer --updated-last='-2 days' --parallel=10 --crawler-service-uri=http://cache_warmer.no-ip/cache-warmer --timeout=15 --hub-pages-file=hub_pages.txt http://mysite.com

Error:

[{"timestamp":1338655244,"reply":"<html>\r\n<head><title>301 Moved Permanently<\/title><\/head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently<\/h1><\/center>\r\n<hr><center>nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":301,"time":0.000288},{"timestamp":1338655244,"reply":"<html>\r\n<head><title>301 Moved Permanently<\/title><\/head>\r\n<body bgcolor=\"white\">\r\n<center><h1>301 Moved Permanently<\/h1><\/center>\r\n<hr><center>nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":301,"time":6.6e-5}]

I ping: ping cache_warmer.no-ip and it available.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.022 ms

Or there is another mistake?

# -- mode: nginx; mode: flyspell-prog;  ispell-local-dictionary: "american" --
### Configuration for the cache warmer (crawler).

server {
    ## This is bound to the loopback interface for both IPv4 and IPV6.
    listen 127.0.0.1:8890;
    server_name cache_warmer.no-ip; # bogus TLD for safety

    limit_conn arbeit 192;

    access_log /var/log/nginx/cache_warmer.access.log;
    error_log /var/log/nginx/cache_warmer.error.log;

    ## Path to the Lua scripts implementing the parallel crawker.
    root /root/.drush/cache_warmer/lua;
    index index.html;

    ## The catch all location.
    location / {
        empty_gif;
    }

    ## The crawler locations.
    location = /cache-warmer {
        content_by_lua_file '$document_root/cache_warmer_requests.lua';
    }

    location = /parallel-reqs {
        content_by_lua_file '$document_root/cache_warmer_client.lua';
    }

} # server

or lua location:
http://i50.tinypic.com/54u6he.jpg

You forgot the port

Posted by perusio on June 3, 2012 at 10:49am

It's:

drush cache-warmer --updated-last='-2 days' --parallel=10 --crawler-service-uri=http://cache_warmer.no-ip:8890/cache-warmer --timeout=15 --hub-pages-file=hub_pages.txt http://mysite.com

Same Problem

Posted by EndEd on June 3, 2012 at 3:45am

I´m having exactly the same problem as superfedya... but in my case is a bunch of 302 status

[{"timestamp":1338694714,"reply":"","status":302,"time":0.380651},{"timestamp":1338694714,"reply":"","status":302,"time":0.378451},{"timestamp":1338694714,"reply":"","status":302,"time":0.166273},{"timestamp":1338694714,"reply":"","status":302,"time":0.156667},{"timestamp":1338694714,"reply":"","status":302,"time":0.364035},]

Is there anything that we are not taking in account?

What does

Posted by perusio on June 3, 2012 at 10:49am

curl -I http://cache_warmer.no-ip:8890/cache-warmer

gives?

Now

Posted by superfedya on June 3, 2012 at 1:13pm

Now is:

[{"timestamp":1338728960,"reply":"<html>\r\n<head><title>500 Internal Server Error<\/title><\/head>\r\n<body bgcolor=\"white\">\r\n<center><h1>500 Internal Server Error<\/h1><\/center>\r\n<hr><center>nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":500,"time":0.012552},{"timestamp":1338728960,"reply":"<html>\r\n<head><title>500 Internal Server Error<\/title><\/head>\r\n<body bgcolor=\"white\">\r\n<center><h1>500 Internal Server Error<\/h1><\/center>\r\n<hr><center>nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":500,"time":0.000805},{"timestamp":1338728960,"reply":"<html>\r\n<head><title>500 Internal Server Error<\/title><\/head>\r\n<body bgcolor=\"white\">\r\n<center><h1>500 Internal Server Error<\/h1><\/center>\r\n<hr><center>nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":500,"time":0.000145},{"timestamp":1338728960,"reply":"<html>\r\n<head><title>500 Internal Server Error<\/title><\/head>\r\n<body bgcolor=\"white\">\r\n<center><h1>500 Internal Server Error<\/h1><\/center>\r\n<hr><center>nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":500,"time":0.000139}]

curl -I http://cache_warmer.no-ip:8890/cache-warmer

HTTP/1.1 500 Internal Server Error
Server: nginx
Date: Sun, 03 Jun 2012 13:10:30 GMT
Content-Type: text/html
Content-Length: 186
Connection: close

Thanks

That's a Lua issue

Posted by perusio on June 3, 2012 at 2:12pm

Do you have LuaSocket installed? In Debian is in the package lua-socket. Are you using Lua 5.1?

No, I use: LuaJIT. Is better

Posted by superfedya on June 3, 2012 at 3:01pm

No, I use: LuaJIT. Is better to use lua 5.1?

I just installed lua-socket. What I need to do next?

I had LuaJIT too and just

Posted by EndEd on June 3, 2012 at 3:15pm

I had LuaJIT too and just installed LuaSocket

Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
> socket = require("socket")
> print(socket._VERSION)
LuaSocket 2.0.2

Still getting with -> curl -I http://cache_warmer.no-ip:8890/cache-warmer

HTTP/1.1 500 Internal Server Error
Server: nginx
Date: Sun, 03 Jun 2012 15:06:12 GMT
Content-Type: text/html
Content-Length: 186
Connection: close

any thoughts?

I just reinstalled Lua 5.1

Posted by superfedya on June 3, 2012 at 5:03pm

It was wrong location.

I changed it (and removed the port, it doens't work with this line in host:
127.0.0.1 cache_warmer.no-ip #IPv4
I tried without results:
127.0.0.1:8890 cache_warmer.no-ip
127.0.0.1 cache_warmer.no-ip:8890):

server {
    ## This is bound to the loopback interface for both IPv4 and IPV6.
    listen 127.0.0.1;
    server_name cache_warmer.no-ip; # bogus TLD for safety

    limit_conn arbeit 192;

    access_log /var/log/nginx/cache_warmer.access.log;
    error_log /var/log/nginx/cache_warmer.error.log;

    ## Path to the Lua scripts implementing the parallel crawker.
    root /root/.drush/cache_warmer;
    index index.html;

    ## The catch all location.
    location / {
        empty_gif;
    }

    ## The crawler locations.
    location = /cache-warmer/lua {
        content_by_lua_file '$document_root/cache_warmer_requests.lua';
    }

    location = /parallel-reqs/lua {
        content_by_lua_file '$document_root/cache_warmer_client.lua';
    }

} # server

curl -I http://cache_warmer.no-ip/cache-warmer

HTTP/1.1 200 OK
Server: nginx
Date: Sun, 03 Jun 2012 16:50:59 GMT
Content-Type: image/gif
Content-Length: 43
Last-Modified: Mon, 28 Sep 1970 06:00:00 GMT
Connection: keep-alive
Keep-Alive: timeout=10

drush cache-warmer --updated-last='-2 days' --parallel=10 --crawler-service-uri=http://cache_warmer.no-ip/cache-warmer --timeout=15 --hub-pages-file=hub_pages.txt http://mysite.com

[{"timestamp":1338742591,"reply":"<html>\r\n<head><title>405 Not Allowed<\/title><\/head>\r\n<body bgcolor=\"white\">\r\n<center><h1>405 Not Allowed<\/h1><\/center>\r\n<hr><center>nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":405,"time":0.000278},{"timestamp":1338742591,"reply":"<html>\r\n<head><title>405 Not Allowed<\/title><\/head>\r\n<body bgcolor=\"white\">\r\n<center><h1>405 Not Allowed<\/h1><\/center>\r\n<hr><center>nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":405,"time":0.000101},{"timestamp":1338742591,"reply":"<html>\r\n<head><title>405 Not Allowed<\/title><\/head>\r\n<body bgcolor=\"white\">\r\n<center><h1>405 Not Allowed<\/h1><\/center>\r\n<hr><center>nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":405,"time":4.6e-5},{"timestamp":1338742591,"reply":"<html>\r\n<head><title>405 Not Allowed<\/title><\/head>\r\n<body bgcolor=\"white\">\r\n<center><h1>405 Not Allowed<\/h1><\/center>\r\n<hr><center>nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":405,"time":4.2e-5},{"timestamp":1338742591,"reply":"<html>\r\n<head><title>405 Not Allowed<\/title><\/head>\r\n<body bgcolor=\"white\">\r\n<center><h1>405 Not Allowed<\/h1><\/center>\r\n<hr><center>nginx<\/center>\r\n<\/body>\r\n<\/html>\r\n","status":405,"time":4.1e-5}]

Nginx 1.2.0, Lua 5.1

You're

Posted by perusio on June 3, 2012 at 5:43pm

forgetting the port. Hence the 405. You're trying to POST to a location that doesn't support it.

Ok

Posted by perusio on June 3, 2012 at 5:36pm

I send you both on a wild goose chase. Solly :( In fact the parallel mode sends a POST to the Lua location. So a HEAD gives a 500, since it generates a Lua error.

Here's how to test it from the command line:

curl -X POST --data 'base_uri="http://d7"&data0="content/exerci-tum-vulpes"' http://cache_warmer.no-ip:8890/cache-warmer

where http://d7 is a D7 test site I have in my machine and data0 is a URI of that site.
Replace those values by your site base_uri and a URI for a content. You should get 200 as reply.

Also you should check your

Posted by perusio on June 3, 2012 at 5:41pm

server logs to see if the cache_warmer is hitting the URIs. Here's what I get from a test:

::ffff:127.0.0.1 - - [03/Jun/2012:19:22:33 +0200] "HEAD /content/illum-iustum-minim-neque-paulatim HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:34 +0200] "HEAD /content/cogo-magna HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:34 +0200] "HEAD / HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:34 +0200] "HEAD /content/nunc-suscipit HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:34 +0200] "HEAD /content/gemino-nunc-odio HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:34 +0200] "HEAD /content/conventio-loquor-nisl-wisi HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:34 +0200] "HEAD /content/immitto-paratus-ratis HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:34 +0200] "HEAD /content/damnum-laoreet-lucidus-minim-torqueo HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:35 +0200] "HEAD /content/exputo-iaceo-loquor-macto-saluto-venio HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:35 +0200] "HEAD /content/lenis-luctus-neo-quae HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:35 +0200] "HEAD /content/lucidus-premo-roto HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:35 +0200] "HEAD /content/patria-refoveo HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:35 +0200] "HEAD /content/caecus-melior-obruo HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:35 +0200] "HEAD /content/esca-pecus-praemitto HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:35 +0200] "HEAD /content/augue-plaga-sudo-suscipit-ullamcorper HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:36 +0200] "HEAD /content/erat-exputo-neo-oppeto-vicis-ymo HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:36 +0200] "HEAD /content/et-illum-sino-wisi HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:36 +0200] "HEAD /content/cui-sudo-ulciscor HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:36 +0200] "HEAD /content/fere-refoveo HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:36 +0200] "HEAD /content/appellatio-decet-illum-scisco-te HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
::ffff:127.0.0.1 - - [03/Jun/2012:19:22:36 +0200] "HEAD /content/abigo-caecus-modo-os-suscipere HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"

ThanksIf the url is:

Posted by superfedya on June 3, 2012 at 6:41pm

Thanks

If the url is: http://madfanboy.com/puteshestvie-dlinnoyu-v-vechnost
What is the URI? http://madfanboy.com?

curl -X POST --data 'base_uri="http://madfanboy.com"&="puteshestvie-dlinnoyu-v-vechnost"' http://cache_warmer.no-ip/cache-warmer

logs:
2012/06/03 18:41:02 [error] 28910#0: *443 Failed to load Lua inlined code: cannot open /root/.drush/cache_warmer/lua/cache_warmer_requests.lua: Permission denied, client: 127.0.0.1, server: cache_warmer.no-ip, request: "HEAD /cache-warmer HTTP/1.1", host: "cache_warmer.no-ip"

The file chmod is 644 and the owner is root. Nginx works as www-data. I changed the owner to www-data and chmod to 755. No results.

No need

Posted by perusio on June 3, 2012 at 7:14pm

to set it to 755. 644 suffices. It's not an executable.

The problem, it seems, is that you installed the command under the root home. And if you're running the server as www-data he wont be able to read stuff under /root. So I suggest you move that to a directory where the code can be read by the server.

I'll have to adjust the root directive on the cache_warmer.no-ip vhost.

My .drush directory is under

Posted by superfedya on June 3, 2012 at 8:09pm

My .drush directory is under /root (via standard pear install drush/drush).
I put cache_warmer directory there.

So, I need to reinstall Drush and put in under another directory? Like a /home or /var/www? Or I can just move cache_warmer folder?

No you can

Posted by perusio on June 3, 2012 at 8:28pm

leave the drush command there. I find strange that centOS installs drush under root. Anyway you just need to move the lua files to a location where the server can read them.

It can be /home/<youruser>/lua/cache_warmer. Just make sure that the server user can read it.

I put the warmer to:

Posted by superfedya on June 3, 2012 at 9:48pm

I put the warmer to: /var/www/sites/all/drush
It works?

drush cache-warmer --updated-last='-7 days' --parallel=10 --crawler-service-uri=http://cache_warmer.no-ip/cache-warmer --timeout=15 --hub-pages-file=hub_pages.txt http://mysite.com >> /var/log/cache_warmer.log

Warmer log:

[{"timestamp":1338759076,"reply":"200\n200\n200\n200\n200\n200\n200\n200\n200\n200\n","status":200,"time":2.956546},{"timestamp":1338759076,"reply":"200\n200\n200\n200\n200\n200\n200\n200\n200\n200\n","status":200,"time":0.72995},{"timestamp":1338759076,"reply":"200\n200\n200\n200\n200\n200\n200\n200\n200\n200\n","status":200,"time":1.807806},{"timestamp":1338759076,"reply":"200\n200\n200\n200\n200\n200\n200\n200\n200\n200\n","status":200,"time":2.123003},{"timestamp":1338759076,"reply":"200\n200\n200\n200\n200\n200\n200\n200\n200\n200\n","status":200,"time":7.04496},{"timestamp":1338759076,"reply":"200\n200\n200\n200\n200\n200\n200\n200\n200\n","status":200,"time":7.124692}]

Single thread log is different:

{"":{"timestamp":1338759560,"status":200,"time":0.000993},"news":{"timestamp":1338759560,"status":200,"time":0.000132},"madnews":{"timestamp":1338759560,"status":200,"time":0.000137},"stories":{"timestamp":1338759560,"status":200,"time":0.220676},"madfaqs":{"timestamp":1338759560,"status":200,"time":0.167722},"madpictures":{"timestamp":1338759560,"status":200,"time":0.182087},"madcomics":{"timestamp":1338759560,"status":200,"time":0.188833},"blog":{"timestamp":1338759560,"status":200,"time":0.211184},"mini":{"timestamp":1338759560,"status":200,"time":0.498953},"e3-2012-madfanboycom-coverage-raspisanie":{"timestamp":1338759560,"status":200,"time":0.000155},"mifist":{"timestamp":1338759560,"status":200,"time":0.236808},"kriticheskie-oshibki-
...

Is normal?
Nginx acces logs:
127.0.0.1 - - [03/Jun/2012:23:42:22 +0200] "POST /cache-warmer HTTP/1.1" 200 51 "-" "-"

How this command will looks for this page http://madfanboy.com/spisok-sobytiy-e3-2012?

curl -X POST --data 'base_uri="http://d7"&data0="content/exerci-tum-vulpes"' http://cache_warmer.no-ip:8890/cache-warmer

Thanks Perusio

It seems correct

Posted by perusio on June 3, 2012 at 10:32pm

the final check is to be done on your site access logs, i.e., the access logs for the vhost serving madfanboy.com.

Note that the single crawler does the HEAD request directly, while the parallel one works by POSTing the URIs to be crawled to the Lua powered location that then uses the Nginx Lua API to issue non-blocking parallel requests.

Thank you very

Posted by superfedya on June 4, 2012 at 1:55am

Thank you very much!

[04/Jun/2012:03:54:08 +0200] "HEAD /lpb-vita-skrinshoty HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
[04/Jun/2012:03:54:08 +0200] "HEAD /prodazhi-massefffect-3-i-drugih-igr-po-dannym-npd HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
[04/Jun/2012:03:54:08 +0200] "HEAD /cenovaya-politika-budushchego HTTP/1.1" 200 0 "-" "Nginx Cache Warmer"
...

It works!

EndEd, did you resolve the problem also? If not ill write a step-by-step instruction :)

Well i´m still struggling

Posted by EndEd on June 4, 2012 at 1:02pm

Well i´m still struggling with the Lua paths. In the crawler vhost access log i get this

127.0.0.1 - - [04/Jun/2012:13:15:08 +0100] "POST /x-crawler-drush HTTP/1.1" 200 76 "-" "-"
127.0.0.1 - - [04/Jun/2012:13:15:08 +0100] "POST /x-crawler-drush HTTP/1.1" 200 76 "-" "-"
...

...but in the crawler vhost error log i get this

2012/06/04 13:15:08 [error] 1529#0: *1 lua handler aborted: runtime error: ...all/drush/cache_warmer/lua/cache_warmer_client.lua:34: module 'socket.http' not found:
  no field package.preload['socket.http']
  no file './socket/http.lua'
  no file '/usr/local/share/luajit-2.0.0-beta10/socket/http.lua'
   no file '/usr/local/share/lua/5.1/socket/http.lua'
   no file '/usr/local/share/lua/5.1/socket/http/init.lua'
  no file './socket/http.so'
   no file '/usr/local/lib/lua/5.1/socket/http.so'
  no file '/usr/local/lib/lua/5.1/loadall.so'
  no file './socket.so'
    no file '/usr/local/lib/lua/5.1/socket.so'
   no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
 [C]: in function 'require'
   ...all/drush/cache_warmer/lua/cache_warmer_client.lua:34: in function <...all/drushcache_warmer/lua/cache_warmer_client.lua:1>, client: 127.0.0.1, server: crawler.nx, request: "POST /cache-warmer HTTP/1.1", subrequest: "/parallel-reqs", host: "crawler.nx:8890"
2012/06/04 13:15:08 [error] 1529#0: *1 lua handler aborted: runtime error: ...all/drush/cache_warmer/lua/cache_warmer_client.lua:34: module 'socket.http' not found:
    no field package.preload['socket.http']
  no file './socket/http.lua'
  no file '/usr/local/share/luajit-2.0.0-beta10/socket/http.lua'
   no file '/usr/local/share/lua/5.1/socket/http.lua'
   no file '/usr/local/share/lua/5.1/socket/http/init.lua'
  no file './socket/http.so'
   no file '/usr/local/lib/lua/5.1/socket/http.so'
  no file '/usr/local/lib/lua/5.1/loadall.so'
  no file './socket.so'
    no file '/usr/local/lib/lua/5.1/socket.so'
   no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
 [C]: in function 'require'
...

with dpkg --get-selections | grep lua i get

liblua5.1-socket2                               install
lua5.1                                          install

dpkg -L lua5.1 returns

/.
/usr
/usr/bin
/usr/bin/lua5.1
/usr/bin/luac5.1
/usr/share
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/lua5.1.1.gz
/usr/share/man/man1/luac5.1.1.gz
/usr/share/doc
/usr/share/doc/lua5.1
/usr/share/doc/lua5.1/copyright
/usr/share/doc/lua5.1/README.Debian.gz
/usr/share/doc/lua5.1/changelog.Debian.gz

and dpkg -L liblua5.1-socket2 gives

/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/liblua5.1-socket2
/usr/share/doc/liblua5.1-socket2/copyright
/usr/share/doc/liblua5.1-socket2/README
/usr/share/doc/liblua5.1-socket2/changelog.Debian.gz
/usr/share/lua
/usr/share/lua/5.1
/usr/share/lua/5.1/socket
/usr/share/lua/5.1/socket/url.lua
/usr/share/lua/5.1/socket/tp.lua
/usr/share/lua/5.1/socket/ftp.lua
/usr/share/lua/5.1/socket/http.lua
/usr/share/lua/5.1/socket/smtp.lua
/usr/share/lua/5.1/socket.lua
/usr/share/lua/5.1/ltn12.lua
/usr/share/lua/5.1/mime.lua
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/lua
/usr/lib/x86_64-linux-gnu/lua/5.1
/usr/lib/x86_64-linux-gnu/lua/5.1/socket
/usr/lib/x86_64-linux-gnu/lua/5.1/mime
/usr/lib/x86_64-linux-gnu/liblua5.1-mime.so.2.0.0
/usr/lib/x86_64-linux-gnu/liblua5.1-socket.so.2.0.0
/usr/lib/x86_64-linux-gnu/liblua5.1-unix.so.2.0.0
/usr/lib/x86_64-linux-gnu/liblua5.1-unix.so.2
/usr/lib/x86_64-linux-gnu/lua/5.1/socket/unix.so
/usr/lib/x86_64-linux-gnu/lua/5.1/socket/core.so
/usr/lib/x86_64-linux-gnu/lua/5.1/mime/core.so
/usr/lib/x86_64-linux-gnu/liblua5.1-mime.so.2
/usr/lib/x86_64-linux-gnu/liblua5.1-socket.so.2

also echo $PATH gives

/usr/share/lua/5.1/socket:/usr/share/lua/5.1:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

I'm on Ubuntu 12.04

Any ideas why is trying to look to http.lua in the wrong place?

Somehow you gave a different

Posted by perusio on June 4, 2012 at 1:18pm

path to the lua libraries when compiling Nginx. That's what I suspect. Hence it cannot find the library.
What does:

ldd /usr/sbin/nginx | grep lua

gives?

it gives libluajit-5.1.so.2

Posted by EndEd on June 4, 2012 at 1:24pm

it gives libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x00007fd63a09d000)

i purged luajit... :/ Do i need to recompile nginx? could you tell me if i need to do something else before recompile?

well i tryed to recompile and

Posted by EndEd on June 4, 2012 at 1:36pm

well i tryed to recompile and gives me this error
./configure: error: ngx_http_lua_module requires the Lua library.

Any insight will be much apreciated :)

In the nginx wiki i find

Posted by EndEd on June 4, 2012 at 1:45pm

In the nginx wiki i find

    # tell nginx's build system where to find lua:
    export LUA_LIB=/path/to/lua/lib
    export LUA_INC=/path/to/lua/include

before the ./configure phase

could you tell me where i suppose to point taking in account my liblua5.1-socket2 & lua5.1 installation paths?

Add this to the

Posted by perusio on June 4, 2012 at 1:47pm

top of the config file for the Lua module :

LUAJIT_INC="/usr/include/luajit-2.0"
LUAJIT_LIB="/usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2.0.0"

should i add that taking in

Posted by EndEd on June 4, 2012 at 1:53pm

should i add that taking in account i dont have luajit installed? only liblua5.1-socket2 & lua5.1?

yep... get this adding module

Posted by EndEd on June 4, 2012 at 1:58pm

yep... get this

adding module in ./modules/nginx-lua-module
checking for LuaJIT library in /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2.0.0 and /usr/include/luajit-2.0 (specified by the LUAJIT_LIB and LUAJIT_INC env) ... not found
./configure: error: ngx_http_lua_module requires the Lua or LuaJIT library and LUAJIT_LIB is defined as /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2.0.0 and LUAJIT_INC /usr/include/luajit-2.0, but we cannot find LuaJIT there.

could you tell me where i suppose to point taking in account my liblua5.1-socket2 & lua5.1 installation paths?

you could see my paths here:

http://groups.drupal.org/node/233858#comment-765403

It's better to

Posted by perusio on June 4, 2012 at 2:12pm

use the LuaJIT. Just do:

aptitude install luajit

well im totally lost

Posted by EndEd on June 4, 2012 at 2:28pm

well im totally lost here...

aptitude install luajit installs 'libluajit-5.1-common' & 'luajit'

'dpkg -L libluajit-5.1-common' gives:

/.
/usr
/usr/share
/usr/share/luajit-2.0.0-beta9
/usr/share/luajit-2.0.0-beta9/jit
/usr/share/luajit-2.0.0-beta9/jit/bc.lua
/usr/share/luajit-2.0.0-beta9/jit/v.lua
/usr/share/luajit-2.0.0-beta9/jit/dump.lua
/usr/share/luajit-2.0.0-beta9/jit/dis_x86.lua
/usr/share/luajit-2.0.0-beta9/jit/dis_x64.lua
/usr/share/luajit-2.0.0-beta9/jit/dis_arm.lua
/usr/share/luajit-2.0.0-beta9/jit/dis_ppc.lua
/usr/share/luajit-2.0.0-beta9/jit/bcsave.lua
/usr/share/luajit-2.0.0-beta9/jit/vmdef.lua
/usr/share/doc
/usr/share/doc/libluajit-5.1-common
/usr/share/doc/libluajit-5.1-common/copyright
/usr/share/doc/libluajit-5.1-common/changelog.Debian.gz

dpkg -L luajit gives

/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/luajit
/usr/share/doc/luajit/copyright
/usr/share/doc/luajit/changelog.Debian.gz
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/luajit.1.gz
/usr/bin
/usr/bin/luajit-2.0.0-beta9

... I suppose to change

LUAJIT_INC="/usr/include/luajit-2.0" with ?
LUAJIT_LIB="/usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2.0.0" with ?

i installed liblua5.1-socket2 & lua5.1 and get rid off luajit because was the only way i found that i get lua with luasocket

Oops

Posted by perusio on June 4, 2012 at 2:38pm

solly :( that's the binary JIT compiler for Lua 5.1. You need the library and the include files.

aptitude install libluajit-5.1-2
aptitude install libluajit-5.1-dev

well im in square one...

Posted by EndEd on June 4, 2012 at 3:16pm

well im in square one... after recompile nginx i get this from the crawler vhost error log

  ...all/drush/cache_warmer/lua/cache_warmer_client.lua:34: in function <...all/drush/cache_warmer/lua/cache_warmer_client.lua:1>, client: 127.0.0.1, server: crawler.nx, request: "POST /cache_warmer HTTP/1.1", subrequest: "/parallel-reqs", host: "crawler.nx:8890"
2012/06/04 16:03:41 [error] 9503#0: *33 lua handler aborted: runtime error: ...all/drush/cache_warmer/lua/cache_warmer_client.lua:34: module 'socket.http' not found:
  no field package.preload['socket.http']
  no file './socket/http.lua'
  no file '/usr/local/share/luajit-2.0.0-beta10/socket/http.lua'
   no file '/usr/local/share/lua/5.1/socket/http.lua'
   no file '/usr/local/share/lua/5.1/socket/http/init.lua'
  no file './socket/http.so'
   no file '/usr/local/lib/lua/5.1/socket/http.so'
  no file '/usr/local/lib/lua/5.1/loadall.so'
  no file './socket.so'
    no file '/usr/local/lib/lua/5.1/socket.so'
   no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
 [C]: in function 'require'

could you tell me how to install luasocket so it could get installed in those searched paths?

Ok

Posted by perusio on June 4, 2012 at 4:04pm

lets go by steps:

What does ldd /usr/sbin/nginx | grep lua gives?
And dpkg -l *socket* | grep lua?

ldd /usr/sbin/nginx | grep

Posted by EndEd on June 4, 2012 at 4:37pm

ldd /usr/sbin/nginx | grep lua gives:

libluajit-5.1.so.2 => /usr/local/lib/libluajit-5.1.so.2 (0x00007ff25ee54000)

dpkg -l socket | grep lua gives:

ii liblua5.1-socket2 2.0.2-6 TCP/UDP socket library for Lua 5.1
ii luasocket 2.0.2-6 TCP/UDP socket library for Lua 5.0
un luasocket-dev (no description available)

I finally get it working after copying the content from /usr/lib/x86_64-linux-gnu/lua to /usr/local/lib/lua

I suppose thats not the right way to install all this :(

No

Posted by perusio on June 4, 2012 at 7:31pm

in fact. Where are you getting your nginx from? If you install from a source package then you can customize it. But I'm not aware of all the details regarding Nginx on Ubuntu.

If this suits you here's my debian rules file that I use to build my Nginx package.

I installed it from source

Posted by EndEd on June 4, 2012 at 9:03pm

I installed it from source using this:

./configure --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-log-path=/var/log/nginx/access.log --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid --with-debug --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_flv_module --with-http_geoip_module --with-http_gzip_static_module --with-http_realip_module --with-http_stub_status_module --with-http_ssl_module --with-http_sub_module --with-ipv6 --with-file-aio --with-http_secure_link_module --with-http_mp4_module --with-http_image_filter_module --add-module=./modules/nginx-upstream-fair-module --add-module=./modules/nginx-upload-progress-module --add-module=./modules/nginx-http-auth-request-module --add-module=./modules/nginx-sticky-module --add-module=./modules/nginx-http-footer-filter-module --add-module=./modules/nginx-http-concat-module --add-module=./modules/nginx-devel-kit-module --add-module=./modules/nginx-lua-module

that configure portion i taked from your debian rules i was aware off. I´m using your nginx config for drupal btw. we are trying to use your module "cache warmer" to warm boost caches.

I've been trying to get Nginx

Posted by btmash on June 6, 2012 at 11:47pm

I've been trying to get Nginx and SSI (I'm coming from the stock distribution of Aegir) to work with it other using the esi module without any luck so far (I didn't want to start a new thread since this one seems to have started in that path). The ESI module seems to be doing things correctly based on the kind of code that I see (). And while it does not work, when I have not put in the SSI pieces into the config, I am able to access my url and it provides the piece that should get rendered out. However, when I add to my config the code that was on http://groups.drupal.org/node/197478 with a few modifications:

  ssi on;
  ssi_silent_errors on;
  location ~ ^/(?<esi>esi/.*)$ {
    internal;
    include /etc/nginx/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root/index.php;
    fastcgi_param QUERY_STRING q=$esi;
    fastcgi_pass 127.0.0.1:9000;

    set $nocache "";
    if ($http_cookie ~ SESS) {
      set $nocache "Y";
    }
    fastcgi_cache speed;
    fastcgi_cache_key $scheme$host$uri$args;
    fastcgi_ignore_headers Expires;
    fastcgi_cache_bypass $nocache;
    fastcgi_no_cache $nocache;

    add_header X-nginx-Cache $upstream_cache_status;
    expires epoch;
  }

It was initially displaying a page where the include line was (which, when rendered, showed the page where the include line was, and so one...making for one crazy page). So it seemed like the ssi on rules needed to go into the location directive. When I did that, the page, however, still didn't show (nothing shows up in the error logs. I removed the internal rule to see the url.
However, when I tried to access the url that was working before, it didn't show anything now and I'm quite stumped. I've tried using the barracuda config, the nginx-for-drupal config along with my own silly config changes but nothing seems to work. Any guidance on the issue would be very welcome.

Comments

Group organizers

New groups

Group notifications