Nginx HTTP concat module

Posted by perusio on December 9, 2011 at 3:45pm

This might be of interest to you.

Of course there's a Drupal module to be made. This will void most of the issues around CSS and JS aggregation. Making it simpler and more
efficient. Nothing beats C for speed and efficiency.

The module should work also with modconcat, that is, if you're still using that type of legacy web server technology ;)

Comments

Oh, this is awesome, man. You

Posted by Fidelix on December 9, 2011 at 4:01pm

Oh, this is awesome, man.

You think this combination would beat Advagg?

Well there's

Posted by perusio on December 9, 2011 at 5:55pm

the minification aspect that is not handled by the module. Either you take the pains to always use a minified version of your CSS/JS or you have to implement it at the app level. Right now I'm not certain that CSS minification is a critical issue. Perhaps JS is, specially when you're using some heavy library. OTOH if you're using that library in a non-minified version, you're doing it wrong.

I'll start without minification and test. I'll compare both with advagg and core aggregation. Then decide if the cost/benefit of minification is worth the added complexity and effort in programming.

A whacky idead would be to turn everything assinchronous and use something like incron and run yui compressor for minification.

I kinda like the idea of ripping out the aggregation part from drupal. It will be faster and safer. Let's see.

IMO, the css minification is

Posted by Fidelix on December 9, 2011 at 6:20pm

IMO, the css minification is important (but as you said, not critical). I've seen cases with some heavy css themes where the results were really great.
Anyway, there is server-side GZIP to __help__ with that, right?

See this comparison:
http://stackoverflow.com/questions/807119/gzip-versus-minify

GZIP + Minify is still better.

Doing the minification with incron+yui would be a really interesting solution, though.
Me avise se precisar de ajuda. Valeu!

AdvAgg

Posted by mikeytown2 on December 10, 2011 at 3:01am

AdvAgg already compresses & minifies and builds the aggregates asynchronously. The HTML doesn't reference the aggregate until it exists (most of the time). nginx-http-concat looks like it doesn't take advantage of bundling; performance optimizations on the 2nd+ page the user visits (see the advagg readme for more info).

Not having Drupal deal with aggregation would be nice; but to take full advantage of it bundling you need a database of the files included. I've been thinking about creating/using a JS loader that has ALL css/js files ever needed for your Drupal site and putting it in one file. The browser then loads in the files/strings needed from the master file. Pushing out a new change creates a new master file. This would require jQuery to be loaded externally (AdvAgg JS CDN SubModule). This idea of mine would be a new AdvAgg sub module; I shouldn't have to modify AdvAgg core to handle a situation like this.

Another thing to consider is CSS Embedded Images. Be sure to use my fork of it if running with AdvAgg. Also say it works in this thread so we can get the module maintainer to think about accepting the patch http://drupal.org/node/1078060.

In short bring on the benchmarks :) But be sure to hit more then one page as most sites have slightly different CSS or JS files on different parts of the site, thus requiring different aggregates. Only hitting one page will favor having one aggregate; and if you only have one page, putting the JS & CSS inline is ideal (http://drupal.org/node/1311574) as that gets rid of 2 http requests.

I also agree that CSS minification isn't that important; JS minification is important though.

Hello mikeytown2 (Mike)

Posted by perusio on December 10, 2011 at 4:14am

Sure thing I will try those suggestions. Note that the Nginx module uses a cache, the open files cache which interfaces with the OS buffer cache.

So the database thing you're referring to I believe it's kinda of already baked in at an OS level. Here's my plan.

Disable all drupal based aggregation.
1. Copy all JS and CSS files to two different directories, just like your module and core based aggregation does. The reason for that is
  that it's easier to configure Nginx this way, since there's no reliance on regexes and the open files cache can be really effective. So
  there are strong incentives for placing the files in a common location.
2. Just insert a proper preprocessing function that creates the modified URIs so that the module is used. The only caveat here is that we must
  careful not to go overboard. Which is the rule in most real drupal sites with dozens and dozens of installed modules and the corresponding
  JS and CSS files. So the preprocessing functions need to concatenate the files such that we don't surpass the OS pagesize and thus the
  maximum allowed URI size.
3. Test it against core aggregation and advagg.
4. Note that nginx does gzipping quite efficiently hence there's very little incentive to perform the gzipping yourself. You can disable static
  gzipping so that we save on a stat() call, since if on and when not relying on try_files Nginx will try always first to find file.gz before file.

After some exploration

Posted by perusio on December 12, 2011 at 1:26am

of the issues involved there's no alternative to doing it as core does it right now. The issue being that since Drupal allows for CSS and JS files to be scattered throughout the file system you need to resolve all url and @import declarations before moving the files to a common dir.

So as much as I dislike the regexes that lurk in drupal_load_stylesheet and drupal_build_css_cache there's currently not much point in replicating that code in a contrib module for concatenating the files.

This would be

Posted by perusio on December 10, 2011 at 5:46pm

cool to use: http://wiki.nginx.org/NginxHttpStripModule. Unfortunately there's no state machine for supporting JS and CSS. It only deals with HTML :(

First baby steps

Posted by perusio on December 12, 2011 at 10:52am

of a D6 version are here. Stay tuned :)

Great, perusio. I thought

Posted by Fidelix on December 12, 2011 at 12:49pm

Great, perusio.
I thought (not sure why) you were doing this for D7, though...

Sure thing

Posted by perusio on December 12, 2011 at 5:51pm

there will be a D7 version also. It's not that hard, it's just a few functions that have different signatures. Also the .info file of course.

Whats the goal?

Posted by mikeytown2 on December 12, 2011 at 8:55pm

AdvAgg takes a total of 3.9ms when the cache is fully primed to do what it does on every page load (tack on another 5ms if this patch is not applied to Drupal). If your developing this in hopes of speeding up PHP that's a bad reason to do this. If your goal is to speed up the frontend OR to make the CSS/JS workflow even easier then that would be a worthy cause. ~4ms isn't a lot to worry about in terms of performance.

Not really

Posted by perusio on December 12, 2011 at 11:09pm

I mean, less PHP is definitely a plus. But my main beef is with regexes and the interaction of the cache with the aggregates.

I want to make it simpler and faster, if not faster at least as fast. No DB, no regexes, no cache interaction, just the plain simple files. Let's see.

Here's what I have right now

Posted by perusio on December 16, 2011 at 4:50am

The CSS files are shipped as one for each media type.
There's no file creation whatsoever. To resolve things like url() and @import those stylesheets get embedded.
The cache is completely decoupled from the aggregation procedure. I'll add some drupal_static support to the function that resolves the paths.
But that's it.
I'm having issues with JS related to CTools. I know that you Mike, also had problems with CTools similar problems. The problems
are also showing up on admin. So there's something that needs fixing to get them both to work properly.
Since I'm embedding the stylesheets with images it would be trivial to get a base64 of it embed all images on the page and
still improve performance. My only beef with that is dealing with IE, I know it has issues with it. I'll look into your fork of the CSS Embedded Images.
It's trivial to provide, and I will do it, a field for specifying a different host for serving up CSS and JS. Also bear in mind that the HTTP concatenator
handles any type. Is just that you need to specify the MIME type of the asset you want to concatenate via concat_types.

I think that by the weekend there will be an alpha release. The most pressing thing right now is fixing admin and CTools.

Just a quick update

Posted by perusio on January 20, 2012 at 7:48pm

I'm derelict on this project. The good new is that two more are coming before I fix this and have a working release. Stay tuned :)