S3 static site module

Andy Tawse's picture


I wanted to get feedback about whether releasing our module sounds like a good idea.

Our company has developed a module that publishes static Drupal nodes to S3 as they are saved or during a full site crawl. We've used this on a few projects where there is a need to cope with large traffic spikes (e.g. websites that support TV broadcasts) and where the content is relatively simple but there is still a need for content management. So quite niche but hopefully useful to some.

I've had a good look and can't find anything that does the same. The Boost module, which was our inspiration in some ways, didn't provide exactly what we needed - complete separation from Drupal (never hitting it) and the Boost module crawler is deprecated.

There's more detailed information here http://numiko.com/labs/2012/12/static-drupal-sites-for-high-traffic/

A future improvement could be to allow destinations for the static files other than S3 (pluggable modules).



DYdave's picture

Hi Andy,

Thanks a lot for your contribution and letting us know about this module, it certainly sounds like a great idea.

Just a quick reply to suggest you might consider cross posting this page in the High Performance DUG, I would surely be interested in hearing if any guys in this group would have additional ideas, experiences or questions.

Future evolutions surely sound very promising.

Hi @Andy, Thanks for

jygastaud's picture

Hi @Andy,

Thanks for feedback.
I think your module can be very helpful and it will be great if you share it.

Once done, you can update the page "Creating a static archive of a Drupal site" on wiki : http://drupal.org/node/27882

As an alternative to your project, I saw (but never tested yet) the "Save to FTP project" : http://drupal.org/project/savetoftp which looks to have an S3 option.

Some Ideas

mikeytown2's picture

"drupal_http_request() with an added querystring parameter"

Use a httprl non blocking request and set a custom header or user agent instead of using a query parameter.

Also look at the expire module for some simple crawler logic.


patchygreen's picture

Hi Andy

Just wondering if you have released this module to the community?

I'm looking for something that does exactly what you describe to push content to S3.


Hi, unfortunately we haven't

Andy Tawse's picture

Hi, unfortunately we haven't been able to go any further with this due to being too busy but hopefully soon.

Also interested...

tcowin's picture

Would be a great addition...

we're looking forward to it being released!

Any word on releasing the module

sohail.aslam@gmail.com's picture

I am also interested in using the module. Would it be possible to create a sandbox project to house the module?

very helpful

naveenvalecha's picture

It will be very helpful if you share the module.

Naveen Valecha

Contributed Module Ideas

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week