Amazon Web Services (S3, EC2)

Amazon Web Services Discussion in Boulder

greggles — Sat, 31 May 2008 16:51:41 +0000

Per coloradostartups.com an Amazon Web Services evangelist will be in Colorado this coming week to give a talk and evangelize.

From the post:

Mike will be in town visiting TechStars on Tuesday, June 3rd. While he’s here, and since he’s such a nice guy, he’s also holding a get-together at The Cup in Boulder for anyone interested in Amazon Web Services. This will be at 10am on Tuesday. This is the guy to talk to if you want to learn more

If you're reading this as part of the Amazon Web Services group (as opposed to the Denver/Boulder one) note that you can invite these guys to come to a meeting in your town as well...

Denver/Boulder Colorado User Group (DBUG)

Amazon FPS link

bfelix — Mon, 05 May 2008 16:16:38 +0000

Just curious if anyone has integrated Drupal with Amazon's FPS. We are evaluating Drupal from a number of perspectives but we are looking at some microtransaction (<$10) business model which makes Amazon FPS appealing. Seems like there are links to Authorize.net but not yet to FPS?

Thanks in advance,
Brad

Amazon Web Services (S3, EC2)

Resources for Drupal with Amazon Web Services

fintang — Mon, 28 Apr 2008 07:46:20 +0000

Thought it would be a good idea to have a resources wiki page where we could all share stuff relating to Drupal on Amazon

Modules

Amazon EC2 Console - http://drupal.org/project/ec2
AWS Base API http://drupal.org/project/aws
AWS EC2 API http://drupal.org/node/214944
AWS XML API http://drupal.org/node/219893

AMI's

http://www.io1.biz/blogs/drupalxdebugvimkcachegrind-ec2-ami (selectable version of drupal + preconfigure xdebug/vim and output for kcachegrind/wincachegrind)
http://www.io1.biz/blogs/drupal-amazon-ec2-developers-testing-ami (Ability to select the version of drupal at install)
http://bals.be/public-amazon-ec2-ami-with-drupal-5-1 (Drupal 5.1)

Articles

http://tag1consulting.com/Drupal_and_Amazon_EC2_Quick_Start
http://www.johnandcailin.com/blog/john/lamp-amazon-ec2-shaping-nicely
http://www.johnandcailin.com/blog/john/lamp-performance-elastic-compute-...
http://willy.boerland.com/myblog/automatic_loadbalancing_on_amazon_ec2
http://buytaert.net/drupal-in-the-cloud
http://groups.drupal.org/node/9167
http://aws.typepad.com/aws/2008/04/block-to-the-fu.html

Handy Docs

http://docs.amazonwebservices.com/AWSEC2/2008-02-01/GettingStartedGuide/
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=134...

Amazon Web Services (S3, EC2)

Amazon releases elastic IPs - so you can resolve your domain name to your AWS instance

Amazon — Sun, 27 Apr 2008 22:18:01 +0000

In order to host most Drupal sites you've needed a reliable IP which you can have a DNS mapping for. This is now available with AWS.

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=134...

What other barriers need to be resolved for AWS to become the Drupal platform of choice?

Amazon Web Services (S3, EC2)

SysAdmin Amazon EC2/S3 & Drupal setup | SM

fudge — Tue, 25 Mar 2008 22:53:21 +0000

Hello,

Require a EC2/S3 expert to tweak and configure a drupal site running on AWS that we notice not performing as it should.
We are preparing for launching a project in the next week and we need to make sure all is configured correctly.
We have provisioned the Centos+Virtualmim AMI and installed drupal afterwards.

Appreciate your assistance.

Regards
George
chrisg.eu {at} gmailcom

Amazon Web Services (S3, EC2)

FOSDEM 2008 presentation: Combining AWS with Drupal

toemaz@drupal.org — Mon, 25 Feb 2008 14:05:28 +0000

If you are interested in a case on Drupal + AWS, you may want to watch my presentation captured at FOSDEM 2008. You can the slideshow as well. I used the Wikifonia project as the case study.

Want to start developing? Check out the aws module.

Amazon Web Services (S3, EC2)

SQS Changes its API and pricing

dragonwize — Wed, 06 Feb 2008 13:55:25 +0000

Amazon has just announced a new API and much lower pricing structure for their SQS service. Check out the migration guide.

Amazon Web Services (S3, EC2)

Amazon Web Services API Module

dragonwize — Sun, 06 Jan 2008 21:14:21 +0000

I have just uploaded the beginning of my Amazon Web Services API module.

This module is for other module developers that want to consume Amazon's web services. It abstracts the services to functions allowing developers to easily use them without dealing with the lower level details of creating, sending, and receiving the requests. Currently, only REST style requests are supported but SOAP is planned for a later release.

Currently I have the main api module and about 80% of the S3 module written with others following fairly quickly. The modules are easy to use by just calling the functions you need. There has been little testing so far but I have made every attempt to make it PHP4 & PHP5 compatible (for now). The modules also still allow a great deal of flexibility by trying to ensure that all features of the services are still available to use.

The module requires the REST Client module. I did not use drupal_http_request() because it fails in dealing with large files and allowing the flexibility needed for REST requests. This is a new module I have written also but have purposed it for module developers to use with other REST services.

As these are API modules, I would love to hear from and collaborate with other developers. Contact me via this post, the module issue queue, or IRC.

Amazon Web Services (S3, EC2)

ec2 image for Drupal

greggles — Sun, 06 Jan 2008 13:07:32 +0000

I keep seen these images for Django or MovableType or whatever - does anyone see value in an AMI for Drupal? I'm not sure if it would be valuable enough or not. I just setup a site based on the Gutsy Public Base Image and on Victor's guide to taking a base Ubuntu server to Drupal host. It wasn't impossible, but it was harder than I'd like someone using ec2 to try out a system to have to endure.

But there's the real question for me - is it all that common for someone to use an ec2 image to try out something like Drupal? Do you think those public images from other web-app projects really help anyone?

And, final question - if you're interested in seeing this would you be interested in helping to build/maintain an image? Does anyone already have such an image they'd be willing to make public?

Amazon Web Services (S3, EC2)

S3 integration module

budda@drupal.org — Mon, 27 Aug 2007 13:29:44 +0000

I was expecting a Drupal module for S3 to already be in existence by now, as there isn't I have a proposal for a module I'm considering developing. Any feedback or suggestions on the implementation/features are greatly welcome.

This is not intended to be a full FileAPI-like module, but more a helper for general node related usage.

Support for node attachments (Core upload.module), CCK filefield uploads etc that are attached to a node.
Upon publishing of a node, any associated files are streamed over to Amazon S3 bucket(s).
Via hook_nodeapi() new attributes are added to nodes that provide a fully qualified link to the mirrored file on S3
Option to delete the local version of the file from the server once its mirrored to S3.

Example theme functions to be used in template.php to display links to the S3 versions of files.

Amazon Web Services (S3, EC2)

Drupal developer(s) and administrator(s) | Company Confidential

jaysmall — Wed, 06 Jun 2007 13:37:07 +0000

Apologies for the post across too many groups. Now corrected; won't happen again.

I am posting on behalf of a client with an existing Drupal implementation in a start-up Web business. This client has an imminent need for the following, most likely on a contract basis:

-- One or more developers with skills implementing Drupal; using modules including CCK, Flash Video, Views, PayPal and/or e-commerce and Imagecache; customizing and enhancing themes; and developing custom modules or other custom functionality. Developers should be well versed in LAMP managed-hosting environments, especially development in PHP and MySQL, and should be prepared to build user interfaces to precise design specifications using valid XHTML and CSS. The client will place a premium on developers who also understand video encoding, transcoding and optimization; edge caching; and content delivery networks.

-- One or more project managers to supervise the work of the developer(s) specified above. PMs should be highly tolerant of the ebb and flow of start-up development, frequently changing business requirements and communications responsibilities sometimes even in off hours. PMs should also be well versed in the Drupal platform and modules as described above, and LAMP-based development and systems administration, including dealing with support systems and administrators at remote, managed hosting environments.

Both roles require excellent English-language and general communications skills, high availability often on short notice, patience in working with nondevelopers to explain capabilities and limitations of technology, and tolerance of the unpredictable nature of start-up business.

Candidates for either type of role will be required to sign confidentiality agreements to get details of the business entity. Compensation will vary by types of services provided, and equity in the new business is an option. The client would also entertain the option of working with an agency that can provide resources to fill both roles.

Please provide links to Drupal-based projects you have developed, and the specs of those projects. Reply to me directly (jay[dot]small[at]smallinitiatives[dot]com) with questions or expressions of interest. I will be screening candidates on behalf of the client. Thanks in advance for your interest!

Video

SoC 2007 proposal - Drupal Automated Staging Kit

Allister Beharry — Fri, 23 Mar 2007 17:33:28 +0000

Hello all,
Sorry for cross-posting; but I think people working on different aspects of Drupal may have interest in this idea.

I've submitted a SoC 2007 Drupal proposal on automating the whole process of creating a complete Drupal site with a LAMP stack in a self-contained virtual machine image. A formatted PDF version is here:
http://www.abeharry.info/SoC2007_DrupalAST_FullProposal.pdf
Here is the abstract:

The Drupal automated staging toolkit is a proposed set of code libraries, file schemas and parsers, and code generators, for
automatically creating a Drupal site with specific module code versions, sample users and data, and a specific LAMP stack
configuration for hosting the Drupal site. The toolkit also has the ability to stage this generated site on an existing physical server
location, and also as a self-contained virtual machine consisting of a minimal Linux environment, required LAMP software, and the Drupal
site.

The automated staging toolkit is designed to be part of an automated unit- and regression testing environment, by providing testers with a
simple, fast way to automatically generate a complete Drupal site running specific code versions, and using specific LAMP server
configurations. It is also intended for use as part of a performance and scalability testing environment by providing the ability to
rapidly build and then benchmark the effects of different application, web and database server configurations on Drupal site performance and
scalability.

This toolkit will be used in the following way:
1. The tester creates or reuses an XML(or other structured) file using
a schema describing the Drupal site code-tree, including modules
installed/enabled/disabled, and the versions of each module to be
used.
2. The tester creates or reuses an XML file using a schema describing
the LAMP stack web server, PHP/application server, and
database server configuration; e.g Apache vs. Lighttpd, mod_php
vs.FastCGI, choice of op-code cache, MySQL vs. PostgreSQL, and so on.
3. The tester creates or reuses an XML file using a schema describing
the sample users and content data the site will contain.
4. The parsers take each file and generate scripts in a lightweight
language (Python or Ruby or PHP-CLI.) These scripts, when executed,
use functions in the code libraries to download Drupal modules,
generate database scripts and datasets, and write server configuration
files.
5. Given a physical server target location, the Drupal modules,
database scripts and server configuration files are deployed to the
designated server location, to produce a new, ready-to-test Drupal
site.

Time and resource permitting, a builder in the toolkit will also use the generated Drupal site and servers' configuration as input to build
a self-contained virtual machine image in Xen, VMWare, or potentially the Amazon EC AMI format. This virtual image can also be be used in
testing environments, including advanced performance testing scenarios such as evaluating clustering, distributed database topologies, and
alternative storage and computing models like Amazon S3 and Elastic Cloud. The ability to rapidly generate self-contained Drupal virtual machine images will also be extremely valuable to Drupal consultants and solution providers for marketing, prototyping and demonstrating
Drupal solutions to potential clients, and large organizations and ASPs like CivicSpace looking to take advantage of the massive benefits
of virtualization technology from VMware, Xensource, Amazon, and the like.

Allister.

SoC 2007

soc project

keizo@drupal.org — Thu, 08 Mar 2007 22:28:21 +0000

Has anyone made progress with a general integration with Amazon S3? I am interested in a summer of code project integrating S3 to serve static content and file uploads. Any thoughts?

Amazon Web Services (S3, EC2)

Duju is a Drupal sandbox machine

matt@antinomia — Thu, 01 Mar 2007 07:11:15 +0000

If you're interested in generating Drupal sandboxes at the push of a button, check out the Dojo Duju at [EDIT] ~~http://www.its-coming.com/~~ ~~http://duju.hopto.org/~~ (now offline).

What's a Duju? It's a Drupal sandbox that comes preconfigured with several popular modules and was developed with an eye towards being able to collaborate on concepts and demos of Drupal modules in a public venue. This approach uses Drupal's multi-site feature to run all Dujus from the same codebase and creating a way for Duju users to access their specific Duju directory but not those of other Dujus.

There are several security holes in the method. If you can be of assistance in fixing them, that would be awesome!

Please note, the web site [EDIT] ~~www.its-coming.com~~ duju.hopto.org itself is no way intended to be a permanent enterprise. It is merely demonstrating a proof of concept which needs to be refined. Also, consider it volitile and capable of disappearing at any minute (i.e. don't put anything of value on there).

Drupal Dojo

streaming speed from S3

Bèrto ëd Sèra@drupal.org — Tue, 30 Jan 2007 21:58:39 +0000

For my information, can one have a decent streaming speed from S3 servers? Our material is mostly going to be audio and video. Thanks

Amazon Web Services (S3, EC2)

Some more thoughts on Amazon S3 and EC2...

matt@antinomia — Sun, 14 Jan 2007 01:56:54 +0000

Greetings! I don't intend for this to be an advertisement for Amazon, so if folks want to discuss other similar services that's totally cool, and we might even change the name of the group. But, I've been spending (too many) hours researching Amazon Web Services recently and thought some people might be interested to know what it's all about and how they might be able to use this stuff...

Boris made a post last April that captured some interest with regard to using Amazon Simple Storage Service (S3) with Drupal. Since then, Amazon has released the Elastic Computing Cloud (EC2), which allows one to deploy virtual servers (equivalent to "1.7Ghz x86 processor, 1.75GB of RAM, 160GB of local disk, and 250Mb/s of network bandwidth") on the fly in seconds. The services work together, and there is no fee for data transmission between S3 and EC2. Pricing is reasonable for both services, billed at a metered rate determined by your use of resources.

So, what does this mean? Well, a lot of things... I've thought of a few ways you could use this. I'm sure you can think of more, and that's what this group is all about.

A few references if anybody is interested: First, dopry's fileapi.module which will theoretically support using S3 as a file mount point (i.e., use S3 instead of the /files directory). Second, a patch I made to the backup.module which allows you to back up your Drupal site regularly, automatically, and effortlessly. This has been invaluable to our company in making sure all of our clients sites (on various servers and hosts) are regularly backed up. It has some PEAR dependencies (HTTP_Request, HMAC_Crypt) that will likely keep it from going mainsteam, but it's interesting nonetheless.

I've also been talking with arthurf about his media_mover.module, which will basically allow YouTube functionality within Drupal (upload videos, conversion to flash video, storage on an offline [S3] filesystem). The one major hangup (it seems) is figuring out how to distribute ffmpeg in a way that's easily accessible to most people. I know little about creating distributable binaries (I freak out when I hear the word 'dependency'). If it's possible, that's awesome. If not, we could use Amazon's EC2 to create a disk image with ffmpeg compiled (and maybe Drupal installed and configured) which would basically allow anybody to create their own YouTube clone for $0.10/hr...

Anyhow, enough rambling... Hope some people are interested in this. :)

Cheers,
Matt

Amazon Web Services (S3, EC2)

Automatic site backups with Amazon S3

matt@antinomia — Sun, 17 Dec 2006 05:34:07 +0000

Hello, I've recently submitted this patch for the backup module which allows automated regular backups of a Drupal installation to Amazon S3. It currently uses Geoffrey Gaudreault's Amazon S3 PHP class, which requires the Crypt_HMAC and (hacked) HTTP_Request PEAR modules, as well as forcing the site developer to alter some variables within the code of the class. I'm hoping the File API will offer a much easier way to integrate the backup.module with Amazon S3.

How much work has been done on the File API with S3 so far? Does it require Crypt_HMAC and hacked HTTP_Request as do other PHP implementations of S3?

Amazon Web Services (S3, EC2)