High performance mail sending out of Drupal site

I've had a brief look at MailGun while I was messing around in the RackSpace cloud. Looks pretty neat but I don't really know how to use it.

Another important consideration is making sure the mail "appears legit" so as not to get caught in spam filters etc.

This is pretty much the next item on my todo list so very interested to see what others have to say! :)

BC

Very simple to use, and uses dedicated mail servers, http://drupal.org/project/mandrill , can override all drupal mail via http://drupal.org/project/mailsystem

Much better than using your server to email and being blacklisted.

Out of interest, do you have anything set up to handle bounce messages properly, i.e. if an email server returns that either it thinks your message is spam, or that the recipient address doesn't exist, to have delivery properly handled? Without a system in place to handle these, and a bazillion other issues, these you are looking to have your server(s) blacklisted very quickly. As others have said, it is recommended today to use a proper email delivery service for bulk emails because they'll take care of the large number of problems that arise, and we're speaking from experience here, not just repeating something from Information Week.

There are 86,000+ seconds in a day, and most emails take 2-3 seconds to send. Thus, only 43,000 emails can be sent from a mail server per day. Or less. Unless, you scale the hardware, or optimize. Drupal will not do what you are asking.

What you want is a mailing list server, and Drupal sends one email to the list server, which then sends your 10,000+ emails a day.

Drupal modules are definitely not the way to go... unless you need each email to be personalized. Then, scaling becomes a problem.

Hint: Use a module that spawns a child process, and avoids the "wait" of the email process returning with a code.

I've read the whole discussion above and I agree with others that sending email directly from Drupal page calls isn't sustainable.

To make this work you will need a custom module that leverages Drupal's Queue API. Each time you need to send an email add the required data to the queue. This data is likely to include, the name of the email template, the user object and if it relates to an item the item object. Have cron running every minute processing X items in the queue each run, so then you can send X messages per minute. You should consider running this on a separate node so it doesn't impact on web users during time of heavy load.

You will want to be running postfix (or another good SMTPd) on the same box that you're using for running cron. This will accept and queue the messages for delivery. You will need to ensure that you have SPF and DKIM setup properly along daily checking that your server or the network block hasn't been blacklisted.

Now for the silly cloud nonsense that you don't want to hear about - feel free stick your fingers in your ears and shout "LA LA LA! I'M NOT LISTENING! LA LA LA!", because others might find this information useful.

I would use a real Queuing system, such as RabbitMQ or depending on where the site is hosted AmazonSQS or Rackspace Cloud Queues. Like the scenario above you dump the data into the queue on an as needs basis. On the other end of the queue have a standalone application that is designed purely for preparing emails, which probably means using Python or Ruby. The email prep app which would run on a separate VM/box would then send the email out via a third party SMTP service such as Mailgun or Mandrill. These services stake their business on doing the best they can to get your messages delivered. This decoupled approach helps you distribute load and if you throw some HA services into the architecture should minimise downtime.

You can now take your fingers out of your ears.

Drupal is good for many things, but is not good for all things. Pay as you go cloud systems get costly due to software overhead.

I'm wondering what specific parts of drupal your application requires and how much of it you can shut down or bypass.

I'm not versed on the cron processing overhead, but when you start looking at throughput, ie pages per second, you have to consider how drupal operate and what you need it to do. The bootstrap operation loads each enabled module and all the system variables before considering your request. That can encompass accessing hundreds of files and a whole bunch of database calls. Opcache and other memory cache's store the files resident in memory, but still, each file is checked for being changed. In D7 that can range from less than 100 files to up around 400 depending on the number of modules. I've started to look at D8. D8's oop organization results in an even greater number of files, albeit small files. Thus, the more modules you enable the slower it gets.

Once your request is bootstrapped, the actual processing begins. If you are creating a node, countless hooks are called to see if any module needs to modify it before saving. Fortunately, the caching system saves each node as a render array to enable quick fetching as a serialized blob. The block system has to process and assemble the blocks, the menu system churns to load menu items and check translations of menu items. And the theme system has to spit it all out.

On quite modest hardware, a stripped down D7 can process pages in about 300mSec. On my D8 standard install I see 1-2 Sec with opcache (>5Sec without).

So it seems that if you really need a system that cranks, you have to look at the 'cost' of the software overhead in the application. Your email processor must be a lean mean fighting machine. I would suggest taking whichever opensource mail server you like, and modify it to pull data from the drupal database.

I'm currently looking for a good project, so let me know if you'd like assistance.

Have you considered that there are very valid reasons why people bring up these services? It's not about simplicity or just "offloading the work". Setting up a server and a script (even Drupal) to send mass mails is actually rather simple. We all know that. You know who else knows that? Administrators of email servers. Others have already mentioned you have to deal with things like bounces, your IP being blacklisted (and I will tell you first hand, you can get a server and an IP, which hasn't been used in a couple of years, and there's a really good chance you'll have to fight with an email service to get that IP removed from the blacklist. AOL is a biggie here and a PITA to deal with).

What I haven't seen mentioned and something that is a huge problem as your number of sends increase, and why these cloud services are so popular is IP throttling. Most major email services only allow X number of emails from a single IP in a given time frame, including Gmail and Hotmail. That's one of their main ways to fight spam. These cloud services get around that by having multiple IP addresses. They also have worked with these providers to have their limits increased, something they won't do for just anyone and a reason why these service providers also have strict guidelines on what can be sent and how to deal with it.

Having said that, if you still want to go your own, then one of the best methods I have seen and done is to use a queuing service that Drupal can write the email to and then a separate script actually sends out the email. The system I did for a client last year was rather simple. Emails that needed sent were composed in Drupal. Drupal then sent that to a collection in MongoDB. I then had a simple NodeJS script running that monitored that MongoDB collection using tailable cursors (Mongo's version of PubSub), so that as soon as something got added, Node would send it.

Why this route? Simple - reliability. Mongo is persistent, so if the server crashed for whatever reason, things were still there. You could do it with MySQL though. Check a table for emails that aren't sent, send them, then check again. If nothing is there, then wait a few seconds and check again. This client already had Mongo, so I decided to go with that. As far as Node, it's very easy to thread using child process. So I can have up to X processes sending out emails at once. Each process only takes about 8mb of memory and only the master needs the Mongo connection. Sounds complicated, but I was able to achieve all this in only 2 days of coding.

Another reason - simple math. You can run Drupal's cron every minute. Drupal creates a lock so it only runs once job at a time. There are 1,440 minutes in a day. You say up to 200,000 emails per day, so let's take that since that is going to be your highend goal and we don't want an "oh crap, it broke" scenario. 200,000/1440 = 138 emails per cron job. Even if an email takes 1/2 a second to send, you are still out of time, and we aren't even considering any other Drupal tasks that will run on that cron. Meet your newly created bottleneck.

So the best route is to keep this part out of Drupal. You could try PHPMailer, which I have seen handle up to 100,000 emails per day. Of course that came with a pretty significant increase in server load to the point that it was scrapped (you don't want your emailing to start impacting the performance of your site). Even PHPList, tells you that the limit on a single server is about 3,000 emails per hour.

Finally, and possibly the most important issue. You said your one a big VPS and will go dedicated if you need to. Something about all eggs in one basket comes to mind here. Basically we've already determined, through some simple math, that a queuing service will be needed. There's another beauty of that. You don't need to have everything running on one server.

If it were me doing this, I would keep Drupal on that VPS and use smaller VPS' or clouds to handle the email part. Not only are you greatly reducing the impact sending all these emails will have on your site, but you're also creating something that is highly scalable. That's the beauty of what I did in Node. Instead of forking child processes, it can easily send out a batch of emails to a different server. And since everything is rather lightweight, those extra VPS' can be smaller. Your email list grows. Simple, increase the resources on that VPS or add another.

Oh and one other thing. Remember that big issue of sending mass emails from a single IP I mentioned at the beginning? Well you just got around that too.

(Quick note: You could do what I did in Node in PHP. It was quicker and easier for me to write it in Node though, mainly because there were npm packages that helped build this quickly, especially forking and being able to send messages to child processes through different messaging services.)

Of all the Internet Protocols (RFCs), email is the oldest, and most used, and has more features, than any other. More RFCs, more changes over the decades.

Trying to learn email methods on a Drupal Groups discussion ... it's not going to be fast. Learning email methods is never fun. It's hard, hard work.

The reason there are many custom solutions posted here is due to this complexity. No one solution fits all problems. With an expounding of your application, you have gotten closer. Can any list member now give you the fastest way to implement your requirements? Likely not, due to it's complexity.

A large degree of scaling experience is required. One dedicated server will not do all your services, except at the beginning, when volume is low. Getting the scaling experience is a very hard learning curve. It takes months. Of benchmarking various solutions, and configuration tweaks of those solutions.

There are not hard and fast answers for scaling custom email solutions with any CMS, or any other application. You will want to find a forum about email scaling. Hint: It's not here in GDO.

Expectations your initial production release will work, for several months, until it maxes the CPU or data bus of your host computer. At that time you will go into rush mode, trying to find a solution. You will first get a faster Host, more CPUs and faster bus. Just to get some breathing room for finding a long term solution. All the time wishing that email scaling was not as difficult as you learned in this discussion.

The CPUs going through 'context' changes is what slows down a dedicated host running too many daemons. So, your next solution will be to put the mail server on a second host connected via a private network connection. This scaling direction will continue, until you have several Drupal hosts and several email hosts. Scaling will then be achieved by throwing hardware at it, which minimizes the use of your time.

Eventually, you will want to replace all that with highly optimized computer OSes for each service, in order to reduce the number of racks, to just one, and save a lot of money.

That's the history I have seen over the last 20 years implemented by team after team. Good luck.

Well, this group may be the answer how to connect Drupal to mailing solution. Yes, you are right, I think term 'email' came even before 'internet' and has its own history, but would like to hear about things which are pretty narrow in my (our) case.
The task is dead simple - cheap, quick email sending out of website. Rack for sending email... It would be a huge success for the project, and then probably dedicated Certified Email Professional could be hired, but until then only earthy problems.

I'd highly recommend Amazon SES - I used it with EE to send upwards of 7,000 e-mails a day to a very highly-spam-filtered e-mail service (Mailmarshal and Spamassassin, iirc).

http://aws.amazon.com/ses/

As @tripper54 suggested, for this, you typically don't want to have Drupal do the sending, but use dedicated mailing software (if you have to use SES) or a campaign emailing service like mailchimp.

Assuming you want to do it on your own, since you are deploying on AWS, you may want to consider writing this as a Drush plugin (hence a CLI app, not web), and use the AWS SQS Queue API provider for D7, available in https://www.drupal.org/project/aws (direct SQS integration) and https://www.drupal.org/project/awssdk (lower level, but recommended).