Posted by tugis on March 7, 2012 at 3:02pm
Lets say for example, if you have a group (Organic Groups) and, for a certain action, you want to send an email to 200 users. If Drupal handles email serving (trough php mail or smtp or phpmailer), this will painfully slow down page execution.
What I would like to know is how you handle a situation like this. I heard about "Mail queue" module, where emails are putted in a queue and then sent from cron, but I don't like this solution. I would like to send emails in real time and take that responsibility out of Drupal. Is there any way you can pass email sending to an external service?
Comments
In this case the expensive
In this case the expensive part is not the sending of emails, but rather determining who gets what, and customizing each email with the appropriate content, customized for each recipient. The actual sending of the email is not that expensive. Mail queue is your best bet. If you don't like the idea of running all your cron hooks once a minute, you can instead use something like
http://drupal.org/project/ultimate_cron
http://drupal.org/project/elysia_cron
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Mail queue doesn't play well
Mail queue doesn't play well with SMTP and Mimemail module and using Mail Queue, you still have server load to process the queued emails on Cron. I'm talking about a "Social Networking" like site, where you can possibly have hundreds or thousand of emails per day.
I have seen for example, that even sending just one email when an user posts something in another user's page (posting through Ajax and appending status to page stream), the result retrieval becomes really slower (from 500ms without sending email to 2/3 seconds sending one email).
What I'm trying to find is a way to pass the email handling process to another machine (through an external service possibly), and take that responsibility out of Drupal.
HTTP Parallel Request Library
http://drupal.org/project/smtpua & http://drupal.org/project/httprl might get you there
Not sure of anything out of
Not sure of anything out of the box, but it's really pretty easy to do custom using libraries like phpmailer.
Drupal: Notifications are put in a special queue (database, memcache, really about anything!)
Mail Server: PHP script that gets triggered by cron every minute. Checks the queue and if there's anything in it, it sends it out.
I set something like this up for a client a couple of years ago that sends out over 50,000 emails/day. It took me about an hour to get all the code going and was very light weight. I think when it was all said and done it was less than 100 lines.
The other benefit is that it also scales rather easily if you get into really massive amounts of emails.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
Queue
The reason that request time jumps up so high is likely just that the mail send is synchronous, not because it's creating significant load on any the resources involved.
Mail Queue should solve your problem, as users don't care how many seconds Cron takes.
If your server really does get overloaded formatting and shipping mail messages, you don't really have a choice in the matter except to expand. Formatting the SMTP message is not significantly more or less complicated than popping it on a queue or calling an external service, so if you're hitting resource limits there, you probably always will be.
Hook_cron is pretty evil for
Hook_cron is pretty evil for processing large volumes of data, especially as potentially any module you install or write can invoke at any time. If you want to avoid use of hook_cron for processing your queue, and want to have to possibility of getting more real time than every minute, it might be worth considering drush daemon (http://drupal.org/project/drushd).
The way I'd see it working is:
Use notifications api to define templates etc for your email
When you reach the point in your app where you send the email, add the email send request to the notifications queue.
Write a drush command which processes your queue
Implement drush daemon which monitors the queue at a configurable interval (1second+) and if it finds any items on it, sends them
You could go to one further level of abstraction and bypass notifications altogether, your user request that triggers the email send simply adds to a queue (use the queue api) containing the data of the user, the email template type and custom data (ie from a form) and then use drupal_mail to send the mail directly from the daemon monitored queue.
The benefit is that whilst volume are low to average your queue will process in as near to real time as makes no odds, but if you get a spike in load you're just adding to your queue and not impacting user experience as the mail server begins to creak.
Thank you all for the tips
I have implemented the solution proposed by Mikey, with some minor tweaks. That was at I was looking for: a non-blocking method to send emails in real time without affecting page loading time. The httprl project is a great idea!
Drush daemon also sounds interesting. I will also take a look at it.
You need a real queue
Try ZeroMQ. This is used for stuff like trading systems. AFAIK is the most simple and performant queuing system there is. You'll have to code it yourself. I would create a custom
drushscript that boostraps to the lowest possible level and does the talking with ZeroMQ.Just an Idea.
Spawning a child process to run asynchrously to send the email
A year ago I noticed a site using trigger to send emails for a page viewing, resulted in the page display slowing down from less than 1 second, to more than 5 seconds. Sendmail was running on the same system. It has to do DNS queries, several of them, slowing down the sending of queued email. Turning off the email action restored the page display speed.
There are two possible cures.
First, the action of sending the email should be done after the page is served out.
Second, the action could spawn a new process, an independent child, which is responsible for sending the email, and this second process is delayed, not Drupal. Spawning is slow on some systems.
Other aids would be sending the email without having the mail server do DNS lookup to validate the sending email address, and other fine tunings.
Cron was mentioned, and while it could run every minute, having it run a script, that sleeps for 5 seconds, checks for mail to send, avoids the minute delay. If the script is still running, then cron does not restart the script. As the script ends, it spawns a copy of it self. So, the script is always running.
What's the proper solution? The first one. Why? Mail is already a daemon process, and loading the machine down with children or scripts, is not optimal.
Why is the first solution not possible? Does Drupal have a hook to run after the tag 'end html' is sent? Or does Drupal exit at that time? Sounds like a change to core to me. A worthy one. But of low volume. Unless this gets a lot of talk on this thread.
My two cents.
Peter
LA's Open Source User Group Advocate - Volunteer at DrupalCamp LA and SCALE
QueueAPI
I use Queue API backed by beanstalkd for queue operations. It has worked well for me to date, and the bonus is you can host the queue on a different machine from your web/db/app servers if needed. I then 'sit' on the queue with drush waiting for jobs and process them out.
My primary use of this is to generate imagecache images in a distributed manner (see imageinfo_cache sandbox here), but it could work for mail just as easily. It's almost the same approach perusio suggested above, just using Beanstalkd as the queue.
https://www.mailjet.com/ simp
https://www.mailjet.com/
simple solution with a free contingent of 6000 emails a month.
Notifications tools ?
Haven't tested them, at it is unmaintained at the moment, but perhaps http://drupal.org/project/notifications_tools might help (seems to use drush)
Amazon Simple Notification Service
This might work for you.
http://aws.amazon.com/sns/
Amazon Simple Notification Service