Native Drupal Mailing List module

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
kyle_mathews's picture

Hey folks,
I've been busy off and on the past month or so writing a native mailing list for Drupal. It integrates natively with Organic Groups (i.e. each group becomes a list w/ its own email address) and behaves exactly like popular open source mailing lists like Mailman/Minimilist (it should as I copied their algorithms). I'll be releasing it on d.o. soonish but I had a few questions I wanted to ask first.

First -- what should I name it? Mailinglist is the obvious name and is open in the Drupal module namespace. Any objections or better ideas?

Second -- I'd love to discuss possible architectures with some interested developers. Right now the code is... a mess, to be charitable -- mostly due to the many dead-ends I hit before getting things working. I need to do some serious refactoring before doing the initial release and wanted some feedback / ideas on different ways to architect the code.

Comments

Ooooh

Michelle's picture

I love this idea! I can already think of 3 sites I want to use it on. :) "mailinglist" sounds fine as a name.

Michelle

Drupal has the SimpleNews

Garrett Albright's picture

Drupal has the SimpleNews project. It can't automatically create mailing lists based on OG subscriptions, but it's a fairly capable mailer. Did you consider basing your work on that project?

This isn't a mailinglist

kyle_mathews's picture

This isn't a mailinglist manager in the sense of Simplenews where emails go only in one direction -- out -- it's a mailing list along the lines of the Drupal mailing lists -- http://drupal.org/mailing-lists -- which provide a simple way for group email conversations to happen.

WIth this module + OG, you have a Google Groups clone. Where you join a group online. You can search/browse past discussions online. And you can start/contribute to discussions either through the web interface or through the group's email address (it's based on spaces/purl so it uses the purl prefix for the group's email address).

Kyle Mathews

Ah, okay. I get it. Sorry, I

Garrett Albright's picture

Ah, okay. I get it. Sorry, I guess "mailing list" is kind of ambiguous in that regard.

Yeah -- it definitely is. Not

kyle_mathews's picture

Yeah -- it definitely is. Not much we can about that though at this point.

Kyle Mathews

Maybe...

Michelle's picture

Mail Group?

Michelle

current alternatives

greggles's picture

I think there is a current organic group based solution:

Most of this combination is already running here on groups.drupal.org. I'd like to deploy the last pieces of this (mailcomment + mailhandler) in the next few months but there are a few features necessary first. On a similar note, we (GVS) are using this as the basis to rebuild the IxDA.org site. So, I hope you will consider that combination first. It would be great to have more people working on this chain of modules and I'm happy to chat about it to help get you up and running.

Currently using this stack

bonobo's picture

Along with comment upload, to allow attachments to be handled on comments via email --

Setting this up is pretty well beyond the reach of even moderately technical users.

I'm curious to see how Kyle's code/solution compares. Over time, I see the potential for these pieces to morph together, but given that Kyle has something that appears to be working, it makes sense to look at both options.

My .02 --

Cheers,

Bill

I've been using this chain

kyle_mathews's picture

I've been using this chain (and variations) for 1.5 years now so I'm very familiar with those modules -- and their limitations. The beating I took against their limitations over the past year or so are what motivated me to write my own setup. I'm definitely not in favor of module duplication but the amount of time I spent writing code to work around problems (never completely successfully) I found w/ notifications/messaging far exceeded the couple of weeks I spent rewriting them to meet my specific needs.

For the past few years I've been building a g.d.o. like site (https://island.byu.edu) which is used by upwards of 500 student/faculty/administrators at BYU. The basic idea is to create a platform for classrooms/majors/clubs/interest groups to create online spaces to easily share communicate with each other. It's predecessor was a simple mailing list run by mailman. The list did (and still does in its new drupal clothing) get a significant bit of traffic -- ~5 new posts a day or so.

We moved the list to the island originally for two reasons. First, to provide the platform to easily allow anyone to create their own mailinglist/group, and second, to address complains about "off-topic" emails on the list -- mostly people posting jobs. So the list got moved to https://island.byu.edu/isys and a jobs group was created at https://island.byu.edu/jobs.

And everyone was happy... sort of.

The first major complaint from the students was the emails were ugly -- which they were. So I dove into the notification templates and after a lot of tweaking got emails mostly working (though they never were very happy with the dividing line stuff).

The second big complaint was the loss of email history. In a list, you reply to an email and the other list members get your reply /and/ what you're replying to. Which is really useful, and like a lot of things, not noticed until it's gone. Mailcomment has the divider line thingy at the top that you're supposed to reply above. It uses that line to cut out the reply from the rest of the email so it can make a nice clean new comment. The problem of course that it cuts off what you were replying to. This doesn't matter some of the time but occasionally people on lists will reply to an early email and say something like "haha, so funny, totally agree" but since what they were replying to was stripped away and not sent out -- others would reply back confused asking what they were agreeing with.

I sort of fixed that by manually inserting into emails the last 5 comments made on a post which sort of worked but not entirely as often times what someone was replying to wasn't the last comment made. That fix was really painful and quite hackish.

Another problem with that stack is it's quite slow -- meaning the turn-around time from when someone sends an email to when everyone else gets the email can be quite long. Depending on how often you run cron it could be 20-30 minutes before everyone has gotten the email. Which is completely unacceptable for a mailinglist. What we did to fix that was first we did an integration with Exim4, our MTA, so that when new emails were received, exim would call a php script which would figure out what group to put the post/comment in and do it immediately. Then to speed up pushing out emails again, we wrote a PHP daemon to hit messaging and notification's cron functions every 30 seconds. Together those two dropped the loop time down to 3-5 minutes -- which is still 2-3 minutes slower than a real mailing list.

So the above problems we mostly fixed but the below we couldn't which proved the proverbial straws which broke the camel's back.

The email formatting was consistently fouled up. With notifications/messaging, emails sent to a group are first saved to the site and only then sent out. That could be fine except the messaging module insists on running all text it sends out through a Drupal input filter which in subtle and not so subtle ways would often change people's formatting. Particularly uncute was Drupal's insistence on removing any html added to emails -- not very helpful when someone is asking for help on html.

An associated problem -- right now it's pretty much impossible to run the above stack and retain the html from emails. Plain-text emails are just uglier plus people were annoyed that formatting they put in their emails would get stripped.

The last big problem is when people send emails to one of the group lists, they get an email back. There was no way of stopping that (out of the box anyways).

And there's probably a half-dozen other minor annoyances with the old setup I'm not remembering.

In short, the stack you listed above is inadequate for anyone who's ever used a real mailing list. It's somewhat like the uncanny valley concept. As long as my website didn't really act like a mailing list -- no one complained too much. They treated it like a forum with email notification. But the closer I made it to a mailing list, the more they noticed it's deficiencies vs. a real mailing list. So I stopped trying to pretend messaging/notifications was a real mailing list and just wrote one.

Kyle Mathews

Interesting. Thanks for the

Garrett Albright's picture

Interesting. Thanks for the write-up.

I've previously done daemons in PHP as well (particularly Drupher), but I've just used inetd (FreeBSD) or launchd (OS X), which function as "super-daemons" capable of running simpler scripts as daemons. I'll have to keep that PEAR package in mind should I ever need to do anything more complex.

The second big complaint was

greggles's picture

The second big complaint was the loss of email history.

IMO what's really needed for that is just proper mail headers so that they thread on the client side. Then people can see it. We're also using this with Markdown which means that > turns into blockquote. It can require messing with replies a little more to get the right threading but I think people can be trained.

Then to speed up pushing out emails again, we wrote a PHP daemon to hit messaging and notification's cron functions every 30 seconds. Together those two dropped the loop time down to 3-5 minutes -- which is still 2-3 minutes slower than a real mailing list.

I guess I see this, but many lists use manual moderators and often the mail servers along the way slow things down. People should get some tea.

The email formatting was consistently fouled up. With notifications/messaging, emails sent to a group are first saved to the site and only then sent out. That could be fine except the messaging module insists on running all text it sends out through a Drupal input filter which in subtle and not so subtle ways would often change people's formatting. Particularly uncute was Drupal's insistence on removing any html added to emails -- not very helpful when someone is asking for help on html.

Can't this be solved by "escaping html" rather than stripping it? Or tweaking the input format that's used for sending mails. I do generally see what you're saying, but mail clients are notorious for their alternate ideas on proper rendering.

The last big problem is when people send emails to one of the group lists, they get an email back. There was no way of stopping that (out of the box anyways).

http://example.com/admin/messaging/notifications - "Notify poster of own posts
Notifies a node poster about their own posts. Useful principally during testing. Default is OFF."

Right?

In short, the stack you listed above is inadequate for anyone who's ever used a real mailing list. It's somewhat like the uncanny valley concept. As long as my website didn't really act like a mailing list -- no one complained too much. They treated it like a forum with email notification. But the closer I made it to a mailing list, the more they noticed it's deficiencies vs. a real mailing list. So I stopped trying to pretend messaging/notifications was a real mailing list and just wrote one.

Yeah, I get that and I appreciate your detailed list of feedback. I'm just saddened because I've seen you do good work and hope that you could put it toward these modules since I feel confident in them in general.

Just to be clear -- I have

kyle_mathews's picture

Just to be clear -- I have nothing against messaging / notifications. I think they're great modules for what they're designed for -- which unfortunately for me, as I've found out the hard way, is not running mailing lists.

They were designed, essentially, to push content out. Something happens on the site and people who care get informed about it in the way they choose. It's very flexible and solves a wide variety of problems. And they wrote into their code those assumptions. And after a year of trying to work around those assumptions, I gave up and wrote a module for my needs.

And long-term I'd really like to fold the work I'm doing back into the messaging/notifications stuff -- I'm starting my own company and have gobs to do and really would prefer to have help maintaining my code. So hopefully this module will be a temporary fork to prove out a point and then messaging/notifications can be reworked to include it.

BTW, the one big assumption messaging/notification makes that I couldn't ever work around is that all notifications must originate from within Drupal. All /real/ mailing lists (e.g. Mailman and Minimalist) work as follows. Email comes in. The subject is optionally changed (e.g. add a [list-name]), a footer is added, email address of members are added as BCs, and then it's sent out again. And that's it. It's very simple and it works perfectly. And that's what my module does as well. I don't know exactly how notifications/messaging would have to change to allow that workflow but I'd be happy to help make it happen as the flow I described is far superior than using mailcomment et. al.

Kyle Mathews

on architecture

kyle_mathews's picture

Back the architecture question. There are two patterns I could see fitting this module's needs.

First is the more traditional Drupal style of architecture. Where I create an object/array to hold data and pass it around to different functions with built-in hooks to let other modules override behavior.

Second, is create an object with methods that do must of the work internally and drupal-independent. This object would take the raw email, parse out attachments, plain-text, html-text, add footers, adjust subjects, etc. The advantage of this is it's somewhat cleaner (I think) and since it'd be Drupal independent, other coders could take the mailinglist class and create lists for other platforms.

Any thoughts on these two styles? Or is there other options I'm not thinking. I'd really like some feedback here as it'll be much harder to decommit myself once I've gone down a path.

Kyle Mathews

Quick update. All the code is

kyle_mathews's picture

Quick update. All the code is now on github. I'll be cleaning things up plus adding a few more quick features before making an initial release on d.o. Any help / feedback would be greatly appreciated.

For the module's name -- I ended up calling it og_mailinglist.

Kyle Mathews

Oh -- and the link

kyle_mathews's picture

Oh -- and the link http://bit.ly/4XbwR6

Kyle Mathews

thanks

atruser1's picture

Kyle,

This would be awesome. Most of my users prefer Google groups like functionality. However, security is a big issue for us - plus, we would like to be able to send file attachments as well. Is this something you can incorporate soon?

Thanks much,
atruser1

There's already as much

kyle_mathews's picture

There's already as much security in place as Google Groups -- i.e. it checks the originating email address to see if the address is from an active user in that group. Actually Google Groups has a lot of problems with spam -- see this post about problems the jQuery group faced: http://ejohn.org/blog/google-groups-is-dead/

Since this module is a native Drupal solution -- it'd have a leg up as you could run new posts through Mollem or something to check if they're spam.

On file attachments -- yes, that's on the pipeline. Right now File attachments are passed along in the emails sent to other people in the group -- but aren't saved as a file attachment.

Kyle Mathews

Kinda surprised to hear spam

Garrett Albright's picture

Kinda surprised to hear spam is such a problem. You'd think running incoming messages through the same system they use to filter spam for Gmail would be obvious.

From the rumblings I've heard

kyle_mathews's picture

From the rumblings I've heard -- it doesn't sound like Groups is a very high priority project. They've been criticized quite a bit for leaving their valuable archive of usenet articles virtually unusable -- http://www.wired.com/epicenter/2009/10/usenet/

Kyle Mathews

I have some other ideas for

kyle_mathews's picture

I have some other ideas for tightening up security -- but I won't have time to implement them for awhile. I'd be happy to discuss ideas though.

Kyle Mathews

Just what I needed

gmui's picture

Our church website uses Drupal and this is exactly what we've been looking for. I dont' have a lot of time messing around to get Mailman to work so an all-inclusive mailing list solution within Drupal would be great.

Will I need to upgrade to Drupal 6.x to try this out or is it 5.x compatible?

Thanks!
Gary

The module will be Drupal 6.x

kyle_mathews's picture

The module will be Drupal 6.x only. It'd be fairly trivial to backport so if I'd create a 5.x version if someone writes a patch.

BTW, as I hadn't made this clear before -- to work fully as an email list -- you'll need to have root access and some sys admin ability. No more than you'd need to install mailman or other mailinglist software but this module is definitely not shared-host capable.

Kyle Mathews

Funding if it works shared

BamaJohn's picture

We need this functionality but we are on a shared host. If it is practical to adapt it to shared hosting we are willing to help pay for that work and some development time in general.

It might be able to run on

kyle_mathews's picture

It might be able to run on shared hosts... I'm actually still somewhat of a email noob, even after all the mucking around I had to do to get this module working right.

Why it requires your own server right now is I wrote it to use an advanced feature of exim4 -- which lets you use an external script to transport new email. So for any email that comes in, Exim just hands it over to my module and says, "do what you want with that". The module then figures out if there's a group associated with the email address and if there is, it pulls out the body of the email and creates a new node. I'm not sure how to duplicate that sort of thing on a shared host.

Here's the best idea I've come across is to use something called a catch-all email address. Per the wikipedia description (http://en.wikipedia.org/wiki/Catch-all) a catch-all email address is where all emails not sent to an existing email get dumped. So og_mailinglist could just hit that email address every few minutes to see if there's new emails and then process them as normal.

Does anyone know if that would work? Mailhandler (http://drupal.org/project/mailhandler) might have some useful code/concepts here.

Kyle Mathews

Careful with catch-all

Michelle's picture

I used to have them on all my domains and found them very handy. Then I found out the hard way that spammers find them handy, too. :( I got flooded with "out of office" type messages all of a sudden and found out that spammers use domains with catch-alls as their "from" address because the catch-all ensures that they show up as legit emails.

Michelle

catchall is fine

tobias's picture

I think using catchall is fine, as long as you properly handle mail to the domain. That is, deliver postmaster@foo.com mail to yourself, list-related mail to the lists, and bounce the rest.

As long as that happens your domain won't be exploited by spammers.

Cheers,

Tobias

.

Michelle's picture

Er... If you bounce everything else, it's not a catchall, it's a bounceall. :)

The whole point of a catchall is to catch ALL mail. If you're defining specific items to let through and bouncing the rest, that's normal usage, not a catchall.

Michelle

Hi Michelle - Not sure what

tobias's picture

Hi Michelle -

Not sure what you have in mind, but I am thinking of a scenario where you have a catchall mailbox that collects all the mail, then lets your script pick up the mail and handle it. So it wouldn't be the MTA that is bouncing or delivering the mail, but your script. This has the advantage that it can be used by anybody on a shared host.

CiviCRM CiviMail does this on my server to handle reply email and it works incredibly well. It uses imap2soap.pl to do it - I just googled "CiviCRM imap2soap" and came across a bunch of stuff including this recipe that Kyle might find useful: http://back-end.org/blog/stevem/howto_civimail_return_channel_for_multis...

Cheers,

Tobias

.

Michelle's picture

Ah, I see. You're talking about adding another layer between the catchall and the mail handling. That sounds like it would work.

Michelle

CiviMail looks like it'd have

kyle_mathews's picture

CiviMail looks like it'd have some useful ideas/code. That's my favorite part about open source programming. Almost always someone's already done what you want to do -- you just have to poke around the intertubes long enough until you find it. Case in point, the core algorithm for appending footers to new emails on og_mailinglist was copied from Minimalist, a perl mailinglist manager - http://www.mml.org.ua/

Kyle Mathews

Check out fudforum too

tobias's picture

Hi Kyle- Yes, I love this aspect of open source too. :)

On that note, you may also want to look at http://fudforum.org which includes a script called maillist.php to integrate a forum with a mailing list server like mailman. I last played with it in 2006 but it worked extraordinarily well - it even provided methods for importing MBOX files, automatically creating the users and threaded discussions in the forum.

There is also the whole m2f project, which originally linked mailing lists with phpbb but I think has been abstracted to allow connections between other platforms: http://mail2forum.com/

Cheers,

Tobias

CiviMail uses VERP

xurizaemon's picture

VERP is how CiviMail identifies whether the incoming mails are meaningful to it, or irrelevant.

CiviMail generates lots of messages with unique sub-addressed From addresses

civimail+32597827357.325923809.sdgjiosgo@example.com

Bounces might go to a separate address

bounce+32597827357.325923809.sdgjiosgo@example.com

Then when CiviMail clears its mailbox (which need not be a domain-wide catchall; you can just use subaddressing to catch "bounce+<ANYTHING>@example.com" and "civimail+<ANYTHING>@example.com") it handles the ones which are obviously recognisable, and ditches, forwards or moves (IMAP) the ones it doesn't.

VERP: http://en.wikipedia.org/wiki/Variable_envelope_return_path
Email sub-addressing: http://en.wikipedia.org/wiki/E-mail_address#Sub-addressing

VERP saves a lot of message parsing.

If my comments have helped you, please pay it forward!
Use issue queues to discuss module issues - this will help your questions assist others (including yourself!) in future.

technique useful for shared hosts too?

kyle_mathews's picture

This could be a good way to get OGM working for shared hosts. The mailhandler module works by checking a mailbox via POP/Imap on cron for new emails. You could merge perhaps the sub-addressing technique with mailhandler so that emails from groups are sent from group+group-name@example.com so that when people reply, it'd be easy to figure out from the mail header which group the email belongs to.

I'd thought about using something like mailhandler before but what gave me pause before was it'd be a lot of manual work to setup a new email address for each group that's created. But with sub-addressing, that wouldn't be a problem, you'd just set up one (e.g. group@example.com) and use sub-addressing to designate the group.

Kyle Mathews

kyle - look forward to being

petednz's picture

kyle - look forward to being able to review what you have done. Having grappled with the OG + Mess + Notif + MailHandler + MailComment pentagram several times - usually to the point of satisfaction - I understand your motivation to make things more integrated, and think your ideas eg of having the message being sent out to list before it has the chance to be modified by other stages of the process makes great sense.

pete davis : fuzion : connect + campaign + communicate

+1 on pete's point re having

tobias's picture

+1 on pete's point re having the message sent out to list before being modified by drupal. great idea. But also eager to ultimately be able to have replies posted in a threaded discussion on the site, with attachments intact.

-T

this is a terrific idea

tobias's picture

Hi Kyle,

I ammassively excited about this thread and your module, which I think addresses a major requirement I've had for my organization for years and years - we work in Africa where people rely alot more on their email for interacting than online blogs and forums. (google my name and mail2web and you'll see stuff going back many years including apps like fudforum, simple machines forum, mailman... !)

I have been chasing after arthurd since he posted the mail2og module on drupal.org in september 2009, which shows promise but is not completed. It also provides an email address for every organic group and a block to display it in the group so members are aware of it. Messages sent to that email address are posted to the group, and if you have various modules set up (the "trail" described above) then it is posted correctly, with attachments and images displayed inline. There is no requirement to install mailman etc...

I would have been happy to use mail2og - but it seems to be halted as a project. So I'd be glad to help out with your module with testing, requirements etc. Let me know when you post it to drupal.org or how I can help.

Cheers,

Tobias

Module dependencies

tobias's picture

Hi Kyle,

I just downloaded your og_mailinglist code to check it out and it looks like there are some module dependencies that are not explained in the install.txt:

Depends on: Organic groups (enabled), Spaces (missing), Views (enabled), Purl (missing), Context (missing), Context_contrib (missing), Features (missing)

I am eager to install and test this on my box, and would also be grateful for access to a sandbox group on your server so I can see it in action someplace where it is already working. Screenshots of admin interfaces would also be useful methinks so folks can see how it works in practice.

Do you have any thoughts as to when you will be ready to release og_mailinglist for wider testing on drupal.org?

Thanks,

Tobias

Right now og_mailinglist

kyle_mathews's picture

Right now og_mailinglist depends on purl. The reason is each group needs an email-friendly address to use as the list email address. We tried using just the group name at first. We ran into problems fairly quickly as group titles can have all sorts of characters that can't be used in email addresses. Switching to relying on the purl prefix group admins defined for their groups (we're running on the same stack of modules openatrium uses) solved that because it enforces machine-friendly names. It also makes for some nice synergies as the the email address of the group is the same as the group prefix (e.g. the list email address for the example.com/mygroup is mygroup@example.com).

That probably was a short-sighted decision as og_mailinglist is unusable for sites not using spaces/purl. I think the right way to fix this is to add a table to the database that handles the mapping between groups and their email addresses. If you use purl, it'd just use the purl group prefix. If you're not using purl, it'd convert your group title into a email-friendly string and save it to the table.

Kyle Mathews

How about having the

tobias's picture

How about having the alternative of purl prefix or allowing the group manager to provide the shortname to use for the mailing list?

I haven't worked with openatrium yet so will take a look at how it and purl works.

Thanks!

Tobias

Congrats and...

Nigel Cunningham's picture

Hi Kyle.

First, congrats on the first test release.

Now to the reply proper:

I'd really appreciate it if you did implement that table - I'm installing purl & spaces and so on just so I can get og_mailinglist going, and they're making me want to scream! :) Okay. Perhaps they'll make things simpler in the long run, and perhaps I just need to find the right tutorial, but I've already fixed problems that were caused by just enabling purl! It's feeling like a long road to be traveled before I even get to start on that postfix integration issue! :)

</grumble! - Sorry!>

Nigel

sorry :) I'd avoid trying to

kyle_mathews's picture

sorry :)

I'd avoid trying to get OGM running on a production site for now... unless you have a pretty good idea of what you're doing. The easiest way to test OGM is to build Eduglu from its make file -- http://github.com/KyleAMathews/eduglu_make

That way you'll be ensured you have the latest OGM code from github plus all the needed dependencies.

Kyle Mathews

Though if you're going to go

kyle_mathews's picture

Though if you're going to go the Eduglu make file route, wait until later today as I'm about to roll Alpha 2 and things as of right now aren't working.

Kyle Mathews

Nope.

Nigel Cunningham's picture

No, not going the Eduglu route.

I'm happy that I have a good enough idea of what I'm doing (plus lots of help from Google!). I'm planning on doing plenty of testing before this all goes live :)

Nigel

tobias's picture

Hi Kyle -

I've been thinking about this module a lot lately, and hope we can talk soon about it.

A feature I am realizing would be important to me is the ability for group managers to turn the mailing list on and off, and for individual group members to change their delivery preferences - email delivery on/off at the very least, but maybe even weekly digests?

Cheers,

Tobias

works like Google Groups

kyle_mathews's picture

The Module works just like Google Groups. 1 group = 1 mailing list.

Right now group managers can't turn off the mailing list in their group -- though I can see how that'd be useful.

Individual group members have three options for email delivery. Normal delivery, daily digest, or no emails.

Which would you prefer -- weekly or daily digests? I always sign up for daily digests on mailing lists. What advantages does a weekly digest have over a daily?

Kyle Mathews

daily vs weekly digests, list toggles

tobias's picture

Hi Kyle -

Thanks for your replies - this is all very interesting, and I am eager to implement this on Kabissa and help with testing/tweaking.

Awesome that group members can set up their own delivery options.

Weekly digests are nice for people who only access email periodically (like many Kabissa members in Africa who have to travel to get to email) and want to scan a few messages rather than go through a huge list of messages. That said, I do think daily digests are fine. I wouldn't want this to get too complicated.

Is it possible to unsubscribe or turn off delivery via email, eg send mail to listname-unsubscribe@bar.com?

Cheers,

Tobias

Technically possible yes,

kyle_mathews's picture

Technically possible yes, implemented now, no.

Kyle Mathews

Subscribing

Nigel Cunningham's picture

Subscribing. I'm just giving it a try now.

I'm using Postfix, so have to mangle things a little, but I'm getting there!

Oh very cool! If you get

kyle_mathews's picture

Oh very cool! If you get postfix working -- please share back the code. I know a lot of people use Postfix over Exim.

Kyle Mathews

Will do

Nigel Cunningham's picture

Will do. I have lots of projects on the go at once though, so might be a while.

Kabissa and og_mailinglists

tobias's picture

Hi Kyle,

FYI - I have started keeping track of ideas for setting up og_mailinglists on Kabissa on my issue tracker, at the following link. Gathering questions and wishlist for my setup which I will also compare with what is being shared here.

http://roadmap.kabissa.org/issues/41

Thanks!

Tobias

This is very exciting

teamA's picture

I've been searching around for a YahooGroups/GoogleGroups module and this seems to be exactly it. We have a client that has been using YahooGroups for years and is ready to have something he actually owns instead. I think you could end up with a very popular solution on your hands.

How close do you think you are to a beta stage? I am not a very advanced coder, but we would be happy to test and provide feedback.

Thanks for the efforts.

close

kyle_mathews's picture

It's pretty close. Close enough that you could test it w/o too much bother. The hardest thing to setup is the mailserver part of it -- I'm setting up my first one right now actually and am having a bit of go at it.

I'm actually working at getting it setup right now at community.eduglu.org -- the community site for my Eduglu (Drupal education distro) that I'm developing. It's not quite ready but as soon as I've successfully gotten it working there I'll make an alpha release on d.o. with full documentation. Before making the first beta release, I want to make sure it's been tested first on a number of other sites plus add several features including saving attachments to nodes and making it multi-site capable.

Kyle Mathews

awesome - thanks for the update!

tobias's picture

Hi Kyle -

I'm psyched to see you are making such good progress. Please keep us posted and let us know when and how we can help with testing etc. I am signed up now at your testing site and following @eduglu on twitter.

Let me know if I need to install exim to use this module - my preference would be to use a mailbox method if you can swing it.

Thanks,

Tobias

well more power to your

petednz's picture

well more power to your keyboard - i think there are a few of us sitting on the edge of having to go down the M+N+O route and would dearly love to be able to put that time in to helping you on this.

pete davis : fuzion : connect + campaign + communicate

Why not integrate with an actual mailing list

bibstha1's picture

How about the mailing part is handled by a real mailing list instead? Mailman for example.
Drupal will then only have to pull in and send out contents to the mailing list and mailing list does the job forward.
The concerns then are

a. Synchronization of users between mailing list and Drupal system
b. Receiving replies to mailing list as posts / comments in group.
c. Integration of mailing list creation / settings change with drupal

Have you considered this option?

That makes life more complicated in some ways.

Nigel Cunningham's picture

The advantage I see of doing things without mailman is that there's less to set up. I'm currently administering a couple of mailman lists that I'm hoping to replace with this module, partly because then I won't need mailman and will get forum integration.

Yes i thought the whole point

petednz's picture

Yes i thought the whole point of this was to step away from mailman - if you are wanting to go down the OG + Mailman route - check out, or help out, on this perhaps http://drupal.org/project/mailman_groups

pete davis : fuzion : connect + campaign + communicate

+1 agree with this. Looking

tobias's picture

+1 agree with this. Looking forward to an og module that provides list functionality independently of another mail system.

Looks like a nice project,

bibstha1's picture

Looks like a nice project, although it doesn't provide importing emails to forums and sending out comments as emails.
I agree a build in functionality might be quite helpful.

"an actual mailing list"

kyle_mathews's picture

OG Mailing list is and actual mailing list :) So why use two?

Conceptually, a mailing list is pretty simple. You send an email to an address. Some code there figures out who else is associated with that email address. Then the email is sent out again to everyone else. If someone replies, the process is just repeated.

So a mailing list needs three parts:
1) Some way to receive and send emails.
2) Some way to store the mapping between an email list address and the group of people that have "joined" that email list.
3) Code to wrangle the email headers so that when sent out again it goes to the right people.

So why didn't I just use Mailman or some other open source list software? Partly because of concerns other people have mentioned of the additional complexity of managing additional complexity. Partially because I wanted something that did exactly what I wanted and was easily customizable. But the real reason was none of the other list software's were designed to be integrated into some other piece of software. If Mailman had a nice API that would let me use OG to manage groups, that would let me push new nodes created in a group into Mailman, that'd let me pull new emails as they came in into Drupal -- then it'd of been an easy decision, just integrate. But Mailman doesn't let you do that and afaik, neither do any of the other list software.

Kyle Mathews

this is how fud forum works

tobias's picture

Hi bibstha1-

Thanks for the comment.

This is how fudforum works (http://fudforum.org) - not drupal. The way it works is simple: each forum can be configured with two email addresses (one to subscribe to the list and the other to send to the list). You then separately create the list and subscribe the forum to it. It also, if I recall correctly, provides a field to enter regular expressions for text to filter out and not include on the forum (eg quoted text, signatures etc).

I think that would be a reasonable compromise that I could live with, and that even comes with certain advantages.. like the fact that with this scenario group managers could basically use the drupal group to create an online interface for an existing list hosted anywhere. Cool.

All that said, if it's possible for Kyle to create this so it is all handled natively in drupal, without hard to manage external dependencies, then I'd prefer it. I don't have mailman at the moment and am not eager to set it up again on my server. :) I also think it's confusing for people to have to subscribe to two places and see potential nightmare maintenance issues.

Cheers,

Tobias

Ready to test the Alpha

teamA's picture

I will be happy to run through your install instructions once you get there.

Thanks,
Aris

Just a query, how are the

bibstha1's picture

Just a query, how are the list email addresses created?

say i create two group (programmers, designers), would respective emails like programmers@example.com and designers@example.com be created themselves?

group title

kyle_mathews's picture

Right now, email address are created from the PURL prefix the group admin chooses. I hope to change that at some point so the email address can either be created automatically from the group name or be chosen by the group admin.

Kyle Mathews

Subscribing

danielstrum's picture

This sounds like exactly what I am looking for. I really don't have the skills to contribute to this development but I will stay up on how things are progressing.

ds

Light at the end of the tunnel

auzigog's picture

I am in the middle of writing an application to Google Summer of Code to work on a project management tool for activists and organizers: http://wiki.auzigog.com/Drupal_Projects_for_Activists_and_Movements_-_GS...

I had the exact same goal in mind as kyle. I'm glad this project is already underway!

Kyle, I'd love to collaborate on this project, update it for Drupal 7, and make a beta release available if my GSoC project is accepted. I'd also be interested in finding a way to adapt it to postfix or find a way to tie it into a Drupal mailing system without having a delay in messages being sent.

For some background, here are some modules I found along the way:
http://drupal.org/project/mail2og
http://drupal.org/handbook/modules/og2list (old version of the same thing we have here, but based on postfix)
http://drupal.org/project/mailman_manager + http://drupal.org/project/mailman_groups
http://drupal.org/project/mailcomment
http://drupal.org/project/mailing_list (D7 branch, but doesn't seem to work for discussion)

would love the help

kyle_mathews's picture

Hey Auzigog,
Sounds like a great GSOC project!

I'd definitely appreciate another set of eyes on the code (I'm sure I've done some really dumb things) and help adding different features.

I had another lookat og2list and it (was) a lot more cool than I'd thought. They had postfix integration so we'd be able to copy their ideas there.

They had a lot of discussion there about how to run og2list on shared hosts (sound familiar? :D) http://groups.drupal.org/node/1725 It doesn't look like they were ever able to crack that particular nut.

Kyle Mathews

Magic Groups

bensheldon's picture

It sounds like you're pretty far along, but back in 2006 Zack Rosen put together a bunch of scripts for Drupal 4.7 that provided functionality pretty close to what you're describing. It might be useful (or it might not):

http://www.zacker.org/magic-groups-screencast

I've always wished someone created a strong mailing list solution, so I wish you the very best!

magic groups

kyle_mathews's picture

I did run across what you guys did back in the day. Very impressive! Thanks for the encouragement! Hopefully OGM can get the point where it's pretty painless for someone to setup and start running.

Kyle Mathews

OG Mailinglist is ready for testing

kyle_mathews's picture

I was also hoping to be able to say that the code is up on drupal.org... but I can't seem to login to Drupal's CVS system (@#$@# CVS) so we'll worry about that another day. We'll work from github still for the time being.

Anyways, last week I spent 2.5 days refactoring og_mailinglist so it'll work with virtually unlimited numbers of sites and servers on one server. This actually went pretty smoothly thanks to Drush 3.0's site alias functionality.

What's really needed now is better documentation. I've commited to github some initial documentation but I'm looking for a few brave (and knowledgeable -- there's only so much hand holding I can do) volunteers who'll test og_mailinglist on their own servers and work with me what's missing / needs clarification on the documentation.

And just a reminder of the server requirements. You'll need an Ubuntu / Debian box which you have root access. It still requires Space/PURL (haven't had a chance to refactor that away). The easiest way probably to meet all the requirements is to use drush_make to build a new copy of eduglu from my make file on github.

If you can help out, please contact me.

Kyle Mathews

alpha 1 tagged and released

kyle_mathews's picture

Get it while it's hot at http://drupal.org/project/og_mailinglist

Kyle Mathews

Great to see the progress!

Nigel Cunningham's picture

Hi Kyle!

Great to see the progress you're making. Sorry I haven't committed that Postfix work yet - too many balls in the air!

I promise to get there soon!

Nigel

And alpha 2 is out as well

kyle_mathews's picture

In case you missed it, I rolled out the 2nd alpha this morning. See the release notes here:
http://drupal.org/node/798112

Kyle Mathews

Great to see!

Nigel Cunningham's picture

I'm pulling from git, so got it anyway :)

Looking through the diff from April 19 now. Doesn't that seem a long time ago?

  • I'd suggest making the path that's used for temporary files configurable. Some hosted installations might not have access to /tmp. The use of a number for the filename might also conflict with other users of /tmp.
  • Probably also a good idea to delete the file afterwards (security issue?)
  • Files should end in a newline.
  • In og_mailinglist_save_group_thread_subscriptions, the comment should say "Clean bad email addresses from list"?
  • This makes me wonder if bounce messages (received in reply) will get posted.
  • Love the removal of the spaces dependency!
  • "Discussions" in og_mailinglist_send_digest_emails should have a lower case 'd'.
  • Creating the /tmp file could potentially fail. I don't see any error handling for that at the moment.
  • At the top of og_mailinglist.module, line 6, s/Mails/Mail/.
  • I'm not convinced that leaving a group belongs in the subscription settings form. You'd at least need to take account of madatory groups.
  • I'm also not convinced that og_mailinglist should handle sending the email back out. I know that makes threading and such like work properly. It just feels more like something that belongs to one of those subscription/messaging modules.
  • The use of two variables - $is_public and $is_private - in og_mailinglist_transport.inc looks unnecessary and confusing. Perhaps make one $group_public and the other $node_private or (better still) just use one var?

Code review over :) Thanks for all your great work. Now I'll give it a try!

Nigel

code review review

kyle_mathews's picture

I created an issue for the temp file issues at http://drupal.org/node/799138

Files now end in new lines - http://github.com/KyleAMathews/og_mailinglist/commit/2fa3d28db814ac1dee1...

Not sure about what will happen with bounce messages. In any case, I've done nothing so far to support handling them.

On leaving groups on the subscription settings form. OG Mailinglist is being written primarily for Eduglu, a social learning product. I put that in because OG's handling of leaving groups is a mess frankly. You have to go from group to group to group to leave each one which is slow and takes an enormous number of clicks. So it makes sense to me to have that subscription page be the place where members can adjust all their group-related settings. But I can see it wouldn't always be appropriate. Perhaps that feature can be made optional.

"Threading and such" is kind of important. It'd be nice of course to make OGM work with notifications/messaging but I'm about 95% sure their architecture wouldn't support what OG Mailinglist is trying to do so integration would probably require rather extensive work on their part.

I added an explanation of what's going on with is_private/is_public. If you can think of a cleaner way to do that, that'd be appreciated. http://github.com/KyleAMathews/og_mailinglist/commit/690893ec69248e121eb...

Kyle Mathews

Thanks!

Nigel Cunningham's picture

Thanks for your responses.

Re newlines at the end of files, I was meaning at the end of the source code files - sorry for my ambiguity!

I do like the idea of a form in which you can mass-manage your group subscriptions. It's not core functionality for og_mailinglists though - perhaps it could be split off to a separate project?

Re notifications/messaging, okay. I'll acquiesce for now and look at the issue some more once the basic functionality is up and running. You wrote about the reasons for not using notifications/messaging a bit earlier in this thread, right?

Re is_private, is_public, perhaps rename the variables to group_private and node_private?

PS: Thanks for pulling my patches in.

Is it working and worth changing my hosting?

jasonb815's picture

Hi Kyle! I've been looking around a bit for a solution like this and like what I've read here. The problem is that my hosting company (GatorHost) won't let me add the MailParse extension to the plan I have. I'll have to basically triple my monthly payment (which honestly isn't alot at the moment) to go up to a Virtual Server plan where I can add the PHP extensions I need.

Before I do this, I want to make sure :

A) This solution works reliably

B) A. is really the only thing that matters.. :)

thanks again for the work you are doing for the community!

OGM alive and well

kyle_mathews's picture

Yes OG_Mailinglist is alive and kicking and working quite reliably. On island.byu.edu it's handled some 27,000 emails in the past month w/o so much of a glitch. Development is moving forward at a slow but steady rate.

Question though -- will you have root access on the virtual server plan? To install OGM, you'll need that access.

Also, for future new visitors to this thread. We should probably let it die now as it's served it's purpose. If you have more questions, visit us over at the OG Mailinglist Issue queue at http://drupal.org/project/issues/og_mailinglist?status=All&categories=All

Kyle Mathews

Contributed Module Ideas

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: