Hey folks,
I've been busy off and on the past month or so writing a native mailing list for Drupal. It integrates natively with Organic Groups (i.e. each group becomes a list w/ its own email address) and behaves exactly like popular open source mailing lists like Mailman/Minimilist (it should as I copied their algorithms). I'll be releasing it on d.o. soonish but I had a few questions I wanted to ask first.
First -- what should I name it? Mailinglist is the obvious name and is open in the Drupal module namespace. Any objections or better ideas?
Second -- I'd love to discuss possible architectures with some interested developers. Right now the code is... a mess, to be charitable -- mostly due to the many dead-ends I hit before getting things working. I need to do some serious refactoring before doing the initial release and wanted some feedback / ideas on different ways to architect the code.
Comments
Ooooh
I love this idea! I can already think of 3 sites I want to use it on. :) "mailinglist" sounds fine as a name.
Michelle
Drupal has the SimpleNews
Drupal has the SimpleNews project. It can't automatically create mailing lists based on OG subscriptions, but it's a fairly capable mailer. Did you consider basing your work on that project?
The Boise Drupal Guy!
This isn't a mailinglist
This isn't a mailinglist manager in the sense of Simplenews where emails go only in one direction -- out -- it's a mailing list along the lines of the Drupal mailing lists -- http://drupal.org/mailing-lists -- which provide a simple way for group email conversations to happen.
WIth this module + OG, you have a Google Groups clone. Where you join a group online. You can search/browse past discussions online. And you can start/contribute to discussions either through the web interface or through the group's email address (it's based on spaces/purl so it uses the purl prefix for the group's email address).
Kyle Mathews
Ah, okay. I get it. Sorry, I
Ah, okay. I get it. Sorry, I guess "mailing list" is kind of ambiguous in that regard.
The Boise Drupal Guy!
Yeah -- it definitely is. Not
Yeah -- it definitely is. Not much we can about that though at this point.
Kyle Mathews
Maybe...
Mail Group?
Michelle
current alternatives
I think there is a current organic group based solution:
Most of this combination is already running here on groups.drupal.org. I'd like to deploy the last pieces of this (mailcomment + mailhandler) in the next few months but there are a few features necessary first. On a similar note, we (GVS) are using this as the basis to rebuild the IxDA.org site. So, I hope you will consider that combination first. It would be great to have more people working on this chain of modules and I'm happy to chat about it to help get you up and running.
knaddison blog | Morris Animal Foundation
Currently using this stack
Along with comment upload, to allow attachments to be handled on comments via email --
Setting this up is pretty well beyond the reach of even moderately technical users.
I'm curious to see how Kyle's code/solution compares. Over time, I see the potential for these pieces to morph together, but given that Kyle has something that appears to be working, it makes sense to look at both options.
My .02 --
Cheers,
Bill
FunnyMonkey
I've been using this chain
I've been using this chain (and variations) for 1.5 years now so I'm very familiar with those modules -- and their limitations. The beating I took against their limitations over the past year or so are what motivated me to write my own setup. I'm definitely not in favor of module duplication but the amount of time I spent writing code to work around problems (never completely successfully) I found w/ notifications/messaging far exceeded the couple of weeks I spent rewriting them to meet my specific needs.
For the past few years I've been building a g.d.o. like site (https://island.byu.edu) which is used by upwards of 500 student/faculty/administrators at BYU. The basic idea is to create a platform for classrooms/majors/clubs/interest groups to create online spaces to easily share communicate with each other. It's predecessor was a simple mailing list run by mailman. The list did (and still does in its new drupal clothing) get a significant bit of traffic -- ~5 new posts a day or so.
We moved the list to the island originally for two reasons. First, to provide the platform to easily allow anyone to create their own mailinglist/group, and second, to address complains about "off-topic" emails on the list -- mostly people posting jobs. So the list got moved to https://island.byu.edu/isys and a jobs group was created at https://island.byu.edu/jobs.
And everyone was happy... sort of.
The first major complaint from the students was the emails were ugly -- which they were. So I dove into the notification templates and after a lot of tweaking got emails mostly working (though they never were very happy with the dividing line stuff).
The second big complaint was the loss of email history. In a list, you reply to an email and the other list members get your reply /and/ what you're replying to. Which is really useful, and like a lot of things, not noticed until it's gone. Mailcomment has the divider line thingy at the top that you're supposed to reply above. It uses that line to cut out the reply from the rest of the email so it can make a nice clean new comment. The problem of course that it cuts off what you were replying to. This doesn't matter some of the time but occasionally people on lists will reply to an early email and say something like "haha, so funny, totally agree" but since what they were replying to was stripped away and not sent out -- others would reply back confused asking what they were agreeing with.
I sort of fixed that by manually inserting into emails the last 5 comments made on a post which sort of worked but not entirely as often times what someone was replying to wasn't the last comment made. That fix was really painful and quite hackish.
Another problem with that stack is it's quite slow -- meaning the turn-around time from when someone sends an email to when everyone else gets the email can be quite long. Depending on how often you run cron it could be 20-30 minutes before everyone has gotten the email. Which is completely unacceptable for a mailinglist. What we did to fix that was first we did an integration with Exim4, our MTA, so that when new emails were received, exim would call a php script which would figure out what group to put the post/comment in and do it immediately. Then to speed up pushing out emails again, we wrote a PHP daemon to hit messaging and notification's cron functions every 30 seconds. Together those two dropped the loop time down to 3-5 minutes -- which is still 2-3 minutes slower than a real mailing list.
So the above problems we mostly fixed but the below we couldn't which proved the proverbial straws which broke the camel's back.
The email formatting was consistently fouled up. With notifications/messaging, emails sent to a group are first saved to the site and only then sent out. That could be fine except the messaging module insists on running all text it sends out through a Drupal input filter which in subtle and not so subtle ways would often change people's formatting. Particularly uncute was Drupal's insistence on removing any html added to emails -- not very helpful when someone is asking for help on html.
An associated problem -- right now it's pretty much impossible to run the above stack and retain the html from emails. Plain-text emails are just uglier plus people were annoyed that formatting they put in their emails would get stripped.
The last big problem is when people send emails to one of the group lists, they get an email back. There was no way of stopping that (out of the box anyways).
And there's probably a half-dozen other minor annoyances with the old setup I'm not remembering.
In short, the stack you listed above is inadequate for anyone who's ever used a real mailing list. It's somewhat like the uncanny valley concept. As long as my website didn't really act like a mailing list -- no one complained too much. They treated it like a forum with email notification. But the closer I made it to a mailing list, the more they noticed it's deficiencies vs. a real mailing list. So I stopped trying to pretend messaging/notifications was a real mailing list and just wrote one.
Kyle Mathews
Interesting. Thanks for the
Interesting. Thanks for the write-up.
I've previously done daemons in PHP as well (particularly Drupher), but I've just used inetd (FreeBSD) or launchd (OS X), which function as "super-daemons" capable of running simpler scripts as daemons. I'll have to keep that PEAR package in mind should I ever need to do anything more complex.
The Boise Drupal Guy!
The second big complaint was
IMO what's really needed for that is just proper mail headers so that they thread on the client side. Then people can see it. We're also using this with Markdown which means that > turns into blockquote. It can require messing with replies a little more to get the right threading but I think people can be trained.
I guess I see this, but many lists use manual moderators and often the mail servers along the way slow things down. People should get some tea.
Can't this be solved by "escaping html" rather than stripping it? Or tweaking the input format that's used for sending mails. I do generally see what you're saying, but mail clients are notorious for their alternate ideas on proper rendering.
http://example.com/admin/messaging/notifications - "Notify poster of own posts
Notifies a node poster about their own posts. Useful principally during testing. Default is OFF."
Right?
Yeah, I get that and I appreciate your detailed list of feedback. I'm just saddened because I've seen you do good work and hope that you could put it toward these modules since I feel confident in them in general.
knaddison blog | Morris Animal Foundation
Just to be clear -- I have
Just to be clear -- I have nothing against messaging / notifications. I think they're great modules for what they're designed for -- which unfortunately for me, as I've found out the hard way, is not running mailing lists.
They were designed, essentially, to push content out. Something happens on the site and people who care get informed about it in the way they choose. It's very flexible and solves a wide variety of problems. And they wrote into their code those assumptions. And after a year of trying to work around those assumptions, I gave up and wrote a module for my needs.
And long-term I'd really like to fold the work I'm doing back into the messaging/notifications stuff -- I'm starting my own company and have gobs to do and really would prefer to have help maintaining my code. So hopefully this module will be a temporary fork to prove out a point and then messaging/notifications can be reworked to include it.
BTW, the one big assumption messaging/notification makes that I couldn't ever work around is that all notifications must originate from within Drupal. All /real/ mailing lists (e.g. Mailman and Minimalist) work as follows. Email comes in. The subject is optionally changed (e.g. add a [list-name]), a footer is added, email address of members are added as BCs, and then it's sent out again. And that's it. It's very simple and it works perfectly. And that's what my module does as well. I don't know exactly how notifications/messaging would have to change to allow that workflow but I'd be happy to help make it happen as the flow I described is far superior than using mailcomment et. al.
Kyle Mathews
on architecture
Back the architecture question. There are two patterns I could see fitting this module's needs.
First is the more traditional Drupal style of architecture. Where I create an object/array to hold data and pass it around to different functions with built-in hooks to let other modules override behavior.
Second, is create an object with methods that do must of the work internally and drupal-independent. This object would take the raw email, parse out attachments, plain-text, html-text, add footers, adjust subjects, etc. The advantage of this is it's somewhat cleaner (I think) and since it'd be Drupal independent, other coders could take the mailinglist class and create lists for other platforms.
Any thoughts on these two styles? Or is there other options I'm not thinking. I'd really like some feedback here as it'll be much harder to decommit myself once I've gone down a path.
Kyle Mathews
Quick update. All the code is
Quick update. All the code is now on github. I'll be cleaning things up plus adding a few more quick features before making an initial release on d.o. Any help / feedback would be greatly appreciated.
For the module's name -- I ended up calling it og_mailinglist.
Kyle Mathews
Oh -- and the link
Oh -- and the link http://bit.ly/4XbwR6
Kyle Mathews
thanks
Kyle,
This would be awesome. Most of my users prefer Google groups like functionality. However, security is a big issue for us - plus, we would like to be able to send file attachments as well. Is this something you can incorporate soon?
Thanks much,
atruser1
There's already as much
There's already as much security in place as Google Groups -- i.e. it checks the originating email address to see if the address is from an active user in that group. Actually Google Groups has a lot of problems with spam -- see this post about problems the jQuery group faced: http://ejohn.org/blog/google-groups-is-dead/
Since this module is a native Drupal solution -- it'd have a leg up as you could run new posts through Mollem or something to check if they're spam.
On file attachments -- yes, that's on the pipeline. Right now File attachments are passed along in the emails sent to other people in the group -- but aren't saved as a file attachment.
Kyle Mathews
Kinda surprised to hear spam
Kinda surprised to hear spam is such a problem. You'd think running incoming messages through the same system they use to filter spam for Gmail would be obvious.
The Boise Drupal Guy!
From the rumblings I've heard
From the rumblings I've heard -- it doesn't sound like Groups is a very high priority project. They've been criticized quite a bit for leaving their valuable archive of usenet articles virtually unusable -- http://www.wired.com/epicenter/2009/10/usenet/
Kyle Mathews
I have some other ideas for
I have some other ideas for tightening up security -- but I won't have time to implement them for awhile. I'd be happy to discuss ideas though.
Kyle Mathews
Just what I needed
Our church website uses Drupal and this is exactly what we've been looking for. I dont' have a lot of time messing around to get Mailman to work so an all-inclusive mailing list solution within Drupal would be great.
Will I need to upgrade to Drupal 6.x to try this out or is it 5.x compatible?
Thanks!
Gary
The module will be Drupal 6.x
The module will be Drupal 6.x only. It'd be fairly trivial to backport so if I'd create a 5.x version if someone writes a patch.
BTW, as I hadn't made this clear before -- to work fully as an email list -- you'll need to have root access and some sys admin ability. No more than you'd need to install mailman or other mailinglist software but this module is definitely not shared-host capable.
Kyle Mathews
Funding if it works shared
We need this functionality but we are on a shared host. If it is practical to adapt it to shared hosting we are willing to help pay for that work and some development time in general.
It might be able to run on
It might be able to run on shared hosts... I'm actually still somewhat of a email noob, even after all the mucking around I had to do to get this module working right.
Why it requires your own server right now is I wrote it to use an advanced feature of exim4 -- which lets you use an external script to transport new email. So for any email that comes in, Exim just hands it over to my module and says, "do what you want with that". The module then figures out if there's a group associated with the email address and if there is, it pulls out the body of the email and creates a new node. I'm not sure how to duplicate that sort of thing on a shared host.
Here's the best idea I've come across is to use something called a catch-all email address. Per the wikipedia description (http://en.wikipedia.org/wiki/Catch-all) a catch-all email address is where all emails not sent to an existing email get dumped. So og_mailinglist could just hit that email address every few minutes to see if there's new emails and then process them as normal.
Does anyone know if that would work? Mailhandler (http://drupal.org/project/mailhandler) might have some useful code/concepts here.
Kyle Mathews
Careful with catch-all
I used to have them on all my domains and found them very handy. Then I found out the hard way that spammers find them handy, too. :( I got flooded with "out of office" type messages all of a sudden and found out that spammers use domains with catch-alls as their "from" address because the catch-all ensures that they show up as legit emails.
Michelle
catchall is fine
I think using catchall is fine, as long as you properly handle mail to the domain. That is, deliver postmaster@foo.com mail to yourself, list-related mail to the lists, and bounce the rest.
As long as that happens your domain won't be exploited by spammers.
Cheers,
Tobias
.
Er... If you bounce everything else, it's not a catchall, it's a bounceall. :)
The whole point of a catchall is to catch ALL mail. If you're defining specific items to let through and bouncing the rest, that's normal usage, not a catchall.
Michelle
Hi Michelle - Not sure what
Hi Michelle -
Not sure what you have in mind, but I am thinking of a scenario where you have a catchall mailbox that collects all the mail, then lets your script pick up the mail and handle it. So it wouldn't be the MTA that is bouncing or delivering the mail, but your script. This has the advantage that it can be used by anybody on a shared host.
CiviCRM CiviMail does this on my server to handle reply email and it works incredibly well. It uses imap2soap.pl to do it - I just googled "CiviCRM imap2soap" and came across a bunch of stuff including this recipe that Kyle might find useful: http://back-end.org/blog/stevem/howto_civimail_return_channel_for_multis...
Cheers,
Tobias
.
Ah, I see. You're talking about adding another layer between the catchall and the mail handling. That sounds like it would work.
Michelle
CiviMail looks like it'd have
CiviMail looks like it'd have some useful ideas/code. That's my favorite part about open source programming. Almost always someone's already done what you want to do -- you just have to poke around the intertubes long enough until you find it. Case in point, the core algorithm for appending footers to new emails on og_mailinglist was copied from Minimalist, a perl mailinglist manager - http://www.mml.org.ua/
Kyle Mathews
Check out fudforum too
Hi Kyle- Yes, I love this aspect of open source too. :)
On that note, you may also want to look at http://fudforum.org which includes a script called maillist.php to integrate a forum with a mailing list server like mailman. I last played with it in 2006 but it worked extraordinarily well - it even provided methods for importing MBOX files, automatically creating the users and threaded discussions in the forum.
There is also the whole m2f project, which originally linked mailing lists with phpbb but I think has been abstracted to allow connections between other platforms: http://mail2forum.com/
Cheers,
Tobias
CiviMail uses VERP
VERP is how CiviMail identifies whether the incoming mails are meaningful to it, or irrelevant.
CiviMail generates lots of messages with unique sub-addressed From addresses
civimail+32597827357.325923809.sdgjiosgo@example.com
Bounces might go to a separate address
bounce+32597827357.325923809.sdgjiosgo@example.com
Then when CiviMail clears its mailbox (which need not be a domain-wide catchall; you can just use subaddressing to catch "bounce+<ANYTHING>@example.com" and "civimail+<ANYTHING>@example.com") it handles the ones which are obviously recognisable, and ditches, forwards or moves (IMAP) the ones it doesn't.
VERP: http://en.wikipedia.org/wiki/Variable_envelope_return_path
Email sub-addressing: http://en.wikipedia.org/wiki/E-mail_address#Sub-addressing
VERP saves a lot of message parsing.
If my comments have helped you, please pay it forward!
Use issue queues to discuss module issues - this will help your questions assist others (including yourself!) in future.
technique useful for shared hosts too?
This could be a good way to get OGM working for shared hosts. The mailhandler module works by checking a mailbox via POP/Imap on cron for new emails. You could merge perhaps the sub-addressing technique with mailhandler so that emails from groups are sent from group+group-name@example.com so that when people reply, it'd be easy to figure out from the mail header which group the email belongs to.
I'd thought about using something like mailhandler before but what gave me pause before was it'd be a lot of manual work to setup a new email address for each group that's created. But with sub-addressing, that wouldn't be a problem, you'd just set up one (e.g. group@example.com) and use sub-addressing to designate the group.
Kyle Mathews
kyle - look forward to being
kyle - look forward to being able to review what you have done. Having grappled with the OG + Mess + Notif + MailHandler + MailComment pentagram several times - usually to the point of satisfaction - I understand your motivation to make things more integrated, and think your ideas eg of having the message being sent out to list before it has the chance to be modified by other stages of the process makes great sense.
pete davis : fuzion : connect + campaign + communicate
+1 on pete's point re having
+1 on pete's point re having the message sent out to list before being modified by drupal. great idea. But also eager to ultimately be able to have replies posted in a threaded discussion on the site, with attachments intact.
-T
this is a terrific idea
Hi Kyle,
I ammassively excited about this thread and your module, which I think addresses a major requirement I've had for my organization for years and years - we work in Africa where people rely alot more on their email for interacting than online blogs and forums. (google my name and mail2web and you'll see stuff going back many years including apps like fudforum, simple machines forum, mailman... !)
I have been chasing after arthurd since he posted the mail2og module on drupal.org in september 2009, which shows promise but is not completed. It also provides an email address for every organic group and a block to display it in the group so members are aware of it. Messages sent to that email address are posted to the group, and if you have various modules set up (the "trail" described above) then it is posted correctly, with attachments and images displayed inline. There is no requirement to install mailman etc...
I would have been happy to use mail2og - but it seems to be halted as a project. So I'd be glad to help out with your module with testing, requirements etc. Let me know when you post it to drupal.org or how I can help.
Cheers,
Tobias
Module dependencies
Hi Kyle,
I just downloaded your og_mailinglist code to check it out and it looks like there are some module dependencies that are not explained in the install.txt:
Depends on: Organic groups (enabled), Spaces (missing), Views (enabled), Purl (missing), Context (missing), Context_contrib (missing), Features (missing)
I am eager to install and test this on my box, and would also be grateful for access to a sandbox group on your server so I can see it in action someplace where it is already working. Screenshots of admin interfaces would also be useful methinks so folks can see how it works in practice.
Do you have any thoughts as to when you will be ready to release og_mailinglist for wider testing on drupal.org?
Thanks,
Tobias
Right now og_mailinglist
Right now og_mailinglist depends on purl. The reason is each group needs an email-friendly address to use as the list email address. We tried using just the group name at first. We ran into problems fairly quickly as group titles can have all sorts of characters that can't be used in email addresses. Switching to relying on the purl prefix group admins defined for their groups (we're running on the same stack of modules openatrium uses) solved that because it enforces machine-friendly names. It also makes for some nice synergies as the the email address of the group is the same as the group prefix (e.g. the list email address for the example.com/mygroup is mygroup@example.com).
That probably was a short-sighted decision as og_mailinglist is unusable for sites not using spaces/purl. I think the right way to fix this is to add a table to the database that handles the mapping between groups and their email addresses. If you use purl, it'd just use the purl group prefix. If you're not using purl, it'd convert your group title into a email-friendly string and save it to the table.
Kyle Mathews
How about having the
How about having the alternative of purl prefix or allowing the group manager to provide the shortname to use for the mailing list?
I haven't worked with openatrium yet so will take a look at how it and purl works.
Thanks!
Tobias
Congrats and...
Hi Kyle.
First, congrats on the first test release.
Now to the reply proper:
I'd really appreciate it if you did implement that table - I'm installing purl & spaces and so on just so I can get og_mailinglist going, and they're making me want to scream! :) Okay. Perhaps they'll make things simpler in the long run, and perhaps I just need to find the right tutorial, but I've already fixed problems that were caused by just enabling purl! It's feeling like a long road to be traveled before I even get to start on that postfix integration issue! :)
</grumble! - Sorry!>
Nigel
sorry :) I'd avoid trying to
sorry :)
I'd avoid trying to get OGM running on a production site for now... unless you have a pretty good idea of what you're doing. The easiest way to test OGM is to build Eduglu from its make file -- http://github.com/KyleAMathews/eduglu_make
That way you'll be ensured you have the latest OGM code from github plus all the needed dependencies.
Kyle Mathews
Though if you're going to go
Though if you're going to go the Eduglu make file route, wait until later today as I'm about to roll Alpha 2 and things as of right now aren't working.
Kyle Mathews
Nope.
No, not going the Eduglu route.
I'm happy that I have a good enough idea of what I'm doing (plus lots of help from Google!). I'm planning on doing plenty of testing before this all goes live :)
Nigel
is there going to be a toggle to subscribe to mailing lists?
Hi Kyle -
I've been thinking about this module a lot lately, and hope we can talk soon about it.
A feature I am realizing would be important to me is the ability for group managers to turn the mailing list on and off, and for individual group members to change their delivery preferences - email delivery on/off at the very least, but maybe even weekly digests?
Cheers,
Tobias
works like Google Groups
The Module works just like Google Groups. 1 group = 1 mailing list.
Right now group managers can't turn off the mailing list in their group -- though I can see how that'd be useful.
Individual group members have three options for email delivery. Normal delivery, daily digest, or no emails.
Which would you prefer -- weekly or daily digests? I always sign up for daily digests on mailing lists. What advantages does a weekly digest have over a daily?
Kyle Mathews
daily vs weekly digests, list toggles
Hi Kyle -
Thanks for your replies - this is all very interesting, and I am eager to implement this on Kabissa and help with testing/tweaking.
Awesome that group members can set up their own delivery options.
Weekly digests are nice for people who only access email periodically (like many Kabissa members in Africa who have to travel to get to email) and want to scan a few messages rather than go through a huge list of messages. That said, I do think daily digests are fine. I wouldn't want this to get too complicated.
Is it possible to unsubscribe or turn off delivery via email, eg send mail to listname-unsubscribe@bar.com?
Cheers,
Tobias
Technically possible yes,
Technically possible yes, implemented now, no.
Kyle Mathews
Subscribing
Subscribing. I'm just giving it a try now.
I'm using Postfix, so have to mangle things a little, but I'm getting there!
Oh very cool! If you get
Oh very cool! If you get postfix working -- please share back the code. I know a lot of people use Postfix over Exim.
Kyle Mathews
Will do
Will do. I have lots of projects on the go at once though, so might be a while.
Kabissa and og_mailinglists
Hi Kyle,
FYI - I have started keeping track of ideas for setting up og_mailinglists on Kabissa on my issue tracker, at the following link. Gathering questions and wishlist for my setup which I will also compare with what is being shared here.
http://roadmap.kabissa.org/issues/41
Thanks!
Tobias
This is very exciting
I've been searching around for a YahooGroups/GoogleGroups module and this seems to be exactly it. We have a client that has been using YahooGroups for years and is ready to have something he actually owns instead. I think you could end up with a very popular solution on your hands.
How close do you think you are to a beta stage? I am not a very advanced coder, but we would be happy to test and provide feedback.
Thanks for the efforts.
close
It's pretty close. Close enough that you could test it w/o too much bother. The hardest thing to setup is the mailserver part of it -- I'm setting up my first one right now actually and am having a bit of go at it.
I'm actually working at getting it setup right now at community.eduglu.org -- the community site for my Eduglu (Drupal education distro) that I'm developing. It's not quite ready but as soon as I've successfully gotten it working there I'll make an alpha release on d.o. with full documentation. Before making the first beta release, I want to make sure it's been tested first on a number of other sites plus add several features including saving attachments to nodes and making it multi-site capable.
Kyle Mathews
awesome - thanks for the update!
Hi Kyle -
I'm psyched to see you are making such good progress. Please keep us posted and let us know when and how we can help with testing etc. I am signed up now at your testing site and following @eduglu on twitter.
Let me know if I need to install exim to use this module - my preference would be to use a mailbox method if you can swing it.
Thanks,
Tobias
well more power to your
well more power to your keyboard - i think there are a few of us sitting on the edge of having to go down the M+N+O route and would dearly love to be able to put that time in to helping you on this.
pete davis : fuzion : connect + campaign + communicate
Why not integrate with an actual mailing list
How about the mailing part is handled by a real mailing list instead? Mailman for example.
Drupal will then only have to pull in and send out contents to the mailing list and mailing list does the job forward.
The concerns then are
a. Synchronization of users between mailing list and Drupal system
b. Receiving replies to mailing list as posts / comments in group.
c. Integration of mailing list creation / settings change with drupal
Have you considered this option?
That makes life more complicated in some ways.
The advantage I see of doing things without mailman is that there's less to set up. I'm currently administering a couple of mailman lists that I'm hoping to replace with this module, partly because then I won't need mailman and will get forum integration.
Yes i thought the whole point
Yes i thought the whole point of this was to step away from mailman - if you are wanting to go down the OG + Mailman route - check out, or help out, on this perhaps http://drupal.org/project/mailman_groups
pete davis : fuzion : connect + campaign + communicate
+1 agree with this. Looking
+1 agree with this. Looking forward to an og module that provides list functionality independently of another mail system.
Looks like a nice project,
Looks like a nice project, although it doesn't provide importing emails to forums and sending out comments as emails.
I agree a build in functionality might be quite helpful.
"an actual mailing list"
OG Mailing list is and actual mailing list :) So why use two?
Conceptually, a mailing list is pretty simple. You send an email to an address. Some code there figures out who else is associated with that email address. Then the email is sent out again to everyone else. If someone replies, the process is just repeated.
So a mailing list needs three parts:
1) Some way to receive and send emails.
2) Some way to store the mapping between an email list address and the group of people that have "joined" that email list.
3) Code to wrangle the email headers so that when sent out again it goes to the right people.
So why didn't I just use Mailman or some other open source list software? Partly because of concerns other people have mentioned of the additional complexity of managing additional complexity. Partially because I wanted something that did exactly what I wanted and was easily customizable. But the real reason was none of the other list software's were designed to be integrated into some other piece of software. If Mailman had a nice API that would let me use OG to manage groups, that would let me push new nodes created in a group into Mailman, that'd let me pull new emails as they came in into Drupal -- then it'd of been an easy decision, just integrate. But Mailman doesn't let you do that and afaik, neither do any of the other list software.
Kyle Mathews
this is how fud forum works
Hi bibstha1-
Thanks for the comment.
This is how fudforum works (http://fudforum.org) - not drupal. The way it works is simple: each forum can be configured with two email addresses (one to subscribe to the list and the other to send to the list). You then separately create the list and subscribe the forum to it. It also, if I recall correctly, provides a field to enter regular expressions for text to filter out and not include on the forum (eg quoted text, signatures etc).
I think that would be a reasonable compromise that I could live with, and that even comes with certain advantages.. like the fact that with this scenario group managers could basically use the drupal group to create an online interface for an existing list hosted anywhere. Cool.
All that said, if it's possible for Kyle to create this so it is all handled natively in drupal, without hard to manage external dependencies, then I'd prefer it. I don't have mailman at the moment and am not eager to set it up again on my server. :) I also think it's confusing for people to have to subscribe to two places and see potential nightmare maintenance issues.
Cheers,
Tobias
Ready to test the Alpha
I will be happy to run through your install instructions once you get there.
Thanks,
Aris
Just a query, how are the
Just a query, how are the list email addresses created?
say i create two group (programmers, designers), would respective emails like programmers@example.com and designers@example.com be created themselves?
group title
Right now, email address are created from the PURL prefix the group admin chooses. I hope to change that at some point so the email address can either be created automatically from the group name or be chosen by the group admin.
Kyle Mathews
Subscribing
This sounds like exactly what I am looking for. I really don't have the skills to contribute to this development but I will stay up on how things are progressing.
ds
Light at the end of the tunnel
I am in the middle of writing an application to Google Summer of Code to work on a project management tool for activists and organizers: http://wiki.auzigog.com/Drupal_Projects_for_Activists_and_Movements_-_GS...
I had the exact same goal in mind as kyle. I'm glad this project is already underway!
Kyle, I'd love to collaborate on this project, update it for Drupal 7, and make a beta release available if my GSoC project is accepted. I'd also be interested in finding a way to adapt it to postfix or find a way to tie it into a Drupal mailing system without having a delay in messages being sent.
For some background, here are some modules I found along the way:
http://drupal.org/project/mail2og
http://drupal.org/handbook/modules/og2list (old version of the same thing we have here, but based on postfix)
http://drupal.org/project/mailman_manager + http://drupal.org/project/mailman_groups
http://drupal.org/project/mailcomment
http://drupal.org/project/mailing_list (D7 branch, but doesn't seem to work for discussion)
would love the help
Hey Auzigog,
Sounds like a great GSOC project!
I'd definitely appreciate another set of eyes on the code (I'm sure I've done some really dumb things) and help adding different features.
I had another lookat og2list and it (was) a lot more cool than I'd thought. They had postfix integration so we'd be able to copy their ideas there.
They had a lot of discussion there about how to run og2list on shared hosts (sound familiar? :D) http://groups.drupal.org/node/1725 It doesn't look like they were ever able to crack that particular nut.
Kyle Mathews
Magic Groups
It sounds like you're pretty far along, but back in 2006 Zack Rosen put together a bunch of scripts for Drupal 4.7 that provided functionality pretty close to what you're describing. It might be useful (or it might not):
http://www.zacker.org/magic-groups-screencast
I've always wished someone created a strong mailing list solution, so I wish you the very best!
magic groups
I did run across what you guys did back in the day. Very impressive! Thanks for the encouragement! Hopefully OGM can get the point where it's pretty painless for someone to setup and start running.
Kyle Mathews
OG Mailinglist is ready for testing
I was also hoping to be able to say that the code is up on drupal.org... but I can't seem to login to Drupal's CVS system (@#$@# CVS) so we'll worry about that another day. We'll work from github still for the time being.
Anyways, last week I spent 2.5 days refactoring og_mailinglist so it'll work with virtually unlimited numbers of sites and servers on one server. This actually went pretty smoothly thanks to Drush 3.0's site alias functionality.
What's really needed now is better documentation. I've commited to github some initial documentation but I'm looking for a few brave (and knowledgeable -- there's only so much hand holding I can do) volunteers who'll test og_mailinglist on their own servers and work with me what's missing / needs clarification on the documentation.
And just a reminder of the server requirements. You'll need an Ubuntu / Debian box which you have root access. It still requires Space/PURL (haven't had a chance to refactor that away). The easiest way probably to meet all the requirements is to use drush_make to build a new copy of eduglu from my make file on github.
If you can help out, please contact me.
Kyle Mathews
alpha 1 tagged and released
Get it while it's hot at http://drupal.org/project/og_mailinglist
Kyle Mathews
Great to see the progress!
Hi Kyle!
Great to see the progress you're making. Sorry I haven't committed that Postfix work yet - too many balls in the air!
I promise to get there soon!
Nigel
And alpha 2 is out as well
In case you missed it, I rolled out the 2nd alpha this morning. See the release notes here:
http://drupal.org/node/798112
Kyle Mathews
Great to see!
I'm pulling from git, so got it anyway :)
Looking through the diff from April 19 now. Doesn't that seem a long time ago?
Code review over :) Thanks for all your great work. Now I'll give it a try!
Nigel
code review review
I created an issue for the temp file issues at http://drupal.org/node/799138
Files now end in new lines - http://github.com/KyleAMathews/og_mailinglist/commit/2fa3d28db814ac1dee1...
Not sure about what will happen with bounce messages. In any case, I've done nothing so far to support handling them.
On leaving groups on the subscription settings form. OG Mailinglist is being written primarily for Eduglu, a social learning product. I put that in because OG's handling of leaving groups is a mess frankly. You have to go from group to group to group to leave each one which is slow and takes an enormous number of clicks. So it makes sense to me to have that subscription page be the place where members can adjust all their group-related settings. But I can see it wouldn't always be appropriate. Perhaps that feature can be made optional.
"Threading and such" is kind of important. It'd be nice of course to make OGM work with notifications/messaging but I'm about 95% sure their architecture wouldn't support what OG Mailinglist is trying to do so integration would probably require rather extensive work on their part.
I added an explanation of what's going on with is_private/is_public. If you can think of a cleaner way to do that, that'd be appreciated. http://github.com/KyleAMathews/og_mailinglist/commit/690893ec69248e121eb...
Kyle Mathews
Thanks!
Thanks for your responses.
Re newlines at the end of files, I was meaning at the end of the source code files - sorry for my ambiguity!
I do like the idea of a form in which you can mass-manage your group subscriptions. It's not core functionality for og_mailinglists though - perhaps it could be split off to a separate project?
Re notifications/messaging, okay. I'll acquiesce for now and look at the issue some more once the basic functionality is up and running. You wrote about the reasons for not using notifications/messaging a bit earlier in this thread, right?
Re is_private, is_public, perhaps rename the variables to group_private and node_private?
PS: Thanks for pulling my patches in.
Is it working and worth changing my hosting?
Hi Kyle! I've been looking around a bit for a solution like this and like what I've read here. The problem is that my hosting company (GatorHost) won't let me add the MailParse extension to the plan I have. I'll have to basically triple my monthly payment (which honestly isn't alot at the moment) to go up to a Virtual Server plan where I can add the PHP extensions I need.
Before I do this, I want to make sure :
A) This solution works reliably
B) A. is really the only thing that matters.. :)
thanks again for the work you are doing for the community!
OGM alive and well
Yes OG_Mailinglist is alive and kicking and working quite reliably. On island.byu.edu it's handled some 27,000 emails in the past month w/o so much of a glitch. Development is moving forward at a slow but steady rate.
Question though -- will you have root access on the virtual server plan? To install OGM, you'll need that access.
Also, for future new visitors to this thread. We should probably let it die now as it's served it's purpose. If you have more questions, visit us over at the OG Mailinglist Issue queue at http://drupal.org/project/issues/og_mailinglist?status=All&categories=All
Kyle Mathews