Strategic development of groups and d.o

japerry's picture

As many have heard, the Association has acquired more resources to work on the groups migration. I'm hoping this discussion can talk about how we do this.

Facts:
GDO generates ~ 100k page views a month, drupal.org is 5.3 million
GDO has ~1200 total groups

There are approx 3 major types of users on GDO:
* Local User Groups
* Working Groups
* Projects

There has been no commitment to create a commons2/d6 to commons3/d7 migration module, and no work has been done on commons_migrate for over 6 months.

APIs would need to be written in order to link content on GDO to D.o

Of the 3 maintainers (https://groups.drupal.org/node/20724), only one (sreynen) seems to be active. (redacted what could be considered private data)

From these facts, a few options have been proposed:

1) Migrate to commons3 and push patches/changes upstream to commons:
Pros:
* Content would stay on GDO
* Offers many requested features out of the box.
* Being drupal7, it will be easier to integrate into other d.o infrastructure.
Cons:
* Limits ability to use features that might not be compatible with commons3, eg: Omega, Flag, etc.
* Content strategy may not be best suited for GDO
* API integration would be a significant task
* Migration would be a significant task

2) Migrate to a custom drupal7 site (potentially based on commons3)
Pros:
* All the pros of Option #1
* Allows GDO to use features that commons will not be able to support
Cons:
* Content strategy may not be best suited for GDO
* API integration would be a significant task
* Migration would be a significant task
* Wouldn't stay in sync with commons3, would require separate maintenance (similar to current GDO)

3) Migrate data to drupal.org
Pros:
* Closer integration with other data on d.o
* Reduces maintenance costs
* Brings localization to drupal.org
* Could be done iteratively (similar to jobs, we migrate events, user groups, etc as separate projects)
* Could help organize content strategy better
Cons:
* Makes drupal.org more complex

* Still needs migration (and would need to make a redirect mapping from GDO to D.o)

In general, my feeling is that the Association has the resources to ingest groups.drupal.org into drupal.org. The main reason for GDO existing (according to webchick) was because d.o iterated too slowly and GDO was a place for new features to be used. This made sense when there was no one (paid) responsible for GDO or d.o, but is not the case anymore.

It seems next steps include getting user feedback from Community Tools and others in the design committee, to find out what users really are doing on GDO. But we also need to get a general idea of what people are doing on drupal.org, and identify if GDO is being used as a bandaid for a larger ecosystem problem.

While its my sense that many will want to see GDO live on its own, I'd like to get more data behind why it needs to explicitly be split off D.o. Research from this week in Amsterdam seems to indicate that it might not need to be.

Comments

I think we should go with the option with fewer Pros

sreynen's picture

For what it's worth, I don't like the idea of merging Groups into Drupal.org, but I don't see how I can participate in this discussion in more detail due to how it was framed. You've packed a lot of assumptions and private Association knowledge into a question accompanied by a preferred answer. For example, you think the Association has the resources to run Groups, but most of us have no idea what those resources are, and you haven't asked the person doing most of the work currently (me) what resources might be needed.

While this is a fair

danigrrl's picture

While this is a fair critique, it might help us to understand the status of the work you've been doing to date. While I've been keeping an eye out for updates, as of Austin, work on the upgrade initiative was stalled at best, and it was pretty clear to those of us working on this that the original plan had shifted significantly.

Has there been movement on this issue since Austin from your end?

Movement

sreynen's picture

I was not in Austin, and was not aware at the time of Austin that migration was stalled. I was at the time still under the impression that the migration would be completed soon. I would still be under that impression today if I hadn't randomly bumped into japerry recently and asked him about his new job. From that conversation, I gathered that migration was basically starting from scratch, but no one ever actually told me that. No one from the previous vendor nor the Association has ever given me a clear idea of what was happening with migration. Apparently that information is being shared more openly at DrupalCons, so I suspect you know far more than I do about the status of migration.

I don't see functional improvements to Groups as directly tied to migration, and I continue to believe it's a mistake to tie the two together, as that slows both and both are already too slow. To that end, since finding out that migration was stalled, I've started actively improving the current D6 Groups site. That's all happening in public in the issue queue, but to summarize: I've reviewed every issue in the queue and updated the status where necessary to provide a good baseline for more significant changes. And the first of those significant changes, revamping the new group moderation process to be clearer to everyone involved, is nearly complete. I may finish that immediately after this. My next priority will probably be improving notification emails, as that seems to be the biggest room for improvement. I'm also regularly checking the about page to make sure it still lists me as site maintainer.

I was not in Austin, and was

danigrrl's picture

I was not in Austin, and was not aware at the time of Austin that migration was stalled. I was at the time still under the impression that the migration would be completed soon. I would still be under that impression today if I hadn't randomly bumped into japerry recently and asked him about his new job. From that conversation, I gathered that migration was basically starting from scratch, but no one ever actually told me that.

As I was involved in the discussions that uncovered the migration being stalled, I have to take a minor issue with this statement. But this also could be my memory, as I seem to recall you were in the Community Tools meetings that happened post-Austin where I made it clear that we had found out migration wasn't being prioritized. It's very possible that, in all the other things that I've been doing, that you weren't in those meetings, and thus didn't know about this.

I don't see functional improvements to Groups as directly tied to migration, and I continue to believe it's a mistake to tie the two together, as that slows both and both are already too slow.

On this, I think you're exactly right, and I wonder if you're documenting/summarizing those improvements somewhere outside of the Issue Queue? It would be good to have some concise documentation that specifies what factors we need to consider when the migration actually occurs.

Also, I just want to caution, as someone who's been privy to user research (and doing user research) in the community for well over a year: one of our biggest mistakes is assuming that "happening in the Issue Queue" and "happening in Public" are equal. A shockingly small percentage of our community is actually comfortable enough with the Issue Queue to even go in there, and frankly, we make it actively difficult to find. To that end, thank you for summarizing those changes here.

Two things to add to what Dani said

rootwork's picture
  1. If you haven't been kept in the loop about these discussions, that sucks. As an outsider (just a group manager who is interested in moving this stuff forward) the process and the status of the process has been pretty opaque to me, especially outside of 'cons. But I didn't imagine that to be true for folks like you who were playing critical roles. Would it make sense to start scheduling regular meetings/check-ins for the migration moving forward? That's just my suggestion from the outside looking in.

  2. I agree that functional improvements to Groups are not directly tied to the migration. That said, I think it makes sense to have the conversations with group managers now to find out what they need, as it's conceivable that some of that information would impact some of the migration. For instance, there could be some feature that would make the migration more complicated and from user research could be identified as something that could be dropped. Or there could be a feature set the community wants that helps indicate a direction for the migration (for instance, for or away from Commons, or for or away from d.o integration).

I totally get that we don't want to have to figure out the upgrade AND make things shiny and perfect and rainbows all at the same time. But I think it makes sense to start the conversation with folks now -- so that those conversations, and report-backs from them, are happening in parallel to the technical planning. If it turns out that it makes sense to make not one functional change to Groups in the course of the migration, and save all the feature requests/changes/fixes until after the upgrade, I have no issue with that. I just want us to have the (qualitative) data.

I also think that talking to group managers about this process, even if the context is "we're upgrading things first, don't expect anything new soon," will help make the process more apparent to everyone and give them a sense of investment in supporting it.

I'm certainly not married to

japerry's picture

I'm certainly not married to one answer or another (yet).

As a maintainer of Commons, I'd love to see GDO run on it. On the other hand, I also want to see content put in places that make sense for developers, user groups, and end users; which led to many discussions this week tending to favor putting content back into d.o over upgrading GDO.

The Association has 11 tech-team staff members as of Oct 2014, with 5 dedicated towards development and maintenance of *.drupal.org

I posted this discussion because its important to be transparent about why certain decisions are being made.

And yes, I'm looking for some good reasons why GDO should remain separate, if you can think of another way to frame it -- I'm all ears :-)

APIs would need to be written

greggles's picture

APIs would need to be written in order to link content on GDO to D.o

Why?

One of the issues facing

japerry's picture

One of the issues facing groups is how it interacts with the rest of drupal.org.

Looking at statistics, GDO is far below other sites in regards to usage. Previous user studies have indicated that finding content is hard partially due to content not being on the same site.

If we were to migrate GDO to D7, one of the things to evaluate is how content on GDO would match up to current and future content on D.o, IE: Events and Project pages.

Details

sreynen's picture

One of the issues facing groups is how it interacts with the rest of drupal.org

Please explain this in more detail. I can think of only two documented requests for information sharing between Groups and Drupal.org: user profile photos and issue tracking within Groups. The former requires tighter integration between the two. The latter does not, and I think would be better solved by a system that allows information from any site (i.e. RSS feeds) to be shared within Groups. I don't doubt there are other reasons to more tightly integrate the two, but it's not at all clear to me how you've arrived at the conclusion that these problems would be best solved by merging the sites, nor why they should be involved in the migration at all.

Looking at statistics...

Where can we see these statistics? Again, I don't doubt the information you're summarizing, but I have no way of knowing how you get from there to your conclusions, which makes it nearly impossible to question your conclusions. For example, much of the activity of Groups happens via email. Are you including email activity in this conclusion?

There are a few places where

japerry's picture

There are a few places where the Association is looking at better integration:
* Events
* Profiles
* Development Discussions linked to profiles

The statistics are derived from Google Analytics and db queries on nodes, etc. For example, all of the groups content only is 7% of what d.o has.

Please remember that my comments are not necessarily my personal comments but of others as well. We (infra staff and those at the sprints) have iterated between having many sub-sites to having a large drupal.org.

I can't stress enough that no conclusions have been made. Just simply putting out the facts and some options.

An overview of what we're proposing here.

danigrrl's picture

An important thing to highlight in this conversation is the idea that different groups serve different purposes. To Japerry's point, and per our discussions on the Community Tools Team (which you've been part of), there are three fundamental types of groups:

  1. "Working Groups," which get together to discuss issues related to a specific project, e.g. Media or Drupal UX;
  2. "Interest Groups," which get together based on a shared interest, like the Higher Ed or LGBTQ group;
  3. "Regional Groups," which get together based on shared location.

The main things we've been discussing, and this is a very recent discussion (as in, it started this week), are:

  1. How do we create a situation wherein people working on a project can actually reach out to the broader community, as they may be able to do within Groups, without having to "rehash it" once it gets into the Issue Queue? This has been a major problem, for example, with just about every UX process that's happened in Drupal for at least the past 4 years, leading many of us to go straight to the Issue Queue for design discussions, which basically shuts out a significant percentage of the community.
  2. How do we create a better overall experience for group managers and maintainers, particularly around things like creating membership directories, providing support and discussion for each other, and posting events? This is a conversation that's been happening since Austin, and it's something we need to look into further. This is the research we're kicking off now.

All of this is to say that nothing we're doing now takes you out of the process, or should impact the improvements you're making to GDO currently. And we want you involved in the process. But from my end personally, the DA has identified the upgrade of GDO as a priority for Q1 2015, and I want to make sure we understand the featureset that group managers need, and identify the problems we try to solve, before that work happens. This will make sure that we are focusing on a truly user-centered solution, rather than a "let's get this done," technically-focused, solution. Does that make sense?

What happens post-migration?

sreynen's picture

I would actually love to be taken out of the process. The role of managing Groups isn't particularly fun. But it's a role someone needs to do, and ideally someone who can put far more time into it than I can. My minimal activity should not be taken as a sign that Groups doesn't need ongoing work. I would love to see the Association hire a paid replacement for me, but that doesn't seem to be where we're heading.

I see discussion of substantial changes to Groups, starting without almost no input from anyone who has ever managed the site, and with apparently no plan in place for anyone to manage Groups in the future. I have been looking at migration as the beginning of more significant improvement to Groups, but this discussion makes it feel like the end of that improvement, if not the end of Groups entirely. That worries me primarily as a general member of the Drupal community and the Association. As a volunteer on Groups, I'm happy to have less responsibility.

I would also love to see Groups have a more user-centered approach, and a "let's get this done" approach to migration seems like the best way to get there. Simply moving to Drupal 7 with no other changes would make a huge improvement in our ability to attract contributors for future UX improvement. Including more changes in the migration actively excludes potential volunteers from the process, because the migration involves private data. And moving functionality to Drupal.org further excludes contributors because the process of getting contributions deployed there is much slower.

So adding more changes to the migration adds more reliance on Association staff time. To whatever extent we can actually replace volunteer work with paid work, I'm all for it. But after 2 years of relying on internal staff from the previous vendor to get this done and repeatedly hearing it would be done soon, and seeing no discussion of what will be different this time, I'm not ready to trust that the Association won't hit similar delays. And I don't want to see other improvements also delayed if that happens.

On your 2 areas of focus, #1 seems closely tied to the migration question here. I personally favor communication between separate sites as the way forward on that. A unified platform is an obvious alternative, but again, that reduces the potential contributor pool. #2 doesn't seem tied to migration at all. Those are areas we're improving now and can continue to improve completely independent of migration.

Scott, great input -- and I'm

japerry's picture

Scott, great input -- and I'm just starting to get my head wrapped around what groups means, what the content means, etc. I think its extremely important that you're involved with how we not only migrate content but make sure we can maintain groups in the future. Right now its not entirely sustainable from a maintenance -or- moderation standpoint. Its too early to figure out what it'd look like post migration.

Unfortunately the migration to commons3 isn't really cut-and-dry. If it were, and the vendor was behind it, I think it'd be a really strong case for us to move to (and we probably already would have). But as some of us discussed in Amsterdam, the migration module is -not- done and still has a bit of work to do, and commons has a bit to go before it can be in feature parity with GDO.

So I think the association (me in particular) has a bit of work ahead to get the content migrated to D7 regardless of platform. This has caused other people to ask content related questions, and if we're spending a lot of time writing a custom migration, why not make sure we want to put the content in places that make more sense?

The 'makes more sense' question is what I'm asking. I put out my first brainstorm, and Scott brought up some great counter arguments. Its my goal to come to some consensus that figures out what users need, and build a site that meets those needs. I'm personally on the fence regarding external services fed sites vs one monolithic site. My current favorite is focusing on regional and interest groups within a GDO style community.drupal.org, and moving project based discussions to d.o.

In the end I don't want to spend 3 months working on the GDO migration to X when users really needed Y.

Identifying features from group managers

rootwork's picture

I'm not sure I have an opinion yet on maintaining GDO or folding it into d.o -- need to hear more. But I just want to highlight that with Dani's help at the sprint I compiled all of the user feedback and issues from group managers to date:

https://www.drupal.org/node/2349785

The idea is to compile evidence in that meta issue for particular issues (interviews of group managers are continuing) and then create sub-issues for particular features as part of the upgrade/migration. I included a bunch of existing issues, but it's possible that in discussion on the meta issue or after user interviews we decide to close some of those feature requests.

I know identifying the features is a different conversation than the larger one here, I just wanted to highlight it because I think the two conversations can happen in parallel.

It seems that user groups and

japerry's picture

It seems that user groups and events really would be a good candidate to remain separate from drupal.org.

However, we might be seeing a naming/branding issue with groups, which is causing confusion. Per some discussions this morning, here is another alternative proposal:

1) Turn projects into groups, and move project/development based group discussion over to drupal.org.

2) Remove jobs from GDO (this is already in process with jobs.drupal.org)

3) Rebrand groups.drupal.org into community.drupal.org, and focus on events and user groups. Use commons or something similar to manage this.

Quick clarification on point 2

DYdave's picture

2) Remove jobs from GDO (this is already in process with jobs.drupal.org)

Just wanted to confirm that if jobs are removed from GDO, it will become impossible to post a job in the community, without having to pay (that is to say, posting on jobs.drupal.org), or am I missing something here?

Thanks in advance.

Free jobs plans

sreynen's picture

I'm clearly not well-informed on Association plans, but I think the plan is still to add a free job option on jobs.drupal.org before removing jobs from Groups.

Happy to hear that

DYdave's picture

Thanks @sreynen for this prompt reply and sorry for stepping in with this unrelated matter/concern.

Currently it doesn't seem to be possible to post free job ads on jobs.drupal.org, but I guess it's probably part of the transition and will hopefully be possible by the time jobs have been completely moved out of GDO.
I'll try to keep myself better informed by following discussions on this topic and read more of the posts from the DA.

Thanks!

In Amsterdam, there were

japerry's picture

In Amsterdam, there were discussions about how to make free job postings, but further research is needed before that is implemented. I imagine once that is done, jobs will be turned off of GDO (IE: won't be able to post new jobs), but old ones will still remain.

Long term, I'm not sure what will happen to the old data. The 'read-only' part of jobs will probably be on GDO for a while (3-6months?), in which after that, those jobs should be filled. I agree with Ezra that it'd be optimal to have some type of archival of old jobs. How we do that is not clear yet.

Who is doing this?

sreynen's picture

Who exactly is responsible for managing the jobs transition on Groups? I thought I was, but it sounds like that's no longer the case, as I'm unfamiliar with the plans mentioned here. I'm happy to learn someone else is taking this on, but I need to know who to assign jobs issues and forward jobs emails to.

Also, who is managing Groups?

sreynen's picture

Who is responsible for managing access on Groups? Again, I thought I was, but I noticed that you've granted yourself the admin role. So it looks like I don't need to worry about access anymore? And do I need to be managing Groups at all or is your admin role a sign that the Association is taking over Groups in general?

Sorry if this sidetracks the discussion further, but my role in managing Groups has a very direct impact on my perspective on the future of Groups. My previous comments are mostly irrelevant if I'm not actually responsible for the site anymore.

hmm, no changes in roles or

japerry's picture

hmm, no changes in roles or responsibilities have been requested or discussed yet.

The original point of this discussion was to jot down some notes (again, these points are not necissarily my point of view), to bring some brainstorming to what will happen to groups as we move the content to drupal7. I fear that the openness of these notes have caused more harm than good.

One of our next steps is to setup a followup call/video with all the stakeholders from AMS and this thread to discuss whats next for groups.

Community Tools Team discussion is this Friday

danigrrl's picture

Would it help to talk this Friday? We're having the Community Tools meeting, and you are welcome to join. Sreynen is already part of that group.

I fear that the openness of

greggles's picture

I fear that the openness of these notes have caused more harm than good.

The opposite. In-person calls and followups via private mails/video are exclusionary. These plans have always been done in public and should be done in public in an open-source, volunteer oriented community.

Agreed

rootwork's picture

The process has been pretty opaque for awhile now. Being more transparent about it seems like a step in the right direction.

is it strategic to share private data?

greggles's picture

Hi,

Why is the last login time/date for specific people in the body of this post? That is private information from inside the site's database. It is not on their profiles. The fact that you can access it doesn't mean that you should access it nor that you should share it to the world.

There was discussion about making that part of people's profile back in ~2010 and it was decided that it was private information that should not be shared. That information is not part of the flow of your post. It doesn't support a point you are trying to make. Is this how you strategically plan an initiative, by posting private information in a non-sequitur that discredits and shames current volunteers?

You have shown some bad judgment here. Since it relates to private information that is managed by the DA, I suggest that you should not query a drupal.org database until the DA has concluded a privacy policy. As a helper that enables following the privacy policy, you should also create a simple table that is published in the contribute to drupal.org guide that lists which field data can be made public under what circumstances:
* always ok to make public (e.g. username)
* never ok to make public (e.g. email address, IP address)
* ok to make public in aggregate and isolation from other profile elements (e.g. country of access based on IP address could be aggregated and displayed as long as it's not presented along side the username)

The data was not intended to

japerry's picture

The data was not intended to shame, attack or put anyone on the spot. It was to simply point out that activity by the maintainers has been low, and we need a better strategy to keep maintainers engaged or cycled through to meet the needs of groups (whatever form they take). I figured if I had said 'only sreyen was active' that i'd get a response asking for proof instead of accusations. Its relevant because a strategy for content needs to include a process for those moderating, and the current process doesn't appear to be working.

I was not aware of the 2010 discussion, and thus removed that data from the original post.

Greggles, thanks for sharing

danigrrl's picture

Greggles, thanks for sharing this information about security procedure. However, I have to ask: how does this response do anything other than derail the actual discussion that we're trying to have? While I respect the need for privacy (and I agree that the sharing of this information may have been a mistake), it seems to me that it would have been very easy to mention this to Japerry privately, allow him to edit the post, and keep comments in this post on actual discussion related to the future of GDO.

Avoid de-valuing the work of others

ezra-g's picture

It's disappointing to me to see that private data about my Drupal.org user account was shared publicly without my permission. More disappointing to me is that sharing a timestamp and claiming that it represents the last time I was active in working on GDO is misleading and de-values time and effort I've put towards work on GDO since my last login, including among other work flying to DrupalCon Amsterdam to attend a working session on GDO and making myself available to help with the migration effort going forward as someone with advanced knowledge of the target platform.