Hosting company came forth and offered to sponsor NYC DUG hosting needs

in the interest of supporting other community centric projects, and thinking long-term about access to data and control over infrastructure (which tend to become very heated and divisive debates), can we first look at what options already are available to us?

if you have video data that needs a home, or audio recordings of drupal meetups, post them to archive.org.

we can be certain that archive.org will continue to exist in the long term; there are already people working on code that allows drupal to make use of archive.org for media file storage.

Let's put thoughts about the larger Free Software ecosystem and community centric collaborations that Drupal exists within first. From that perspective, a for-profit hosting company in delaware would not be my first choice.

I have to say that I am bit uneasy about setting up an NYC specific piece of Drupal infrastructure that isn't controlled in the same way that the rest of D.O or G.D.O (or any foo.D.O). That means a specific connection between the existing Drupal infrastructure community and this addition to it.

Given how many gigs of original content that we have, I can see a need for putting it somewhere, but there are a ton of issues that need to be addressed. The details of what this server will be doing seem really sketchy to me, and I would really like to have some questions answered before anyone moves on this.

• Who owns the hardware? Who manages it?
• Who gets root on the server?
• What will this server actually be hosting? Drupal site(s)? Raw Video files? Edited video files? etc.
• How do we decide what new sites get added to the server?
• If there is a main, or singular Drupal site, who gets a full admin account, and what is the method by which all of that is determined?
• If at some point in the future the NYC DUG doesn't like this service, or those people in control of it, and the majority of the group decides to move away, what happens to the data? Who owns it? How does it get moved?
• What domain names are we pointing at this server? Who owns those? Who decides what they are? Who controls the DNS?

I don't see the need for our DUG to have its own Drupal site for anything. In fact, I am thoroughly against it. G.D.O is an ever expanding, ever improving project of the Drupal community for the global Drupal community. Additional sites only water down that effort and make things harder to find. One of Drupal's great strengths is the way in which the community has managed to keep the vast majority of information available about Drupal on a drupal.org domain. It makes things easier to find, and ensures that this information will be available for as long as the Drupal project itself exists.

1) Provide up to 6 months back worth of meetup Raw video footage (or more if need be)- Why would anyone need such a thing? well to be frank, i do not believe i have the time nor desire to edit a video file of some one else's presentation. while i enjoyed listening to it, i think i will only work on my own to present it in best light. because of my time and desire limitations, i offer everyone the raw files (along with how-to tutorials) to be able to edit the video files with their own (or some one else's) presentations/content. the power is in your hands.

That makes sense. Having raw editable video available to all is a good thing.

2) Video streaming right from the site - yes, hostmysite wants to advertise their own solution as alternative to youtube or something, but that doesnt hurt us any, they get what they want and we get what we need.

I'm confused. They wan to offer us hosting or their branded video streaming service? I am really against the latter. People who donate infrastructure should get the cred that they deserve. However providing a branded service is not the same as providing disk space and bandwidth, or bagels at the camp.

I would like to see any video solution use an option that comes from the Drupal community itself, like Media Mover, or Emfield, so that this content can be incorporated as a part of G.D.O.

3) user ssh and FTP accounts - due to the number of power users in our group (know what to do with servers etc etc) i would like everyone to have the power (if they so choose) to help out with the group, therefore giving them sandboxed access (so nothing can be broken) to the server will greatly increase the number of people that can help out with the group management.

This comes back to the meta question of what this server actually be hosting. This may or may not be moot.

If there is a single Drupal install I don't see how you can "sandbox" it off from everyone else. The only way to do that is to give everyone their own site, at which point I wonder why bother with the server. The whole point of a sandbox, as far as I understand it, is so that people can break it, without having to worry that they are breaking something people depend on. The entire benefit of sharing and collaboration will be broken if we jail people off from each other.

4) our own mail server - as we grow (which we area already doing) the number of discussions also grows on our site. as it was pointed out already, this creates a lot of discussions that not everyone wants to take part of (or even know about). therefore having control over mail server gives us the power to customize mailing lists and mailing filters to sort which user gets which email. very granular control that everyone can enjoy. members can still get all the emails if they so choose.

Are you suggesting that we stop using the groups.drupal.org? The current way that mailing lists are handled are as part of that site, so the only way that having our own mail server will change any of this, is if we abandon that site. Again, this dillutes and not strengthens the global Drupal community.

Furthermore, who will own and be responsible for those email addresses, and the archives moving forward. If its not the Drupal Association (as it s with G.D.0), then who is it?

Any limitations in list filtering capabilities exist because no one has scratched that itch yet, and not because the G.D.O's mail server isn't configured properly. I support an NYC DUG project to build those features, but we don't need our own mailing lists.

Q: How will we use this hosting so it doesnt damage our existing group site?
A: we will upload all the content to our host, and link to it from the g.d.o/new-york-city discussion pages. I will speak to moshe weitzman to allow integration with more rich content so we can link to more content from the g.d.o site which is hosted on our server.

You have a very limited sense of what damaging our existing site means. Its not about traffic or page loads. Damage will happen when we decide that we no longer want to use the infrastructure that the Drupal community uses, and instead use our resources and effort to build something that competes. I don't use compete to mean that we have anything against G.D.O, but rather that every person hour we use on something outside of that, is an hour not spent improving the Drupal user group experience for every member of every DUG internationally.

Also there is no long term assessment in your answer. Sure we could make the g.d.o/new-york-city a JSON call shell to our own site. But who owns that data when there is some falling out between those who control the server, and those who make up the DUG. It faces the risk of permanently disappearing from the internet, and I don't want to lose all of our group's effort. If you don't believe that this may happen, you aren't seriously looking at the history of free software which is littered with forks and dead projects specifically because of falling outs. We should never think we are immune from that, and fight to keep everything owned by everyone for the good of all of us.

1) 180GB+ space : for the videos and other content we produce

This seems low for raw video content. I'm no expert on the size of video stuff, but we should get at least 300GB.

2) 512MB ram : standard amount ? we probably will never reach that level but its better be safe than sorry

That also seems a bit low. Most servers ship with at least 2G of RAM these days. I wouldn't even try with less than 1G.

3) Mailing server: we dont blast too many emails so something basic would suffice.

I honestly don't think that this server should be configured to do anything more than send system mail.

4) Fat pipes (throughput and bandwidth limits): we will need large data transfer limit because of the videos that we will be hosting

Yup.

5) LAMP stack : linux, apache2, mysql5, php5.2+ : standard stuff for this kind of server

Which distribution of Linux? They are not all created equal.

6) configuration for sandboxed ssh and management accounts: must be sandboxed so people cant break other people's stuff.

I'm still totally unclear as to what the goal is with this stuff. But its basic linux sys admin stuff.

7) .... ?

An ethernet card that can support a gig of throughput. You don't want three people downloading video who are fighting over 100MB of throughput.

8) $$$$ ?

What? I thought this was being donated. Are you asking the DUG that has little to no money to cough up server fees for a server that is of dubious benefit to the entire Drupal community? Aren't we better off donating to the Drupal Association, and working with them on building a video hosting server?

Maybe I'm misunderstanding you, but paying for (a possibly branded) service doesn't sound like a good deal for us.

Overall, I'm unclear on the need for any of this. I am very concerned about the ownership issues. I am absolutely against any solution which turns the group's attention away from using groups.drupal.org/new-york-city. Our use of that site is what connects us to the Drupal community in a very big way. Though New Yorkers like to think that we are the center of the universe, we are not. We don't need to separate ourselves onto a virtual island away from other Drupalers. All pieces of our infrastructure should be explicitly owned by the community at large, and never by individuals within that community, particularly if there is money involved. All of that content needs to be owned by the community as well.

I will post usage ideas as they crawl into my mind. forgive for the many comments to come.

another good use case is offering free hosting for meetups and camps so that new members get their own subdomain hosted on our server where they can install drupal with out help and then play around with it. Alot of people heard about drupal and want to use it, but just dont know how to take the first steps. This is where we offer helping hand with the first few steps.

Wow gnat, that was definitely TMTR, but I did catch one bit that I wanted to respond to. The part about not being able to "sandbox" with a single code base. Isn't that what multisite is really good at doing- keeping multiple sites with a single code base? We can separate out by user creating an appropriate subdomain such as gnat.example.com or ixlr8.example.com. It's a little tedious to set up by hand, but there's always the option of using hostmaster 2. I'd be happy to get that running so we can provision sites out to anyone who wants a sandbox.

Same cautions as have been stated before. This is a huge undertaking without the committed underpinnings to maintain it. Furthermore this is not, so it appears, managed hosting.

Who is doing the OS patches, the security upgrades, who fixes it when apache crashes, does the logrotating, sets up the ssh access.

That's a whole bag of hornets you do not want to open up on a volunteer basis. I do this stuff regularly and I certainly wouldn't volunteer for it. And I would (granted cynically) question the dedication of a volunteer sysadmin.

The idea of building a complex sandboxing mechanism on top of it give me the willies. And I haven't even gotten into why I don't like it for community reasons.

Sorry, I don't think it is necessary, I don't think it is a good idea, and I think it exposes the group to a variety of unconsidered consequnces by entering into such an arrangement. I would advise against it.

However, thank you to the gentleman offering. Would love to have him at Drupal Camp and could certainly give credit should he want to donate money.

Finally, if you want to host video in this manner, you want a donated CDN, not a server. No one serves video in quantity off of dedicated servers anymore.

Advomatic

Here's what I'm reading out of this thread. Hopefully I can do this with some more brevity than in my last post.

I'm seeing a couple of different technical desires coming up:

1. There is a desire to make raw video content available to those who are motivated to actually edit it.
2. There is a desire to provide (somewhat) polished content that gets created through our events, such as videos of camp sessions.
3. There is a desire to have sandbox environment, as a place where people in the NYC DUG can come to work collaboratively on teaching and development projects.

I think that the emerging consensus is number 2 is best solved by putting video content somewhere where we are assured continued access, like blip.tv or archive.org, and possibly using a group account. Personally I think a group account is overkill, and we can either link to the video, or use something like emfiled to embed the videos into pages on g.d.o. Generally, I think this is the best solution, as it satisfies the desire to provide this content, as well as ensures that its available to us for years to come, without contracts or ownership.

Say we do that for 2, I really don't think that we need a CDN solely for satisfying 1.There won't be a huge rush on downloading raw unedited video. Putting it on a server, and possibly setting up a torrent so the community could help share the burden if individuals felt so inclined would be sufficient. There's less of an ownership issue here as well, since someone owns the video now, and if it fell off the internet it wouldn't be a tragedy like the loss of the Dojo content, or even edited video.

That leaves the issue of a sandbox. I think that a sandbox is a good idea, and does not even need to be part of the NYC DUG per se. Eric is right that anyone can step up and offer this to DUG for as long as they are able. There needs to be specific declarations that such a site is not the official group site, but a sandbox. This clears up a lot of my original issues, as this removes the sandbox from "group infrastructure" and instead puts it in the realm of "hey folks come use my server".

I would be remiss in not mentioning that Sam is right on with his caution about sys admin stuff. That needs an answer as its a huge logistical question. As do my original root / access / and DNS pointing questions.