When it works, media_mover is an amazing piece of code. But when something goes wrong, media_mover is the Achilles’ heel of Open Media workflows. A few of the stations we worked with during the beta didn't need media_mover, but because we were trying to build on a common platform they got it anyway. Both channelAustin and Portland Community Media had already invested in encoders far superior to what a media_mover/ffmepg_wrapper configuration could do. Those stations were getting all of the disadvantages of media_mover while leveraging very few features. Some of what was built into the Open Media workflow came from the fact that Princetons can't mount external volumes. Using a Princeton was like having an 40GB iPod with 200GB of music. Only a small part of the library could be on the iPod available for playback at any given time. Because of that, stations using Princetons are pushing files to their playback servers and purging them on a regular basis. With the 3.7 release of the Telvue API, with a little code stations can reverse this workflow so that the Princeton asks for MPEGs for the files it knows it will need in the next 2 or 3 days. With the missing files report, the need to have Drupal moving media because much less important.
Current Workflow
Revised Workflow for channelAustin
It would be great to get some standalone code written so that stations could use FFMPEG as a dropbox, but stations that can afford a commercial encoder that comes with a 1-800 number should use them. Because this workflow doesn't require mounts where the webserver can actually fileexist() the MEPG2 or H.264, the Drupal site could be hosted externally. After visiting 8 or 9 PEG stations over the last 2 years and seeing first hand how their networks are configured and managed, moving Drupal off site has to be the preferred configuration. There are definitely downsides to this approach, but the upside is that this is a configuration existing station staff can support.
Comments
I don't use filefield_local.
I don't use filefield_local. Instead of having the users select the file when the om_show metadata is entered I have a media mover config which ingests the files from the dropbox. So, we create the om_show first so we get an unique id to tag the mpeg file so media mover knows which node to attach the file to. This reverses step one and two in the first workflow:
2a. Media Mover harvests original to show node.
Facilities with Content Agent could have the following workflow:
2a. Media Mover harvests original to show node.
Only difference being Content Agent creates the flv and jpg instead of media_mover/ffmpeg.
Also, in one case the auto scheduler won't be doing any syncing of metadata as it will check and see that Content Agent already took care of that. That's the same as here as we sync everything on node_save and node_update.
In situations where you're
In situations where you're going to use an external encoder that media_mover can't control to create the MPEG2/H2.64, flv/H.264, and jpg, what is the advantage of using a filefield to store the path? The reality is that once you have a Synergy or Cablecast ID, you no longer even need the path to the broadcast quality MPEG/H.264. If the file for the VOD is being pushed to Blip/Vimeo/Archive/S3, why build a dependency on local files?
The access Filefields require through mounts and/or symlinks requires permissions be set correctly on the device providing the share, the server mounting it, and the encoder.
My experience has been that is more than the average PEG station can support. If I believed that if media_mover was just set up right once, it would run without an issue I might still recommend it. But with only a dozen people on the planet with any experience cleaning up the files and media_mover cache tables after a problem, I can't configure that for stations who'd have no hope of being able to fix it when something goes wrong.
Hindsight is 20/20 but if all the effort that has gone into Media Mover was invested in code for an open source Content Agent/ProCoder that could communicate with Drupal, where would things be now?
I have no experience with
I have no experience with Content Agent so I'm not sure what it does. Can it push content to Blip/Vimeo/Archive/S3? If so how does the embed code get back into Drupal? That all seems like a job for media mover. I also remember you were really insistent that drupal would be the master for metadata and location of the mpeg files. In order to support facilities with more then one type of playback server or for the purpose of migration.
I think in the long run most of the issues will be solved by migrating from fildfield to mediafield. Then with stream wrappers it doesn't matter where the files are stored.
According to Root6, Content
According to Root6, Content Agent can do anything. Out of the box it is just a fast encoder with a nice UI and a 1-800 number to call when you have problems. Whether it's Content Agent, Telvue Archive Server, or another box watching for new files on the archive volume after those servers encode it that pushes the files into CDNs, getting those files reconnected with their nodes is trivial.
The big advantage to moving away from filefields to storing only paths where possible is that the Drupal site no longer has to be on the same network as the playback servers. This opens up the possibility of implementing at least some of the Open Media modules to far more stations.
Drupal would will still have access to all the metadata available. We're already tracking the files across multiple Princetons. I don't know of a use case where there are multiple Cablecasts and/or Synergies (sp?). Those servers seem designed to work from a master show record regardless of the number of playouts. Once Telvue releases 3.7.1, there isn't an issue with running a Cablecast and Princeton as long as the Cablecast can pull directly from the archive volume and there is something listening for the files the Princeton wants, transferring those, and purging the older files.
The current configuration for Princeton users is an all or none game. If all the parts are working, files make their way from producer's DVD to being aired without issue. When any of those moving parts stop working, everything stops. If Drupal can't move a file to a Princeton, the show can't air. Now that Princeton can ask for files, it seems silly to put that responsibility of responding to file requests into a Drupal configuration were there is so much else going on and any error could prevent the transfer.
Deciding that the missing_files listener should be stand alone and not another Drupal module really changed how I looked at the entire workflow. That combined with the problems we ran into using SMB mounts through Plesk at BAVC, the work I've been doing with Kultura and Blip, and in generally asking myself what could we have done better and what didn't work brought me to this conclusion.
The fact is I can write a solid, secure "remote file lister" and the support for that into om_show in a few hours. That would be a single php file that anyone who can install XAMP/MAMP could get up and running at their station. As long as Drupal could get to the XAMP/MAMP box, everything else the Open Media modules do become possible. If Drupal can't communicated with XAMP/MAMP, new files can't be added but existing shows will continue to air. Even with stations connected via DSL or Cable modem, this seems workable.
Media_mover might still make sense in cases where there are Drupal developers/syadmins on site to support it, but I found those cases to be the exception within PEG. For locations like Austin or Portland that have already invested in commercial encoders and have limited technical support on site, this seems like a far better configuration if for no other reason they can keep it running themselves.
So, if I'm reading this
So, if I'm reading this discussion correctly, organizations being served by a vendor-provided capture/content/transcode solutions that meet ACM and online file standards can break the need for local OMP run transcoding operations. Which means that OMP could run either on-site in a "light" mode or potentially hosted off site? Could a commercially provided VPS handle this?
I really like the idea of being able to run OMP in a hosted environment and managing the local broadcast files into the system remotely, allowing it to run in both a centralized and decentralized structures. I think it really lets the system scale across most PEG organizations at that point. Enterprise deployments have the opportunity to build, fork and develop independently while organizations that need to focus on other priorities can benefit from the project and contribute in other ways.
I'm a little embarrassed that TelVue's workflow was holding up the ability of OMP to do this. When I first heard of the missing_file API though I got way too excited. I'm not part of the development team at TelVue but I've been assuming that this functionality was the first step on the way to expanding systems integration and workflow flexibility across our product line - no reason that expansion shouldn't apply to using our products in other systems though.
Oh - and while I have my TelVue hat on just a quick response to OMP controlling external vendor provided encoders. We sell encoders that are essentially the same as our broadcast servers. We don't have info on our website about them but MetroEast has five of them. I think Leightronix, Tightrope maybe even Synergy sell similar products. If I understand the multi-machine control methods decently I think that OMP could drive on-site, vendor-provided encoders with advanced functionality without additional code. Could also control vendor-provided encoders with basic functionality (record, play rewind) via VDCP.
Homebrew vs. vendor encoding is a tough nut to crack though... If mediamover/ffMPEG doesn't meet the need for PEGs is there another solid recommendation? Maybe compressor hotfolder mapping of a SMB mounted drive etc?
The Austin workflow also got me excited about OMP handling of live programming. Leightronix was talking about their timeshifting feature at a conference I was at in Wisconsin and there are ways to programmaitcally timeshift content in TelVue's systems as well. From darrick's workflows above (OMP Metadata before file) at least for administrators, then, in theory, live content files could be handled by OMP.... hmmm....
Oh - and I love your diagrams!
This is a great alternative,
This is a great alternative, I know we talked about it previously but never tackled it due to the Princeton limitations amongst other challenges. It will be great to have it as an option. Is anyone funding the development yet?
Open Media Foundation