During the DOM visit at Davis Media Access we discussed importing all of our show metadata into Drupal. We have a library of over 3000 programs produced at our facility over the last 20 years. We want to start the process of digitizing this material and making it available on our website. We also want to allow users to be able to search the programs we have available and order dubs as well as request they be played on our channel (digitized or not.)
Currently the Om Show implementation does not support this. The om_show content type out of the box requires a digital file attached to it. I propose making this optional and adding the option for om_show to attach itself to offline media content type.
DMA has a Cablecast SX2 server. The essential show information in this system is: ShowID (assigned by cablecast), LocalID (legacy ids from our old database, needed as we are not going to assign new ids to all 3000 of our programs), Media Format (DVD, VHS, U-Matic, Mpeg2), Show length and Cue Length (When the show starts on the program.) There is also a many to many relationship between the show data and the media data (for rare cases where shows are on more then one tape.) But, thankfully we don't use that (we just have part 1 and part 2 as two separate shows.)
My first run at implementing this I just started adding the above as CCK fields to the om_show content type. But, in the interest of wanting to make this usable for other centers It seemed best to just add a CCK field called external showid to the om_show which links an optional om_cablecast_show type (or optional om_princton_show type, etc. for other server implementations.) I have enough info already to sync the om_cablecast_show node data with the cablecast database (i.e. if I create/update/delete an om_cablecast_show in drupal it will create/update/delete the matching show record in the Cablecast system.)
I'm thinking it's easiest to have Drupal be the master and the cablecast system be the slave. I can do the initial migration of cablecast show info into Drupal. Then for adding new shows not tied to a digital file the user first creates the cablecast_show node and then creates a om_show node attached to that. Our programing directory can then schedule the show in Cablecast and set it to automatically record (leaving open the option for the DOM auto scheduler to do that.) cron.php then runs looks through the dropbox (of which cablecast will encode shows) and updates the field_om_show_original CCK for the drupal show record. From that point Media Mover happily does it job and the show is now online and digitized.
OTOH if the show is expected to be digitized before playback. The user just creates a new om_show node. A generic boilerplate om_cablecast_show type is attached to it (as this is what creates the record in cablecast and returns the proper external id.) The user can then use this external id to upload the file to the dropbox via ftp or over the local network from the NLE stations (if they didn't upload the file via the web interface.) Media Mover will then do it's magic.
My circuitous thoughts which cause me to get nothing done:
What fields are required in om_show for Drupal to do it's job (display the mpeg2 download links, display the flv media player, search the shows, vote on the shows, create the schedule for playback?)
What fields are playback server specific for syncing show information?
What triggers media mover? When the field_om_show_original field is changed?
What if an access center changes playback servers. How do you migrate the playback specific data.
My idea is om_show is abstract. Centers install an optional module specific to their playback server. So when you create a new om_show you are presented a form with the fields specific to your sites implementation. When the auto-scheduler is run the the proper hooks for your playback server are called.
So for our case when we create a new show node I have the optional media_type (taxonomy), length (currently a calculated field from the mpeg file) and cue length field. When created the show is also created in the cablecast system. For DOM they would have the same fields but the show is not created on their server because their server can only hold a limited amount of show info.
When the auto-scheduler is run the cablecast_insert_schedule is run to put the schedule in the Cablecast system. Dom's princton_insert_schedule hook would have to sync all the mpeg files and show info (delete everything out of there and put everything newly scheduled in.) The cablecast schedule hook wouldn't have to this as the shows are already synced.
Anyway, any feedback would be great on this topic. I feel it may be a bit disjointed and rambling but I wanted to throw out where I was currently at with it all.
Comments
Modularity and Hopefully a simple solution
I know that PCM was thinking about how they were going to tackle this (disclaimer: I do not work for nor represent Portland Community Media) and I had the idea in my mind to create another content type called something like "om_show_offline" that had the same fields as om_show and additional fields you would need to track (ex: tape number, leader, tape format. etc...).
Om_autoschedule would schedule this content type the same as om_show. Once cablecast_insert was run you'd have a list of errors representing offline media you need to capture for the day. Import your old tape library data into this new content type. You capture the show before it airs, add a new om_show for the tape you just captured and submit the same way, schedule it into the offline slot and delete the om_show_offline node that represented it.
Kills two birds with one stone (scheduling and patterned by "importance" transcoding of archives) but it'd be a job that would take a while.
This content type would be relatively easy to share or setup a tutorial for and give a modular solution to other stations that will run into this problem.
OTOH I don't really know the scheduling side of OMP so I could be totally off base.
The system we discussed in
The system we discussed in Davis and started implementing Portland was to use a content type called Pre Show. At PCM, Pre Show is populated with by FeedAPI with data that comes from Daniel's VB app used as part of their DVD import process. The idea with Pre Show was that it is the same metadata as show, but without the digital files. Then we use Views, a bit of theming, and Prepopualte module to enable staff to quickly move shows from Pre Show to Show without having to reenter all the metadata.
What fields are required in om_show for Drupal to do it's job (display the mpeg2 download links, display the flv media player, search the shows, vote on the shows, create the schedule for playback?)
The fields used by om_show are the fields that should have been locked after your implementation. Here's the list of fields we locked at PCM...
The Media Mover managed fields aren't locked. Of those fields, the MPEG2 is the only field that is really required. TVFrame and Views we've added during the implementations assume that fields like Flash and Screen Capture will be populated, but those aren't really required.
What fields are playback server specific for syncing show information?
We've tried to build the core as an playback agnostic solution. Nothing in the core content type is Princeton specific, but everything required to schedule a show on a Princeton is there. The only field that I know of that would need to be added is the ID from Tightrope insert you've already identified. I'm still waiting on API information from Tightrope and Leightronix, so it's hard to say if that would be required. Rich Vasquaez has been working on the Synergy solution. I don't believe it would be necessary to store the ID there, but I'm not sure.
Princeton uses the MPEG filename to link video files with their metadata records.
What triggers media mover? When the field_om_show_original field is changed?
Media Mover is triggered if a file exists in a Filefield field when a cron is run.
The next step in Media Mover's config works the same way. You select the field that will be populated with the source (in the case of Flash and Screen Capture they look to the MPEG field), the type of encoding you want done, and where Media Mover should put the new file.
What if an access center changes playback servers. How do you migrate the playback specific data.
Again, I'd like to make all of the core metadata playback agnostic. I'd also like to not only allow migration between servers, but the ability to schedule onto multiple servers during a migration.
I'm still not tracking why a
I'm still not tracking why a pre_show content_type is needed and why om_show has to have a digital file attached to it. Our playback system can play any format (DVD, VHS or mpeg).
i.e.
Om_autoschedule would schedule this content type the same as om_show.
It should be the same content_type.
Once cablecast_insert was run you'd have a list of errors representing offline media you need to capture for the day.
Tell cablecast to capture it from the tape.
Import your old tape library data into this new content type.
Already done as it's just an om_show with no mpeg file attached.
You capture the show before it airs, add a new om_show for the tape you just captured and submit the same way, schedule it into the offline slot and delete the om_show_offline node that represented it.
I'll make the computer do this. After cablecast airs the tape and captures it to an mpeg file.
At PCM, Pre Show is populated with by FeedAPI with data that comes from Daniel's VB app used as part of their DVD import process.
My plan is to just do a mass import of our cablecast show into the drupal om_show content_type once. Then instead of entering the info into cablecast I'll enter it in Drupal. So drupal will be the master for show data and cablecast will be the slave.
As for Media Mover I've been having fun experimenting with it. A couple of pieces of the DOM installation I'm not sure about. One is the ffmpeg_remote_wrapper. Instead of this I'm testing the mm_custom_command module to run "ssh encoding_server ffmpeg -i [harvest_file] -acodec ac3 -ar 48000 -ab 448k -vcodec mpeg2video -f dvd -copyts -s 720x480 -g 18 -b 8000000 -maxrate 9000000 -minrate 0 -bufsize 835008 -packetsize 2048 -muxrate 10080000 [out_file]"
And instead of filefield_local I'm thinking of generating a md5 hash for users to use in the name of the upoaded file. Then media mover can use this key to harvest the file correctly. I'm inpatient so I don't want to wait around for a file to upload before I can enter the show data.
And instead of
And instead of filefield_local I'm thinking of generating a md5 hash for users to use in the name of the upoaded file. Then media mover can use this key to harvest the file correctly. I'm inpatient so I don't want to wait around for a file to upload before I can enter the show data.
I talked about this with John (Humboldt) and Jess (Televue). There was some concern about how long it would take to generate MD5 hashes from such large files. What Jesse proposed (off the top of his head) is that we take several seconds/minutes at the beginning, middle, and end of a file. If the big file sharing initiatives within PEG agree on that pattern for taking those samples, we'd have a standard for digitally finger printing files being exchanged so we don't end up with unnecessarily storing duplicate files.
I'm still not tracking why a pre_show content_type is needed and why om_show has to have a digital file attached to it.
It's all about assumptions... which I know is making asses out of someone and I'm ok with that. I assume that every om_show has at least an Archival MPEG when scheduling and I don't want to have to dig into the files table to check. If the isn't an MPEG doesn't, that's a problem that I assume will be addressed by re-encoding and that the file will eventually get to the archive and playback servers.
I don't know how many stations have this Show Status View, but this is one of the areas John and Rad have been working to improve the UI/UX of the OMS. Obviously lots of red is bad, but we've been dealing with encoding issues in Denver for years so we have a pretty good handle on what to do. Televue is also adding an "MPEG Fix" function with their 3.6 release that should make our lives much easier.
I know there are a dozen ways to approach this where the interfaces and logic around om_show could either have a video or a tape, but that takes time and resources we just don't have. By separating om_show and om_preshow we're providing a framework to support tape tracking without interfering with the "pure digital" approach that is core to Deproduction's vision.
The bottom line is there is no technical reason shows without video can't be in om_show, but to deal with the politics and personalities at play shows that are never going to have files need to be managed in a separate content type.
ahhhh....
That makes total sense. I was thinking about pre shows being used only for series scheduling and import process workarounds but you could totally use it to store meta data and scheduling info for offline assets!
Just to verify - CMC staff can see the differentiation between content type show and content type preshow to import tape archives to fill out the programming schedule assets before airtime right? Or is there a view that still needs to be written to see the scheduled preshow programs that are coming up for staff to followup on?
Thanks for the good response on this thread. I'm way out of my league when if comes to the scheduling side of OMP.
-pete
The staff could alter data
The staff could alter data from om_preshow when the om_show is submitted. Once a om_preshow is converted to a om_show, the om_preshow would be unpublished and the MPEG that's on the Princeton Archive would disappear from File Field local's select list. The version of File Field local that PCM is running only shows MPEGs that aren't already in Drupal.
Currenlty Pre Shows cannot be scheduled. As far as the Knight grant resources, we have only ever been concerned with scheduling digital files. What Darrick brought up while we in Davis turned that discussion upside down when he was looking at Pre Show as a step in the process of digitizing an analog library.
If Davis created their analog library as Pre Shows, they would be able to convert those to Shows as the shows aired and the MPRG was captured. Any developers can also add whatever custom fields he wants for tracking tape info to om_preshow. It is possible that a checkbox could be added to Timeslot Scheduler to include Preshows in the scheduling and the queries and functions we've written for that module could be modified to create the type of tape-op schedule other system create, but I was to be crystal clear here...
I will do what I can to support another developer who's working on that, but we will not be using Knight funds or Deproduction/Civic Pixel development resources to build out that functionality. If DMA, PCM, and MNN got together and agreed on a spec, pool their funds, and hired a developer to add this functionality, we wouldn't do anything to prevent this. We'd be glad to commit any patches submitted to enable tape based workflow that didn't cause problems for the existing digital workflow.
The perspective in Denver is that dealing with tapes and 30/60 minute scheduling grids are only becoming less important, but after visiting several stations last year I understand why most stations are looking for ways to make that transition smoother.
Larger Station Workflow (modularity again)
Honestly this seems like it might be an essential element for stations that are transitioning from tape and also have a staff of more than 10 FTE. I really like derrick's workaround but I think the workflow I suggested above might work better at the larger centers I have worked at in the past due to a number of reasons I don't want to publicly post.
Of course, this is coming from a lurker and not a partner so ya'll should take it with a grain of salt. I also totally understand and hear loud and clear that this project will not and has not supported tape-based playback functionality. It can just see how difficult it could be to manage a tape to file transition in a center.
What I'd imagine would be the two most difficult tasks in that transition would be 1) the prioritizing and queing of tapes to encode and 2) the ability to encode multiple tapes across staff and departments in the center. It seems that the OMP has a pretty simple solution sitting there where a center could essentially cutoff their old scheduling practices and move to OMP in one swift move.
I'd be happy to do what I can to help make preshows function in the way discussed above but it'd be cool to get a request from a partner station to create this functionality. Another benefit to having the ability to schedule preshows would be to OMP schedule live events unless there's another solution to this I haven't read about.
What I've Learned Thus Far
The only field I need added to the om_show type is an external_id. This to keep om_show nodes in sync with cablecast data. For tapes I can enter everything I need into Cablecast. As we want to move away from tapes I see no need to duplicate that in Drupal. I then have a cron task which syncs the cablecast shows to Drupal.
I'm assuming when the om auto scheduler runs it can query the duration from the cablecast database. The auto scheduler can also setup the show to be recorded by the Cablecast system as well at that point.
For Digital files everything will be done in Drupal and follow the DOM workflow. When the om_show node is created or updated hook_nodeapi will be called in a cablecast plugin module to add/update the info in Cablecast. This way until the auto scheduler is up and running we can still schedule the digital files.
My biggest trouble thus far is figuring out media mover. I've attached a workflow diagram to help me figure this all out. Filefield local didn't work at all for me and I'm trying to stick to Drupal core anyway. Plus I'm way to lazy to select a file and then wait for it to be moved after I create node. My workflow is based on just entering the metadata in Drupal as a show node, getting the external cablecast show id after you save the node and then uploading the digital file to the dropbox. After that I'll let the computer do everything else. For tapes I'll enter the metadata in Cablecast (or hopefully the producer already entered it for us in Drupal) and then schedule the program and enable it to be recorded. The Cablecast system will then playback the tape and record it into the dropbox. From that point the computer will do everything else.
I'm thinking Media Mover will work fine for creating the FLV and the thumbnail and plus for doing the fancy stuff like uploading to bliptv or archive.org. But for the initial harvest of the file into the original directory and then conversion into the Mpeg2 file I'm at a loss. I can make it work. It's just slow. In the diagram I"ve added the logic I need to optimize this process. If I could do it in Media Mover well great. But I'm at the point of just wanting to get this all done.