Drupal's integration with new Youtube API to have a video portal profile

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
dipen chaudhary's picture

[ Abstract ]

A quick search of open source frameworks for video portals give results like Phpmotion and Vshare, which are not free or not completely free. Other problem with these applications is that the portal owner/administrator will have to manage media and cost that comes with it.Youtube's initiative in form its new API enables the complete CRUD workflow and video's are managed by efficient google clusters.This brings the opportunity to build a functionality of "out of the box" video portal as suggesed by kbahey .With help of a new module and plugging in with existing functionalities of drupal like cck, feed api we can create polymorphic drupal which can primarily act as video portal.Current contrib modules on drupal.org like emfield, media_mover does a good job but then again are meant and were designed for different purposes.I think we also need a GDATA protocol skeleton in drupal in form of api to talk to google services in a more flexible manner.This project though will use feed api and set namespaces for Youtube over GDATA specifically, The idea is to build a module or couple of modules which can then be rolled out as a installation profile for a video portal.

[Drupal.org Username ]

Dipen Chaudhary
My user node : http://drupal.org/user/100291

[Link to Proposal Discussion ]

http://groups.drupal.org/node/9829

[Benefits to Drupal/open source community]

This will add a installation profile which enables drupal as a video portal and benefits open source by giving functionality like Phpmotion/Vshare even better (as video's are managed by google) and is truly free and open, As youtube decided to give away chromeless player, it will lure developers to make video portals of specific domains like guitar tutorials etc and as youtube API has CRUD now,this will also benefit youtube if domain specific user's prefer to upload their videos on these and the same spirits will be seen in community related sections of youtube like commenting etc. I really think domain specific portals will come up if we can provide a painless framework in conjunction with drupal for developers to create such portals.

[ Project Details ]
I'll be using feed API or if time permits, I would wanna make that skeleton of GDATA protocol for drupal which was a project idea in Gsoc
2006 http://drupal.org/node/60490 and has come up again this year http://groups.drupal.org/node/10142 .. This module ideally should also plug with aggregator module to retrieve new videos based on categories, user and other axises.. GDATA protocol primarily talks XML
and drupal has a pretty stable feed parsing API's and methods, so this project is very much feasible. System will come with a predefined cck
type like image module. I have studied the new Youtube API briefly before assessing effort required and I think we can implement all of
CRUD, community based interactions like commenting, rating and video responses once authentication system is working and interacting
correctly with YouTube. Reading from YouTube doesn't require authentication hence each tag can be parsed and put in a cck
node type with rating,comments and other fields with it for which most of the work is done in emfield, whereas any of CUD or user specific
tasks will require authentication which can be plugged to Youtube's authentication system which is based on tokens as all GDATA based
services.Once a user uploads or updates a video on YouTube appropriate node in drupal gets updated.I think we can also exploit Youtube
Javascript API to define cool events like if a user navigates from page video is paused. Javascript API really gives fine control over
the player.We can extend emfield to have a predefined cck type with all fields now and use that for reading (R of CRUD) and have a
separate module for CUD or move the whole functionality to a new module.My findings shows that this project is very much feasible and
can be completed in given time line.

[DELIVERABLES]

1> For the design uploading workflow is already given in YouTube dev
docs for Read and Write I will create appropriate schemes and designs.

2> Extend emfield for Reading and a new module for all other
authentication based tasks,or a totally separate module as my mentor
would suggest.

3> Customizing installation as a profile and basic plugging with
aggregator module

[ Project Schedule ]

April 14 - May 25 : reading up all necessary documentation, trying
code sample, cvs set up, Design of the project with help of my mentor

May 25 - June 10 : Either extend emfield or do Reading part of the API

June 11 - June 15 : Authentication System

June 16 - June 25 : Things like user feeds, standard feeds like "top
rated", "most watched" etc.

June 26 - July 6 : Uploading, Updating and Deleting

July 7 - July 13 : Community related functionality

July 16 - July 25 : Javascript and Player building

July 26 - august 2 : Aggregator module pluggability

August 2 - August 11 : Coding style and optimization with help of my mentor

[About me ]
I found drupal one and half years ago when I was building a communication portal for my LUG (www.vit.ac.in/lug), I was in the 3rd
year of my college then, after which I did some freelancing using drupal, but pretty basic stuff. I got a big leap when I got a chance
to work on my final semester project with Dr. Sunit Gala, CTO of careseek Inc. (www.careseek.com) in December of 2007. I am still a
trainee here and working on "ontologies of folksonomy using drupal's controlled taxanomy" which will have benefits in tag cleaning based on
classifier built using theory of evidence.I will finish with my project by May 8 2008 and during this process I will have worked on at
least two social networks viz careseek.com, soon to go live on Drupal ( April 14 is expected date), and Ilovebolly.com (First week of May).
In these projects I really got to enhance my drupal skills and extended few modules which will soon patch into the community (After
going live). Few things I worked on were extending fivestar module to have rating on multiple axis, Correcting Node Import to take multiple
node references, Primitive Multi level marketing structure, Correcting cck_field_privacy for some basic error's. My curriculum project on
semantic web made me look into YouTube API to collect metadata for tag cleaning. I will soon start blogging about my drupal experience of last 5 months on www.playdrupal.com, I have at least 40 drafts buried in there which I wish to finish some day.. I rarely update my blog which
is located at http://blog.dipenchaudhary.com because of lack of time. I was once in the documentation team of Dojo Toolkit and helped in
writing book 1.0. You can also find me in contributor's list, but currently fascinated with jquery as its the endorsed JS library for
drupal and fun to code in. My other field of interest are high performance computing which I undertook as my research area for 2
years and have some interesting projects on Clusters and Grid Computing

Comments

I put this on the application page...

Alex UA's picture

For #2 of your deliverables, I'd like to see this plugin to the media mover module. You should also look at the MM BlipTV module, which integrates Blip.TV with Media Mover via their api, and Media Actions, which preceded it.

Please note that the author of the Blip.TV modules believes that someone is working on the YouTube implementation already, so you should really contact the module's author and take their work/advice into account. In fact, I would highly suggest trying to <a href="http://drupal.org/node/238666#comment-785071>work with pfournier on the following:

I am thinking about extracting bliptv code (and eventually YouTube and other providers) to some kind of API module that would be used by Media Mover and Media Action (and other modules).

Alex Urevick-Ackelsberg
ZivTech: Illuminating Technology

Alex Urevick-Ackelsberg
ZivTech: Illuminating Technology

keeping all at one place :)

dipen chaudhary's picture

So that I can keep track and do not spam topics which are about something else, I am copying my reply here to arthurf which was at
http://drupal.org/node/238666


@arthurf
Thanks for your inputs, Right now I am thinking of the best possible integration scheme of drupal with new Youtube API so that we can get a out of the box video portal platform.. I am thinking if mine should be a totally new module or I can use some of the existing modules to achieve my task .. If you got a chance to read my application I was considering only emfields out of the existing modules coz of my ignorance of other modules, that too for displaying videos (which doesn't need authentication) .. Do you have ideas as to how media mover can help this project achieve its task as it takes care of storage and I saw it interacting with S3 of amazon as well ..
My understanding of media_mover is on its 4 pillars harvest, process, storage, Complete

Harvest: For my project, harvesting is usually uploading file from your own computer, Yes I can use getting it from e mail attachment but I see it as a rare case and kind of fancy support which is good to have but not a show stopper, From ftp again is a cool way and can be useful if administrator wants to put his collection of videos in a batch mode or something ..

Process: flv conversion is done at youtube end and I think we should be happy about it .. Adding metadata is again cool but then if conversion is happening at youtube, then I dont see much flexibility offering in this regard ..

Storage: this is rather interesting as media_mover already have support for S3 which is a cool webservice for storing data, We can do same for youtube and makes perfect sense to do it ..

Complete: Once everything is complete insert appropriate links, like the link to newly uploaded video or any updates to our local drupal installation ..

I would want to hear more of your thoughts over my approach, writing this felt good :) .. Can you please reply at my application node so that we dont spoil it for pfournier as the topic of discussion is something else and we are spamming it .. Also your reply at my application will make more people review it ..

My application is located here : http://groups.drupal.org/node/10295

And yes I will try to get roger talking as well .. I need all your inputs so that I can update application .. Needless to say that I am gonna work on it even If it doesn't get accepted as I have already invested my thoughts in it ..

Thanks again ..


Dipen Chaudhary
http://dipenchaudhary.com
http://playdrupal.com (under construction)


Dipen Chaudhary
Founder, QED42 http://www.qed42.com Drupal development

My response...

Alex UA's picture

Here's my response from that thread:

Dipen-

I actually think there are two harvests here, though you might not always need both. First, you could upload to the site via a field. Second, you could harvest videos that are already on youtube (check out what Arthur does with the XPSF playlist and MM here for an example of pulling videos using a term: http://www.24b6.net/?p=223 ). Check out the latest version of asset to see how Roger uses emfield and youtube's feeds to pull vidoes from a feed, it's pretty cool and also may hold some clues on how to do this. So, in the case that you are uploading a file and want to automatically pull back to your site I'd see a two-step harvest process, not sure if that can be automated, but if it can it would be awesome. Also, these sites need time to process your files, assuming you're using their conversion services, so you'll need to pull the file back only after it's ready. Patrick can give you some insights on how this works for the mm_bliptv module.

Do you know if YouTube is going to allow for multiple formats? If so you'll also need to figure out a way to allow people to harvest a specific, or a few specific, file formats.

Alex Urevick-Ackelsberg
ZivTech: Illuminating Technology

Alex Urevick-Ackelsberg
ZivTech: Illuminating Technology

Dipen's Response to me...

Alex UA's picture

hi Alex

Straight forward things first :)

1> This is what is given on youtube in terms of formats, there is not a exhaustive list but then I think its something I will need to test, but the video cant be more than 100Mb

What's the best format to upload for high quality?

YouTube accepts a wide range of video file formats such as .WMV, .AVI, .MOV, and .MPG. For the highest quality, we recommend you upload files directly from your camera software if possible. If that's not possible, export your videos in the highest quality setting available in your editing software.

Generally, we recommend the following settings:

* MPEG4 (Divx, Xvid) format
* MP3 audio
* 30 frames per second

2> I think the process can be automated via cron or batch (effectively same), As for video being showed and indexed on youtube it takes time and can take upto one day .. We get a unique Id for each uploaded video which can be accessed through user's video feed and tag can be read to see what is the status of video ... Quoting from dev docs

An uploaded video will immediately be visible in an authenticated user's uploaded videos feed. However, the video will not be visible on YouTube until it has been processed and indexed. The length of time between when the video is uploaded and when the video is publicly visible on YouTube varies. However, videos are usually indexed within one day and may be indexed in several hours or even less than an hour.

You can check the status of a user's unpublished videos by retrieving the user's uploaded videos feed. Entries in that feed that have not been published will contain the tag. Please note that unpublished videos include videos that are still being processed, that failed to upload or that were rejected after being uploaded.

Whenever a user uploads a video we mark that event as something to be processed via cron, When Its available we pull it from youtube

3> Harvest as a two way process .. I can use a paper and pen to understand this and put my understanding forward .. I will scan my understanding in some hours .. But in a nutshell as I see it

http://www.24b6.net/?p=223 -- this will be applicable in R of CRUD (btw its very cool and is it wordpress :P ? ) As I understand harvesting, its a way to get media. When a user is uploading a video, I think he will either do it from a field (his own computer) or a ftp (if he has some repo of videos) so as far as CU (create, update) are concerned how can youtube be a source of harvesting (or you meant for displaying) ? Youtube will be a source of harvesting, when Displaying of videos happen (either user's previously updated videos or via feed like the example you gave), More on this when I scribble some workflows on paper ...

Again thanks a lot :)

Btw why do you all hate to post on my application page :( ? I appreciate all your inputs and I am enjoying it a lot :)

Thanks again ..
...Dipen

Alex Urevick-Ackelsberg
ZivTech: Illuminating Technology

Not talking to myself....

Alex UA's picture

...I swear. :D

Dipen- When I mentioned various formats I didn't mean on the uploading side, but rather I'm curious about whether YouTube will go the way of Blip.TV and allow users to host various formats off of their servers. On Blip you can have Quicktime, Windows Media, Itunes/Ipod, Flash, and more. I'm guessing YouTube will allow for this at some point, if it's not in the plans already. But- it's something you should look into and at least offer a method in the future for people to extend the module to support other formats.

As far as harvest goes, what I am suggesting is that it would be great to be able to pull videos that you upload as well as from those that you search for (Asset may be the method to combine the two, but I think it would be better for the module itself to accommodate this- at any rate take a look at how Asset searches and pulls from YouTube feeds). As you can see in Arthur's demonstration, it's already possible to use XSPF playlist to pull feeds via a term, and there's no reason that this module shouldn't accommodate users who wish to use this to display their own and other people's videos, side by side. And as far as FTP and HTTP go, we're moving more towards XML and RDF every day, so we'll need to make sure that this can work with those. The semantic web is coming, and I really think that this project needs to take that into account.

And yeah, what's up with Arthur using WordPress? Someone needed to speak out, and you were brave enough to do so. Bravo. ;-)

Alex Urevick-Ackelsberg
ZivTech: Illuminating Technology

Alex Urevick-Ackelsberg
ZivTech: Illuminating Technology

hey again :)

dipen chaudhary's picture

Didnt sleep all night :) gotta go at my training place in like 30 mins :P

Ok I certainly took it on the uploading side, my fault .. Alex, from media_mover's perspective or for that emfield perspective how does it affect a module if youtube is hosting different formats and disseminate video's in different formats (I think it will, I read couple of weeks ago about high definition video's and flv's may not give that ) I get a unique url for each video and I get youtube's player which stream's the video. Now difference between Blip.tv is I can download those .mov file etc to my computer using a get source url (previously I could I am not sure abt now). How will different formats affect me until they are hosted on youtube and youtube gives a player which streams them .. Please let me know if I am missing something here .. As I see I just need a url and get my displaying correct as I dont wanna be concerned with hosting the videos.

Certainly getting video's using terms, search, subscription is on my mind I even mentioned in my application.. I would have used aggregator module and feed api to do it before but I know better now (MM and Asset), Also with new API subscriptions and everything is in place and we can have a block which gets video's based on terms (tags) added to drupal while a user uploads new video's and tag's them, We store all the metadata on our portal as well ...

Btw as for semantic web, I think I have already done some work on what you are suggesting .. http://groups.drupal.org/node/10297 .. This is second application I proposed I have not written it officially on gsoc page because I need some input .. I did a quick prototyping of getting youtube metadata via their old api's .. I am yet to work on the classifier and I got interesting links from reinhard after seeing my semantic web application .. links are http://drupal.org/project/moat and http://scot-project.org/

I tried to contact aaron as well .. I am gonna contact roger in couple of hours ..

Thanks :)


Dipen Chaudhary
http://dipenchaudhary.com
http://playdrupal.com (under construction)


Dipen Chaudhary
Founder, QED42 http://www.qed42.com Drupal development

sleep is the enemy

Alex UA's picture

The video formats won't effect you at all on the upload part, but you'll have to give people an option for which video format to use when they pull it back to their own site. If you install emfield and enable the embedded video field, then visit admin/content/emfield on your site you'll note that under blip.tv there are multiple options for which format to use. Ideally, we'll (I help maintain emfield too) move the settings out of their and into the content type so that you can select different formats per content type you create, rather than having one video type per site. Either way, you'll need to give users the ability to pick whichever video they want. This shouldn't be too hard, as the xml file associated with a video lists the formats that are available (at least on Blip).

I would highly suggest using FeedAPI instead of Aggregator- the FeedAPI Mapper is how I'm able to create emfield video nodes from a feed on my site (I linked to it above), and FeedAPI seems to be the most robust of the contrib aggregation modules. I definitely plan on pulling from YouTube feeds, and I'm assuming many others will as well, so you should try to make a good way to interact with feeds.

Aaron's finishing up a huge project at work (the launch of a big site), so he's probably unavailable for another few days. I'm sure he'll chime in when he has a moment.

And that's awesome, re: the semantic web work. I did note that on your application.

Alex Urevick-Ackelsberg
ZivTech: Illuminating Technology

Alex Urevick-Ackelsberg
ZivTech: Illuminating Technology

Aaah yeh choice to the user ...

dipen chaudhary's picture

I get it, the user should have the choice to select the format ... Thanks for all the patience ,...
I am gonna check out FeedAPI Mapper tonight :)
I am gonna update my application tonight, thats back breaking :P (I think I am insomaniac) Thanks for all the advice :) and letting me know of all potential people who could help .. I am gonna write back if I have doubts over the design


Dipen Chaudhary
http://dipenchaudhary.com
http://playdrupal.com (under construction)


Dipen Chaudhary
Founder, QED42 http://www.qed42.com Drupal development

and any tips on how to contact roger

dipen chaudhary's picture

He doesnt have contact setting enabled, his domain is probably parked domain with Jeniffer Lopez adds on it (so cant comment there) ..
What do you say Should I file a issue :P ? Support request that is .. I kind of felt bad doing that on emfield :|


Dipen Chaudhary
http://dipenchaudhary.com
http://playdrupal.com (under construction)


Dipen Chaudhary
Founder, QED42 http://www.qed42.com Drupal development

I sent him an e-mail...

Alex UA's picture

I sent Roger an e-mail asking him to chime in here, but you could try asking in the asset queue as well.

Alex Urevick-Ackelsberg
ZivTech: Illuminating Technology

Alex Urevick-Ackelsberg
ZivTech: Illuminating Technology

updated profile

zroger's picture

i updated my g.d.o profile, so you should be able to use my contact form.

How can I use asset ?

dipen chaudhary's picture

@Alex - Thanks :)

@Roger - Sorry for delay, I was travelling coz of some formalities in my college .. Application deadline is long gone but I wanna pursue it nonetheless .. I wanted to know how can I use Asset module to achieve my problem statement. I had great discussion with Alex and some with arthur on how to use media mover and have thought of using MM for harvesting, storage and complete ..
How can I exploit asset ?


Dipen Chaudhary
http://dipenchaudhary.com
http://playdrupal.com (under construction)


Dipen Chaudhary
Founder, QED42 http://www.qed42.com Drupal development

Any update on the project Dipen?

krem's picture

Hi guys,

I am very interested on this media portal functionalitie for drupal, and was wondering if any further discussion or development has taken place?

I still need to understand many concepts of the project but once I feel confortable, I will be willing to help!

Bye

Project nearly complete

aaron's picture

beeradb has nearly completed the YouTube API wrapper module, and should release it in a few days. http://aaronwinborn.com/blogs/aaron/youtube-api-drupal-way

Aaron Winborn
AaronWinborn.com (my blog)
Advomatic, Web Design for Progressive Advocacy, Grassroots Movements, and Really Cool Causes

Aaron Winborn
Drupal Multimedia (my book, available now!)
AaronWinborn.com
Advomatic