A notion of browsing collections is naturally associated with videos. Having videos classified into a pre-existing hierarchy of genres is one way to make the browsing task easier. The goal of this task would be to take user generated videos (along with their sparse and noisy metadata) and automatically classify them into genres. A public genre hierarchy like ODP (Open Directory Project) can be used as a target for this task.
Evaluations can be based on purely video content features as well as a combination of content and metadata features. Features that bring in information from other public data sources can also be used (eg. Object detectors trained on a separate public dataset). Thinking of new (and surprising) features is recommended!
Any dataset that reflects a breath of content is acceptable, and of course, YouTube and Google Video are a recommended source. Particularly, the data should cover most of the common video genres. If the dataset consists of web videos, sharing a list of links and corresponding labels would be ideal for researchers to compare notes. You may want to consult the The YouTube Data API for retrieving video data.
Evaluation
We propose two types of evaluations for this challenge:
- Offline (direct evaluation): Use a labeled test set to measure precision/recall for the ODP categories.
- Online (indirect): Allow users a browse interface for your classifiers and measure how easily they can find some target concepts (e.g., find a basketball scoring scene). Note that the errors of the classifier can be compensated here since a video can appear in multiple categories, so one could conceive of training for different loss functions here.
The ideal target in this case would match the optimal score for human agreement on the dataset. If 5 raters categorize each video and we have agreement in 92% of the cases, we expect the automatic classifier to hit the same agreement rate.
Feel free to correspond with the challenge authors via the comments form below.
For private correspondence, consult the About page for contact details.
on Feb 11th, 2009 at 7:10 pm
[...] Robust, As-Accurate-As-Human Genre Classification for Video A notion of browsing collections is naturally associated with videos. Having videos classified into [...]
on Feb 24th, 2009 at 5:23 am
[...] Google Challenge: Robust, As-Accurate-As-Human Genre Classification for Video [...]
on Sep 3rd, 2009 at 1:40 pm
[...] Responses to Google Challenge: Video Genre Classification [...]
on Sep 8th, 2009 at 10:22 am
[...] will be presented at the Content Analysis and HCM Session. In addition, we have been accepted to compete in the Multimedia Grand Challenge, in the theme Tangible MM Products, set by CeDeWe, a major photo printing service provider in [...]
on Apr 1st, 2010 at 9:17 am
In “If the dataset consists of web videos, sharing a list of links and corresponding labels would be ideal for researchers to compare notes.”, does “a list if links” mean the links of the web videos and is the “corresponding labels” choosen by yourself or from the open ODP?