There are over 100 billion images on the internet today and continues to grow every day. Image search engines often only surface a portion of those images and often rely on text surrounding an image on a webpage, or image file name. With the growing number of images on the Internet it is important to have the ability to organize and surface the images in the most efficient, meaningful way possible so that more images can be surfaced to searchers.
The challenge to researchers in the multi-media community is to 1) develop a robust way of understanding user intent and 2) generate highly relevant clusters for the given intent and query.
Metrics/Evaluation
There will be 4 criteria for evaluation:
- Precision of estimating successful user intent (goal: 90% success)
- Relevance of clusters.
- Performance efficiency of the underlying algorithm.
- Time it takes the user to find an image or groups of images.
Criteria for evaluation will be how efficient the system is to estimate successful user intent and to then surface relevant, meaningful clusters in the shortest amount of time possible.
Researchers working on this challenge will develop a way to successfully measure user intent, relevance of clusters and performance efficiency. In addition, creativity in presenting clusters and allowing for ease of searching and browsing will also be a key criteria. Lastly, the elegance of the solution will be judged by its ease of integration into a search engine’s pipeline, and the efficiency with which it can understand user intent and process one or more meaningful clusters – this latter part refers to processing speed. If a technology takes too long to provide meaningful clusters vs. another technology can process the clusters very quickly, then the latter is much more attractive.
Dataset/Suggested Queries
We may be able to provide a sample dataset of queries. Stay tune on the Multimedia Grand Challenge website.
Feel free to correspond with the challenge authors via the comments form below.
For private correspondence, consult the About page for contact details.
on Feb 2nd, 2009 at 4:54 pm
Is there going to be a common image data set to work on? Or do we just choose our own large sets?
on Feb 5th, 2009 at 6:45 am
It seems that you (we) have to provide such large sets…because they said : “We may be able to provide a sample dataset of queries” just a sample of queries…
Let’s wait…
on Feb 5th, 2009 at 10:42 pm
could you give a more formalized definition of user intent?
thanks.
on Feb 9th, 2009 at 5:36 pm
For now you folks can feel free to work with an image data set of choice. The solution needs to be widely applicable to web-images, so any large data set that is sufficiently diverse should work for this purpose.
As regards a formal definition for ‘intent’, I am afraid there is not going to be a more formalized version. It is what the user intends with a query or a session of queries that we are after. Task success and satisfaction in the completion of a task are the criteria generally viewed by us as meeting user intent.
on Feb 16th, 2009 at 1:03 pm
Where can I find a large Yahoo! image dataset?thanks
on Feb 20th, 2009 at 4:01 am
After re-reading (again) the challenge description it seems that we need to implement some user interaction (Relevance Feedback) in order to estimate the user intent, doesn’t it?
And the “Time it takes the user to find an image or groups of images” should be measured by means of number of user interactions.
on Feb 23rd, 2009 at 7:13 pm
[...] Robust Clustering Guided by User Intent in Image Search With the growing number of images on the Internet it is important to have the ability to organize [...]
on Oct 26th, 2009 at 9:26 pm
[...] Best Presentation: Christoph Kofler, Mathias Lux. Dynamic presentation adaptation based on user intent classification (response to the Yahoo! image challenge). [...]