Detailed proposal for TREC 2001 Interactive Track

[DRAFT]

A. Motivating principles:

For TREC 2001 the Interactive Track will carry out observational studies which maximize the realism of the searching by using data and search systems/tools publically accessible via the Internet. This suggests that, to the extent possible, searchers be allowed to choose tasks and systems/tools for accomplishing those tasks.

At the same time, the studies for TREC 2001 should be designed to maximize the likelihood that groups will find in their observations the germ of a hypothesis they can test for TREC 2002. This suggests that there be restrictions - some across all sites, some only within a given site - to make it more likely that patterns will emerge. The restrictions will be formalized in 2 sorts of guidelines: one set for all sites and another set that applies only within a site.

B. Cross-site guidelines:

• Each site will observe at least 24 searchers
• Each searcher work in only 1 of the following domains provided by the track to all sites:

  • finding consumer medical information on a given subject
  • buying a given item
  • planning travel to a given place
  • collecting material for a project on a given subject

• Each searcher will carry out 4 searches - 2 the searcher chooses from a list of fully specified tasks provided to all sites and 2 for which only the format is predetermined but which are otherwise up to the searcher to create.
• Each site will collect a minimal standard set of data defined by the track and covering searcher characteristics and satisfaction, effectiveness, and efficiency. These will be submitted as part of the official results.

  C. Within-site guidelines:

  • Within the cross-site guidelines, each site must impose further restrictions of its own choice on ALL its searchers to define an area of interest for observation - to be reported to the track before the observations begin.

    For example, a site could impose inclusive or exclusive restrictions on any (combinations) of the following: the choice/assignment of domain from the 4 provided, the data to be searched, the search system/tools to be used (e.g., search systems, meta-search systems, directories,...), functionality within a given search system/tool, the characteristics of searchers, the time allowed, the pre- search training provided, etc. Sites are also encouraged to coordinate their plans with other sites, form small teams sharing guidelines, etc.

  • Each site will evaluate their searches using any criteria defined in the cross-site guidelines plus any site specific evaluations.
  • As part of the data analysis for TREC 2001, each site will attempt to formulate a testable hypothesis for TREC 2002 and will report this as part of the results for TREC 2001.