Course description

The course explores the requirements, interactive and mathematical models, design choices and the evaluation of Information Retrieval systems (IRS). IR systems refer to automatic (computer-based, such as Web search engines) rather than "manual"/"intellectual" systems.

Although not required, basic understanding of maths (simple probabilities, statistics, ...) and computer concepts is recommended. It is also recommended that students taking this course have taken, are taking or will be taking courses covering topics in:

• Human Information Behavior - for an introduction to the theory, study and practice of human interaction with information.
• Principles of Searching - to better understand what functionality is expected of IR systems.
• Organizing Information - to better understand the issues that need to be considered when designing an information system and approches taken before the time of the computers.
• Digital Libraries - for studying in detail one particular type of information access system.

Course objectives

• the genesis and variety of information retrieval situations;
• the variety of information retrieval models and techniques;
• design principles for information retrieval systems;
• methods for implementing information retrieval systems;
• characteristics of operational and experimental information retrieval systems;
• methods and principles for the evaluation of information retrieval systems.

Class conduct

The course has two parts: 1. "Textbook IR" covers the accepted current wisdom in IR research and practice, i.e. topics that have been thoroughly studied and are relatively well understood. The purpose of this part is to give all students a grasp of what IR is, what problems it tries to solve, which approaches have proved successful and which not. Each meeting day, there will be a lecture and discussions on the scheduled topic. Students are expected to read the relevant chapters or sections from the recommended books before the class and to participate in discussions on that topic. In each class, a couple of students will give a short presentation, doing a critical analysis of a paper or topic. Other students are expected to critically discuss the presentations. 2. "Advanced IR" covers current research topics in IR.According to their preferences, students will choose topics of interest from among a list of proposed topics, and will do a more thorough investigation and give a longer presentation and write a report. The number of topics that each student has to present and the depth and coverage of the analysis will depend on the number of students taking the course. Websites. All the lecture notes will be available online. The course websites will contain links to students websites. Students are expected to maintain websites for the course on their scils accounts, in a subfolder of public_html called 551/614. They will use the websites to publish their presentations and reports, and to upload their homework.

Grading

Students will be graded based on reports, presentations, term project, homework and class participation. The final grade will be a weighted average of the partial grades. The report and presentation of a paper (or topic) should make clear: (1) what problem the paper addresses; (2) what relation it has to prior cited literature and to the current topic discussed in class; (3) what idea it proposes to solve or improve the problem; (4) what was done to implement that idea; (5) what results were found and how well the rseults were interpreted, and (6) what suggestions were made for further work. The report should have a clear and logical structure of the topic and of the argument and an appropriate title or indication of the topic (it can help focus and structure the argument), and to display critical analysis. In addition, the presentation should make good use of visual aids to help the listeners understand the topic and the argument.Critical analysis is essential. Over the semester, each student is expected to give at least 3 presentations: (i) one on a classic paper; (ii) one on a recent paper; (iii) one on an IR topic, based on at least 2 papers (one classic, one recent); this presentation should be accompanied by a report. Practical project work - is expected to display: Good understanding of the topic chosen and of the IR context Creativity Substantial work Below are suggestions of possible projects, but you can propose your own if you have a particular interest. Students are expected to choose a project soon after mid-term, to discuss their progress with the instructor, and to give a presentation at the end of the course. Class participation - apart from active, thoughtful and creative participation in class discussions, critical discussion of presentations and most homeworks are seens as class participation. Although not graded as such, class participation can determine the adjustment of the final grade by one position. Graded homework may be assigned occasionally in the form of exercises designed to re-inforce learning.

Grades

• A - Outstanding achievement. Student performance demonstrates full command of the course materials and evidences a high level of originality and/or creativity that surpasses course expectations.
• B+ - Very good work. Student performance demonstrates above average comprehension of the course materials and exceeds course expectations on all tasks as defined in the course syllabus.
• B - Good work. Student performance meets designated course expectations, demonstrates understanding of the course materials and performs at an acceptable level.
• C+ - Marginal work. Student performance demonstrates incomplete understanding of course materials.
• C - Poor work; not enough effort shown.
• F - Fail.
• IN - Incomplete.

The practical project

The project may be real research (when some research hypothesis is proposed and tested), or mock research (when some of the data, such as subjects' responses to questionnaires, are generated automatically rather than real data), or literature review or practical implementation of a system.

Possible projects:

• Conduct a user experiment on a research problem and analyze the results.
  Examples:

  • Compare the effectiveness of two systems or user interfaces for a certain task
  • Test the usability of a system
  • Repeat an experiment described in a TREC or SIGIR paper.
  Required: a report discussing the research problem and or the purpose of the experiment, the experimental design and setting, the results, conclusions and implications or recommendations for designing IR systems.

  For a realistic experiments that involves 6-16 sujects, the use of questionnaires / interviews and/or the statistical analysis of the experimental results, teamwork is not only acceptable, but encouraged. On the other hand, suitable for individual students would be, for example, to compare two Web search engines based on functionality ("what we expect from an IR system") and the support that their user interfaces offers. Note that the functionality of commercial system is not well / fully documented (order and weighting of query terms, for example), so an informed guess, based on exploration such as observing the output for various inputs, is necessary.

• Give a detailed description and critique of some operational information retrieval system.
  Discuss functionality, usability advantages and diadvantages over similar systems. Draw conclusions and formulate recommendations with regard to the use of the system. Examples:
  • AntWorld
  • Onix
  • Mapuccino
  • Inquery, from CIIR, University of Massachusetts, Amherst
  • MG
    I.H. Witten and A. Moffat and T.C. Bell, “Managing Gigabytes: Compressing and Indexing Documents and Images”, 2^nd ed, 1999. Other bibliography indicated in the MG webpage.
  • Lemur, at Carnegie-Mellon University
  • Cheshire, at UC Berkeley
  • Okapi
    More info on the Okapi project at City University.
  • Lucene

  Required: Cover the installation of the system, the usefullnes and usability of the system in a range of tasks.

• Constructing and documenting (a significant part of) an information retrieval system, or adding new modules to an existing system, and evaluating it by demonstration (this could be an individual or a group project for students with programming skills).

Please read the University's Policy on Academic Integrity for Undergraduate and Graduate Students.