More elaborate ways of merging ranked lists. First, the assessment of a document whose ranked order is highly correlated across retrieval methods provides little information about differences between the methods. Said another way, we can potentially learn the most from those documents whose rank order is most different, and hence a measure of the difference in ranked orders of a particular document might be used to favor “controversial” documents. This factor has the unfortunate consequence, however, of being sensitive to what we would expect to be the least germane documents, those ranked low by any of the methods under consideration. A second factor that could be considered is a “sanity check,” including a random sample near the top of our list. While we might learn a great deal from these samples if users agree that these randomly selected documents are in fact relevant, we expect that in general the retrieval performance of the systems should not depend on random documents.