Classroom Evaluation of the Alexandria Digital Earth Prototype (ADEPT)

 

Gregory H. Leazer, Anne J. Gilliland-Swetland, Christine L. Borgman

Dept. of Information Studies

University of California, Los Angeles

 

Richard Mayer

Dept. of Psychology

University of California, Santa Barbara

 

 

Abstract

 

One important aim of digital library efforts is to enhance student learning by providing important educational resources in appropriate learning environments. The Alexandria Digital Earth Prototype (ADEPT) is a five-year research project that continues and extends the Alexandria Digital Library.  The promise of digital libraries like ADEPT is that they will create opportunities for students to be active learners in locating relevant information, balancing evidence, synthesizing knowledge, and developing their own conclusions.  The evaluation plan for ADEPT centers on two investigations: a study of the classroom use of the system by faculty and stu­dents, and laboratory-based usability study.  The classroom-based evaluation is an investigation with three primary aspects: the use of ADEPT in multiple disciplines; instructor's pedagogical approaches and activities in implementing ADEPT in the classroom; and student learning outcomes associated with ADEPT.  The laboratory-based portion of the evaluation aims to investigate how users' mental models of ADEPT evolve over time and how they are related to student learning.  Underpinning both evaluation components is an investigation of whether and how student learning can be enhanced through the use of ADEPT and, by extension, other digital library systems. This paper describes the evaluation plan, and briefly discusses it limitations.

 

 

1. INTRODUCTION

 

Research activities aimed at the development of digital libraries take two distinct and complementary forms [Borgman 1999]:

 

·         the development of experimental digital library systems, consistent with a larger and continuing research front on experimental information systems, especially information retrieval systems, and

 

·         the development of effective digital library services, consistent with a larger effort aimed at developing and maintaining beneficial information services and institutions, such as libraries, archives, and museums.

 

Recent research programs funded by the National Science Foundation demonstrate a growing concern about the effectiveness of digital libraries, not only in terms of information seeking and retrieval, but also in terms of how they can directly enhance learning. The Digital Libraries Initiative Phase 2, for example, "seeks to provide leadership in research fundamental to the development of the next generation of digital libraries, to advance the use and usability of globally distributed, networked information resources" [NSF 1999].  A more recent program, under the NSF's Division of Undergraduate Education, builds on the work of the Digital Library Initiative, and "aims to [create] a national digital library that will constitute an online network of learning environments and resources for science, mathematics, engineering, and technology (SMET) education" [NSF DUE 2000].

 

Digital library research efforts need to integrate systematic evaluation plans that assess how well design objectives have been accomplished, how effective the services are, and that identify new uses of the digital library in the user community. The objectives and uses of emerging digital library services are increasingly concerned with the support of student learning.

 

 

 

2. THE ALEXANDRIA DIGITAL EARTH PROTOTYPE (ADEPT)

 

The Alexandria Digital Earth Prototype (ADEPT) is a five-year project centered at the University of California, Santa Barbara (UCSB); the education and evaluation component of the project is centered at UCLA.  Investigators from both campuses are conducting a multi-year investigation of technology-supported learning using ADEPT in undergraduate classrooms.  We are studying the use of the system by teachers and students and the associated changes in teaching and learning.

 

The Alexandria Digital Library (ADL), developed under the first Digital Libraries Initiative (1994-1998) is an operational digital library that allows users scattered across the Internet to access collections of maps, images, and other geo-referenced materials from a 1.5 terabyte and growing collection of materials from UCSB's Map and Imagery Laboratory (www.alexandria.ucsb.edu).  The operational version of ADL provides users with access to services allowing them to answer such questions as 'What information is available about a given phenomenon at a particular set of places?' ADL also provides new types of library services relating to gazetteers and other information access tools.  ADL went online in the Fall of 1999 as part of the University of California's new California Digital Library (CDL; www.cdlib.edu).

 

The ADEPT project commenced in Summer, 1999.  It involves more than fifteen faculty from UCSB and UCLA, and a large number of graduate research assistants at both campuses.  Other participants include UCSB's Map and Imagery Library, UCSB's and UCLA's Offices of Instructional Development, the San Diego Supercomputer Center, the California Digital Library, Georgia Tech, the University of Georgia.  ADEPT is funded by the Digital Libraries Initiative (DLI-2, 1999-2004).  The project will provide a broad variety of analysis tools and modeling services.  Users of the ADL collections will be able to construct their own personalized digital libraries from information available over the Internet and to use these 'virtual' digital libraries in creative ways in collaboration with other users. In particular, the project will focus on supporting uses in classroom instruction in multiple disciplines, including the physical, biological, and social sciences, arts, and humanities. To do so, ADEPT is employing the digital earth metaphor for organizing, using, and presenting information at all levels of spatial and temporal resolution through digital environments known as Iscapes (information landscapes).

 

 

3. EDUCATION AND EVALUATION COMPONENT OF ADEPT

 

ADEPT offers an important opportunity to evaluate learning activities and integrate the assessment results into the design of the system. The classroom evaluation component of ADEPT focusses on assessing learning outcomes as a result of implementation of successive ADEPT prototypes in undergraduate classrooms, first in geography and subsequently in other subject areas where geo-referenced information may be useful (for example, urban planning, environmental studies, archaeology, and public health).

 

We are employing a range of research methods, including intensive analyses of individual users and large-scale studies of entire classrooms, using multiple dependent measures such as analyses of problem-solving processes, quantitative analyses of learning outcomes, and qualitative descriptions of user misconceptions.  These converge on understanding how people learn using the ADEPT system. ADL prototypes already developed have been instrumented for sophisticated data collection, including transaction logging and surveys.  We are now extending these capabilities in ADEPT [Borgman, Hirsh, & Hiller 1996; Hill and others 2000].  Results of the usability and evaluation studies will provide continuous feedback to the design of ADEPT services, functionality, and choice of collections.

 

Classroom-based Studies

 

The promise of digital libraries and systems like ADEPT is that they create learning opportunities for students to be active learners in locating relevant information, balancing evidence, synthesizing knowledge, and developing their own conclusions. This emerging constructivist learning paradigm requires new types of literacies that incorporate skills in the use of technology and information [O'Neill, Gomez, & Edelson 1994; Scardamalia, & Bereiter 1993 and 1996].  Within this context we are seeking through the classroom-based studies to address several key questions:

 

·         Does ADEPT have a positive influence on student learning?  In particular, does ADEPT increase student understanding of core geographic concepts and processes?

 

·         How effective is the use of Iscapes and the Digital Earth metaphor in facilitating ADEPT use, and learning by students in different courses?

 

·         Does student collaboration increase as a result of working with ADEPT?

 

·         In what ways might the pedagogical methods employed by faculty teaching the courses change as a result of implementing ADEPT in their teaching activities?

 

The first steps in the classroom studies are to gather baseline data about the performance and demographics of students in the same course for the five preceding years, and to gather information about faculty teaching practices and pedagogical objectives.  These data form a part of a needs analysis designed to identify faculty and student users, their tasks, task context, and the tools, content, collections, and metadata that may be useful in their environments.  Formative evaluation will build upon the needs analyses throughout the project, since needs will change as the system evolves and becomes more integrated into classroom instruction. Summative evaluation will begin midway through the project, and will require triangulating quantitative and qualitative methods to assess short and long-term learning and instructional methods.

 

We are currently evaluating ADEPT in geography classrooms where the use of ADEPT is integral to the curriculum.  Later we will study ADEPT in other disciplines such as history and classics where ADEPT may be an important supplementary resource for student information seeking and curricular enrichment. Faculty in multiple disciplines at UCSB and UCLA have agreed to participate.  We will recruit additional faculty and classrooms over the course of the study, as user needs evolve, as capabilities of ADEPT expand, and as the success of ADEPT attracts other participants.

 

The classroom studies focus on three aspects of the questions outlined above:

 

(1) The usability of ADEPT in multiple disciplines.

 

We are assessing the degree to which the Digital Earth metaphor, implemented via spatial and temporal information retrieval tools, is compatible with teaching and learning activities.  Of particular interest are the ways in which activities vary between disciplines, based on instructional goals, technical expertise, cognitive skills and styles, and ways the variance can be accommodated in design [Bos, Krajcik, & Soloway 1997; Wallace, Krajcik, & Soloway 1997].

 

ADEPT prototypes are being developed and tested, then refined iteratively to study research questions such as the following:

 

·         Do the content and capabilities of ADEPT assist in the scientific reasoning process of the discipline (e.g., geography, environmental studies) under study?

 

·         Given the problem of information overload in multimedia collections as large and diverse as ADEPT, how do faculty and students integrate information from multiple modalities such as text, narrative, spatial, and aural?

 

·         What are the effects of spatial metaphors and mental models on how people learn, construct, and discover knowledge?

 

·         Does the effectiveness of the Digital Earth metaphor vary by discipline and by learning context?

 

·         What is the time to learn and to relearn ADEPT? 

 

·         What are appropriate metaphors and mechanisms for collaborative learning and work in ADEPT? 

 

·         How do activities vary throughout the information life cycle, whether working alone or collaboratively?

 

(2)  Faculty activities in integrating ADEPT into undergraduate classrooms.

 

We hope that ADEPT will enable classroom instructors to convey dynamic processes, enhance the modeling of geographically-related phenomena [Meichtry 1992], and assess students in new ways, for example requiring them to analyze “what-if scenarios” such as ecological disasters. The classroom evaluation is addressing the degree to which instructors employ these capabilities to enhance instruction and to conduct new types of student assessment, such as student reasoning [Marchionini & Crane 1994].  ADEPT will allow instructors to review the evidence that students use in developing a conclusion by revealing the incremental work conducted by the student in answering the question, akin to how math instructors view students' work. The ADEPT evaluation team also is working with instructors to develop new types of assessment techniques.  These results will be compared to student learning using traditional assessment techniques.

 

(3) Learning outcomes associated with use of ADEPT.

 

Individual students within these classes will be tracked to ascertain the extent to which ADEPT is utilized in their subsequent learning activities.   Methods under consideration include pre- and post-implementation faculty and student interviews, specialized assignments designed to elicit indicators of student learning associated with the use of ADEPT, and bibliometric assessments of student paper citation practices. User-based evaluations will employ both experimenter-imposed and subject-generated tasks [Borgman, Gallagher, Hirsh, & Walter 1995]. The summative evaluation will also investigate whether the use of ADEPT can be correlated with successful educational outcomes.

 

To assure the generalizability of our evaluation studies, the majority of classroom research is being conducted in regular courses and classrooms at UCSB and UCLA, supplemented by instructional laboratories maintained at both campuses by departments and by the Offices of Instructional Development.  Both campuses have innovative programs in instructional improvement that are targeted at undergraduate instruction, provide extensive support in the design and production of multimedia instruction, have comprehensive instructional computer networks, and experience with extramural funding. Key professional staff from both OIDs are members of the evaluation team.

 

Baseline demographic data are being gathered through brief in-class student surveys of students together with two short cognitive tests that examine respondents' spatial cognition. The spatial cognition tests may be administered again at the end of the course to see if any change in spatial cognition has occurred. Student volunteers are asked to participate in three feedback interviews per course (at the beginning, mid-point, and end of the term) that provide feedback on the student's perceptions of the subject being taught and observations on the implementation in the course of ADEPT.  Faculty who are teaching the courses in which ADEPT is implemented and who are collaborating in the ADEPT research will participate in pre- and post-implementation interviews of approximately 30-40 minutes in length [King 1991; Gilliland-Swetland, Kafai & Landis forthcoming].  Usage patterns of ADEPT by students and faculty will also be captured automatically by means of transaction logging built into ADEPT.

 

To understand the nature of the impact of ADEPT in different learning situations, participating faculty instructors may provide granular grade data showing a numeric breakdown of student grades by assignment which can be matched to ADEPT topics.  We also are requesting that additional questions on ADEPT be included with end of term class evaluations.

 

 

Usability Studies

 

Laboratory-based usability studies complement classroom evaluation of learning outcomes by addressing cognitive analysis of users' mental models and problem-solving processes.  Consistent with advances in cognitive science, we envision the user as an active sense maker who attempts to construct a mental model of how the system works [Gentner & Stevens 1983; Halford 1993; Mayer 1989]. In using the ADEPT system, the student engages in communication and collaborative problem-solving with another intelligent being, and in the process attempts to construct a mental representation of how that "other mind" works. We hypothesize that successful users are more likely to have developed useful mental models of how the system works, and that redesigning the interface to foster the development of useful mental models will lead to improved performance [Allen 1997; Kieras 1988; Norman 1986]. The role of mental models in information access systems is a particularly fruitful research domain [Allen 1997; Borgman 1986; Walter 1996]. Given the rich research opportunities afforded by this project, we are examining two basic issues that have important theoretical and practical implications--the development of mental models and the role of system metaphors in learning.

 

First, we are assessing students’ mental models at multiple points in learning the system to determine how models develop over time.  Later we will compare how the quality of users' mental model is related to performance on various search tasks. Specifically, we are comparing the students’ mental models of scientific processes to normative models for the discipline under study.  It is useful to conceive of the development of expertise as systematic changes in users' knowledge, including changes in their mental models [Chi, Glaser, & Farr 1988; Mayer 1997a]. We will use a variety of approaches including intensive one-on-one interviews of selected users during the course of an academic year and short-term testing of knowledge at the beginning and end of the year. The tests will include analyses of the strategies that students use to solve various problems, changes in users' mental models as reflected in concept maps or interview questions, and overall outcome performance on benchmark tasks.

 

Second, we are examining the effects of incorporating various multimedia-based metaphors in the interface on improving users' mental models and their performance. Evidence is mounting that multimedia-supported metaphors -- such as the Digital Earth -- can be powerful aids for student learning [Mayer 1997b; Neale & Carroll 1997]. We will compare the cognitive consequences of using various kinds of metaphors for the search process. These studies have implications for interface design as well as for cognitive theories of technology-based learning.

 

4. SCOPE AND LIMITATIONS

 

While evaluating new technologies such as digital libraries in real-world settings is essential to determining their value and benefits, such evaluation is notoriously difficult to accomplish.  We are aware of the limitations of such research, and of the need to develop and adapt new research methods.

 

Prototype digital libraries and emerging services evolve over the course of an evaluation, and thus we are studying a moving target.  Similarly, costs associated with "bleeding-edge" applications may not adequately predict the costs of follow-on efforts.  A related concern is that solutions developed for experimental or demonstration projects may not adapt readily to large-scale implementation.  With these limitations in mind, the research begins with baseline data collection about how courses currently are being taught, and incorporates longitudinal measures of learning and instruction over a multi-year period.  Research methods will be adapted accordingly.

 

 

5.  WORKPLAN AND SCHEDULE

 

This first academic year of the education and evaluation component of the ADEPT project has been devoted to requirements analysis, evaluation design, and pilot testing.  Concurrently, the UCSB-based development team is designing the initial software architecture of ADEPT.  Baseline data collection began in September with observation of geography courses and continued throughout the year.  The fall term (September-December, 1999) was devoted to the design of instruments and the first round of human subjects review.  Winter term (January, 2000) was spent in further development of instruments, completion of the human subjects review, and refinement of the research design.  During the Spring term (April to June, 2000), prototype Iscapes are being deployed at UCSB and UCLA.   We are collecting classroom and student interview data at both campuses on a pilot basis, along with interviews of instructors.  Throughout the academic year and summer, the education and evaluation team is working with the development and implementation teams at UCSB to identify design principles and functionality.  Summer, 2000, will be devoted to data analysis, instrument refinement, software usability evaluation, and initial design of the laboratory studies.  The presentation of this paper in November, 2000, will incorporate data from the baseline studies, the first Iscape deployment, and further study design to that date.

 

Years two through five of the project will continue the usability and evaluation studies with subsequent iterations of Iscapes, in multiple classrooms in multiple disciplines.  A continuing series of research reports will be issued from the education and evaluation component of the ADEPT project.

 

 

6.  SUMMARY

 

The educational implementation and evaluation of ADEPT presents an exciting opportunity to assess the usefulness of personal digital libraries in instruction and learning.  It also enables the study of spatial metaphors as organizing principles for searching and retrieving information within a digital library.  The learner-based evaluation of the ADEPT is noteworthy for the range of methods being explored in attempt to deploy a digital library in multiple disciplines at multiple campuses, for a comprehensive study of the benefits of digital libraries in undergraduate education.

 

 

REFERENCES

 

Alexandria Digital Library.  Available at http://www.alexandria.ucsb.edu, accessed 15 Nov. 1999.

 

Allen, R. B.  1997.  Mental models and user models.  In Handbook of human-computer interaction.  Elsevier, Amsterdam: 49–64.

 

Borgman, C.L.  1986.  The users' mental model of an information retrieval system: An experiment on a prototype online catalogue. International journal of man machine studies 24:47–64.

 

Borgman, C.L.  1999. What are digital libraries? Competing visions. Information Processing & Management 35: 227-243.

 

Borgman, C.L., A. L. Gallagher, S. G. Hirsh, and V. A. Walter.  1995.  Children's searching behavior on browsing and keyword online catalogs: the science library catalog project. Journal of the American Society for Information Science 46: 663–684.

 

Borgman, C. L., S. G. Hirsh, and J. Hiller. 1996.  Rethinking online monitoring methods for information retrieval systems: from search product to search process.  Journal of the American Society for Information Science 47:568–583.

 

Bos, N., J. Karjcik, and E. Soloway.  1997.  Student publishing in a www digital library: goals and instructional support.  In P. Bell, editor, Proceedings of artifact–building in computer learning environments:supporting students' scientific inquiry.

 

California Digital Library.  Available at http://www.cdlib.org/, accessed 10 Jan. 2000.

 

Chi, M. T. H., R. Glaser, and M. J. Farr, editors.  1988.  The nature of expertise.  Hillsdale, NJ: Erlbaum.

 

Gentner, D. and A. L. Stevens, eds. 1983.  Mental Models.  Hillsdale, NJ: Erlbaum.

 

Gilliland-Swetland, Anne J., Yasmin Kafai, and William E. Landis. Forthcoming. Integrating primary sources into the elementary school classroom: A case study of teachers’ perspectives. Archivaria.

 

Halford, G. S., ed.  1983.  Children's understanding: the development of mental models.  Hillsdale, NJ: Erlbaum.

 

Hill, L.L. and others. 2000.  Alexandria Digital Library: user evaluation studies and system design.  Journal of the American Society for Information Science 51: 246-59.

 

Kieras, D. E.  1988.  What mental model should be taught: choosing instructional content for complex engineered systems.  In J. Postka, L. D. Massey, and S. A. Mutter, editors, Intelligent tutoring systems: lessons learned.  Hillsdale, NJ: Erlbaum: 85–112.

 

King, B. 1991. Beginning teachers’ knowledge of and attitudes toward history and philosophy of science. Science education 75: 135-141.

 

Marchionini, G. and G. Crane.  1994.  Evaluating hypertext and learning. ACM transactions on information systems 12:5–34.

 

Mayer, R. E.  1989.  Models for understanding. Review of educational research 59: 43–63.

 

Mayer, R. E.  1997a. From novice to expert.  In Handbook of human-computer interaction. Amsterdam: 781–796.

 

R. E. Mayer. 1997b.  Multimedia learning: Are we asking the right questions. Educational psychologist 32:1–20.

 

Meichtry, Y. J.  1992.  Influencing student understanding of the nature of science: data from a case of curriculum development. Journal of research in science teaching 29:389–407.

 

National Science Foundation.  1999.  Digital libraries initiative phase 2.  Available at: http://www.dli2.nsf.gov/ (accessed Jan. 15, 2000).

 

National Science Foundation, Division of Undergraduate Education.  2000.  National science, mathematics, engineering, and technology education digital library (NSDL). Available at: http://www.ehr.nsf.gov/EHR/DUE/programs/nsdl/ (accessed Jan. 17, 2000).

 

Neale, D.C. and J. M. Carroll.  1997.  The role of metaphor in user interface design. In Handbook of human-computer interaction.  Elsevier, Amsterdam: 441–462.

 

Norman, D.A.  1986.  Cognitive engineering.  In User centered system design.  Hillsdale, NJ: Erlbaum: 31–62.

 

O' Neill, D. K., L. M. Gomez, and D. C. Edelson.  1994.  Collaborative hypermedia for the classroom and beyond: a year' s experiences with the collaboratory notebook. In J. M. Haake, editor, Proceedings of the CSCW ' 94 workshop on collaborative hypermedia systems.  Chapel Hill, NC.

 

Scardamalia, M. and C. Bereiter.  1993.  Technologies for knowledge-building discourse. Communications of the ACM 36:37–41.

 

Scardamalia, M. and C. Bereiter.  1996. Student communities for the advancement of knowledge.  Communications of the ACM 39:36–37.

 

Wallace, R., J. Krajcik, and E. Soloway.  1997.  Digital libraries in the science classroom: an opportunity for inquiry. D-Lib Magazine (http://www.dlib.org/dlib/september96/umdl/09wallace.html)

 

Walter, D.  1996. Technologies for information access in library and information centers.  In Handbook of research for educational communications and technology. New York: Macmillan: 720–752.