Epistemic Network Analysis: Assessment for the Digital Age

David Shaffer

Digital media create new models, approaches, and techniques for learning — as well as new educational outcomes, goals, and needs. Thus digital education creates new challenges for assessment.

Digital learning environments emphasize learning in action. Games, simulations, and other digital tools help learners understand phenomena by working with them from the start in complex situations rather than by first mastering isolated facts and skills and later assembling these conceptual building blocks to solve more elaborate, more complete, more realistic, and more sophisticated problems. In such environments, mastery of basic facts and skills are not an effective measure of expertise. Therefore, useful and meaningful assessments need to focus on performance in context rather than on tests of abstracted and isolated skills and knowledge.

Fortunately, games also provide the potential to assess performance in context, because digital tools make it possible to record rich streams of data about learning in progress. Hardware and software can record the actions that players take in a digital medium as well as interaction between peers and between learners and mentors. But what assessment methods will use this data to measure mastery of complex problem solving — thinking in action — rather than mere remembering of isolated facts and skills?

This talk looks at one way to address this challenge through an approach to assessment known as evidence centered design. Evidence centered design is a framework for developing assessments by systematically linking models of understanding, observable actions, and evaluation rubrics to provide evidence of learning. Central to the concept of evidence centered design is the idea of alignment between learning theory and assessment method, between evidence and hypothesized mechanisms of thinking and learning in a given arena.

The talk examines how evidence centered design can address the challenge of assessment in new media learning environments by presenting one specific theory-based approach to digital learning, known as epistemic games, and describing a method, epistemic network analysis, to assess learning based on this theory. The learning theory is supported by empirical studies, and the assessment method we describe has produced useful results in analyzing learning outcomes. However, the goal here is not to argue for the validity of epistemic games or to promote epistemic network analysis. Rather, the talk uses the theory and its related assessment method to illustrate the concept of a digital educational system: a system composed of a theory of learning and its accompanying method of assessment, linked into an evidence-based, digital intervention.