Tiny House
More About The National Coalition for Dialogue & Deliberation • Join Now!
Community News

Report to the Deliberative Democracy Consortium: Building a Deliberation Measurement Toolbox

This project was charged with creating a toolbox of measures for evaluating democratic deliberation, a toolbox of use to practitioners and researchers of deliberation. With a couple exceptions, there are few measures of the consequences or quality of deliberation with a proven record of detecting effects or quality. Indeed, some observers have suggested that it is unlikely researchers will be able to detect most effects of deliberation, in part because the effects may be small and require repeated deliberation experiences. In an encouraging sign, this 2006 report introduces a set of measures that does detect strong effects of deliberative experiences, even in one-day deliberations with relatively few participants.

In addition to the value such measures may have to individual researchers and practitioners, a toolbox of measures may also have community-wide benefits. If researchers and practitioners could agree on a set of measures to consistently use in evaluating deliberations, the findings would be comparable across these deliberations-allowing researchers and practitioners to make inferences about what features of these deliberations have various effects-what works and what does not. In addition to describing a promising toolbox of measures, this report also introduces a theoretical framework that makes sense of the findings regarding these measures and may help researchers and practitioners better focus their efforts to understand deliberation. Finally, it makes a variety of practical suggestions regarding how practitioners can improve their efforts to demonstrate the value of deliberations. A few methodological rules could, if followed, greatly enhance these efforts.

A deliberation measurement toolbox project is potentially vast. Volumes of material currently exist regarding possible methods of assessment that could be pertinent to deliberation. Deliberation itself consists of an array of social processes whose measurement spans the gamut of possibilities in the social sciences, a vast space indeed. Any attempt to comprehensively catalog possible measures and methods for the study of deliberation would yield too much material to be practical. In addition, other projects are already underway to catalog and recommend measures for practitioners based on many past evaluative efforts by practitioners. In an examination of a number of such practitioner survey instruments for possible inclusion in this Toolbox Project, I concluded that most survey questions employed by practitioners fall short of standards of clarity and precision in social science research and do not appear to have well-elaborated theoretical underpinnings. A rigorous measurement toolbox would need to steer clear of such questions.

What researchers and practitioners should get out of this report:

  • Tested questions (“instruments”) for evaluating deliberations. These are questions that have shown promise for detecting effects of deliberation.
  • A theoretical framework that helps explain the larger significance of the questions and suggests directions for further examining deliberation effects.
  • Practical advice on how to go about rigorously establishing the effectiveness of deliberation and guidelines for how to construct your own survey questions. Practitioners will no doubt want to develop their own questions, in addition to considering the ones suggested here.

This report will begin with the theoretical framework, which helps clarify the questions introduced later. The theory section will explain what core questions are being asked, offer a definition of deliberation, and introduce a theoretical framework called agency theory. The next section will offer some practical advice on how to rigorously establish the effectiveness of deliberation and guidelines for constructing your own survey questions. The third section will offer a variety of survey questions, explain their purpose, and mention how they have been tested.

This report introduces a theoretical framework, agency theory and the related parochial citizens’ thesis, for approaching deliberation research and understanding its practice. It introduces suggestions for how practitioners can improve their efforts to demonstrate the value and efficacy of deliberation, both in terms of how they design such efforts and in how they should write questions for surveys contributing to these efforts. Finally, it presents a variety of measures, many of which show promise for researchers and practitioners.

It is the author’s hope that these measures will with time gain wide use. If researchers and practitioners could consistently use a fairly common set of indicators for deliberative consequences and quality and report their experiences, this would contribute greatly to the accumulation of knowledge of deliberation. A repository of results from different deliberations using the same measures would help researchers and practitioners identify what features of deliberation contribute to given outcomes. Muhlberger encourages readers who use the scales suggested here to contact him so he can begin to create a repository of experiences with these scales.

This Toolbox Project was funded by the Hewlett Foundation through the Deliberative Democracy Consortium and AmericaSpeaks. The Virtual Agora Project portion of this research was funded by the National Science Foundation under Grant No. EIA-0205502.

Peter Muhlberger (2006)

Resource Link: http://geocities.com/pmuhl78/DDCReport.pdf

Also available to download here, in case the original document is moved: http://ncdd.org/rc/wp-content/uploads/Muhlberger-DelibMeasurementToolbox.pdf

  More Resources  

Add a Comment