QUERI Implementation Guide
Section I Part 4: Formative Evaluation
References
1. Isaac S, Michael, W. (1982). Handbook in research and evaluation: For education and the behavioral sciences. San Diego: EdITS publishers, 1982.
2. Hulscher M, Laurant M, Grol R. Process evaluation on quality improvement interventions. Quality and Safety in Health Care 2002;12:40-46.
3. Dehar MA, Casswell S, Duignan P. Formative and process evaluation of health promotion and disease prevention programs. Evaluation Review 1993;17:204-220.
Selected References: Formative Evaluation for Quality Improvement Interventions
Evaluation Practice and Methods: Books
Fink A. (1993). Evaluation Fundamentals: Guiding Health Programs, Research and Policy. Newbury Park: Sage Publications, 1993.
Isaac S and Michael W. (1982). Handbook in Research and Evaluation: For Education and the Behavioral Sciences. San Diego: EdITS publishers, 1982.
Miles MB and Huberman AM (eds). (1994). Qualitative Data Analysis, 2nd ed. eds. Sage Publications, 1994.
Patton MQ. Utilization-Focused Evaluation: The New Century Text. (3rd ed.), Sage Publications, 1997.
Patton MQ. Qualitative Research and Evaluation Methods, Newbury Park: Sage Pubs, 2001.
Rogers EM. Diffusion of Innovations, (4th ed.), New York: Free Press, 1995.
Rossi PH, Freeman HE, Lipsey MW. Evaluation: A Systematic Approach. (7th ed.) Newbury Park, Sage Publications, October, 2003.
Wholey J, Hatry H, Newcomer, K. (Eds). Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishers, 1994.
Yeaton W and Camberg L. Program Evaluation for Managers (Primer). Boston: MDRC (HSR&D), 1997. http://www.hsrd.research.va.gov/publications/internal/progeval.pdf
Evaluation Practice and Methods: Articles
Dehar MA, Casswell S, Duignan P. Formative and process evaluation of health promotion and disease prevention programs. Evaluation Review 1993;: 17:204-220.
Devers KJ, Sofaer S, Rundell, TG. (eds.) Qualitative methods in health services research: A special supplement to Health Services Research. Health Services Research 1999,; 34 (5) Part II December 1999: 1083-1263.
Effken E and Stetler, C. Impact of organizational redesign. Journal of Nursing Administration 1997;27:23-32.
Harvey G, and Wensing M. Methods for evaluation of small scale quality improvement projects. Quality and Safety in Health Care 2003;12:210-4.
Hulscher M, Laurant M, Grol R. Process evaluation on quality improvement interventions. Quality and Safety in Health Care 2002;:12:;40-46.
Kirchhoff K, and Dille C Issues in intervention research: maintaining integrity. Applied Nursing Research 1994;7:32-38.
Quinn D. Proving the effectiveness of community programs in jeopardy: The role of formative evaluation. Quality & Management in Health Care 2001;9(4):433-46.
Stange K, and Zyzanski S. Integrating qualitative and quantitative research methods. Family Medicine. 1989; Nov-Dec;21(6):448-51.
Stead M, Hastings G, Eadie D. The challenge of evaluating complex interventions: A framework for evaluating media advocacy. Health Education Research 2002;:17(3):351- 364.
Stetler C, Creer E, & Effken J. (1996). Evaluating a redesign program: Challenges and opportunities. Series on Nursing Administration, Vol. 8. Ed. by Kathleen Kelly, St. Louis: Mosby Year Book, 1996.
Walshe K, and Freeman T. Effectiveness of quality improvement : Learning from evaluations. Quality and Safety in Health Care 2002;:11:;85-87.
Examples of Health-Related Process or Formative Evaluation
Brink SG, Birney AJ, McFarren AE. Charting your course: Formative evaluation of a prostate cancer treatment decision aid. The International Electronic Journal of Health Education 2000;3(1):44-54.
Cabral RJ, Galavotti C, Gargiullo PM, Armstrong K, Cohen A, Gielen AC, Watkinson L. Paraprofessional delivery of a theory based HIV prevention counseling intervention for women. Public Health Reports. 1996;111 Suppl 1:75-82.
Ekundayo OT, Bronner Y, Johnson-Taylor WL, Dambita N, Squire S. Formative evaluation of a men's health center. American Journal of Public Health 2003; May;93(5):717-9.
Fisher LR, Solberg L, Zander K. (2001). The failure of a controlled trail to improve depression care: A qualitative study. Joint Commission Journal on Quality Improvement 2001:27(12):639-650.
Forsetlund L, Talseth KO, Bradley P, Nordheim L, Bjorndal A. Many a slip between cup and lip. Process evaluation of a program to promote and support evidence-based public health practice. Evaluation Review 2003;Apr;27(2):179-209.
Goldberg H, Wagner E, Fihn S, et al. A randomized controlled trial and CQI teams and academic detailing: Ccan they alter compliance with guidelines? Joint Commission Journal on Quality Improvement 1998;24:130-141.
Mahloch J, Taylor V, Taplin S, Urban N. A breast cancer screening educational intervention targeting medical office staff. Health Education Research. 1993; Dec;8(4):567-79.
Nazareth I, Freemantle N, Duggan C, Mason J, Haines A. Evaluation of a complex intervention for changing professional behaviour: the Evidence Based Out Reach (EBOR) Trial. Journal of Health Services Research & Policy 2002;7:230-8.
Orwin RG. Assessing program fidelity in substance abuse health services research. Addiction. 2000;Nov;95 Suppl 3:S309-27.
Rubenstein LV, Parker LE, Meredith LS, et al. (2002). Understanding team-based quality improvement for depression in primary care. Health Services Research. 2002; Aug;37(4):1009-29.
Stetler, C., Corrigan, B., Sander-Buscemi, K., & Burns, M. Integration of evidence into practice and the change process: A fall prevention program as a model. Outcomes Management for Nursing Practice, 1999;3:102-111.
Web-based resources related to evaluation
Please note that links to these sites are not endorsements of the site, the organization, or content on those sites. They are provided to assist you in identifying potentially useful information, ideas, or additional resources.
US Government Resources
CDC Evaluation Working Group website (http://www.cdc.gov/eval/index.htm) offers information about the work group, a framework for program evaluation, and an extensive resource listing (http://www.cdc.gov/eval/resources.htm).
The National Science Foundation’s Directorate for Education and Human Resources, Division of Research, Evaluation and Communication has a web published User-Friendly Handbook for Mixed Method Evaluations . While the examples and content are related to education and learning evaluations, the handbook has information related to evaluation that can be applied to other settings. Other features include an example evaluation plan, tips for analyzing qualitative data, and example materials – such as example observation guides, interview guides, and so forth.
The Bureau of Justice Assistance is committed to the importance of program evaluation and to developing and enhancing evaluation capabilities at the state and local levels. Evaluation results provide policy makers and program managers with information for future program development and can be used to modify and improve existing programs. The Evaluation Web site (http://www.ojp.usdoj.gov/BJA/evaluation/) is designed to provide State Administrative Agency staff, criminal justice planners, researchers and evaluators, as well as local practitioners with a variety of resources for evaluating criminal justice programs and has a page with links to a variety of evaluation resources (http://www.ojp.usdoj.gov/BJA/evaluation/evaluation-resources.htm).
Other Government Resources
Human Resources Development Canada offers an example of a formative evaluation of a social program. As part of this project they developed an evaluation toolkit . The section on Quasi Experimental Evaluation offers guidance on the issues in quasi-experimental designs, which are commonly used, as well as handling threats to validity and may offer some guidance to those who plan these designs.
Non-Government Resources
The American Evaluation Association (http://www.eval.org/) is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. The site includes Guiding Principles for Evaluators, meetings and events related to evaluation and links to resources for evaluators, including a listing of online texts and books with "how tos" related to evaluation (http://www.eval.org/Publications/GuidingPrinciples.asp).
RE-AIM (http://www.re-aim.org) is a systematic way for researchers, practitioners, and policy decision-makers to evaluate health behavior interventions. It can be used to estimate the potential impact of interventions on public health. The group is affiliated with Kansas State University, and the Robert Wood Johnson Foundation has provided funding for the workgroup and for developing the website. RE-AIM stands for: Reach into the target population; Efficacy or effectiveness; Adoption by target settings or institutions; Implementation—consistency of delivery of intervention; Maintenance of intervention effects in individuals and populations over time.
Resources for Methods in Evaluation and Social Research (http://gsociology.icaap.org/methods/) is a website supported by ICAAP (The International Consortium for the Advancement of Academic Publication) and lists free resources for methods in evaluation and social research. The focus is on "how-to" do evaluation research and the methods used: surveys, focus groups, sampling, interviews, and other methods. Most of these links are to resources that can be read over the web. A few, like the GAO books, are for books that can be sent away for, for free (if you live in the US), as well as read over the web.
The Action Evaluation Research Institute (http://www.aepro.org/ - site currently unavailable 09/12/2006) is a site with information on action research and evaluation.
Formative Evaluation Research Associates (FERA) (http://www.feraonline.com/) is an evaluation group that has 25 years experience with non-profit organizations. The site includes general information on formative evaluation as well as links to other resources.
The Skillman Foundation's website has an evaluation guide (http://www.skillman.org/) that is directed at their grantees or those applying, but which also provides a good overview on evaluation.
The WK Kellogg Foundation has several guides that relate to evaluation, and evaluation guide (http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf), and a guide to the use of logic models to guide program implementation as well as the ensuing evaluation (http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf).