Lies, Damned Lies, and Surveys:Â
Designing Better Surveys for
Program Evaluation and Research
Program Evaluation and Research
MacPFD Scholarly Secrets Series - with Dr. Anthony R. Artino, Jr. (@MedEdDoc)
📅December 9th, 2020
đź’» Delivered virtually
Mark Twain famously expressed his disdain for statistics when he said, “There are three kinds of lies: lies, damned lies, and statistics.” Beyond referencing his own difficulties with math, Twain was making the point that statistics can have persuasive power, even when used inappropriately. The same can be said of surveys—results from a poorly designed and poorly executed survey can have considerable persuasive power. Flawed survey results can send educators off searching for fixes to non-existent problems and fill the scientific literature with unsubstantiated knowledge claims that may never get fully corrected.Â
This workshop introduces participants to a systematic, seven-step design process for creating high-quality surveys fit for program evaluation and research purposes. The workshop consolidates and organizes the abundance of survey design literature that exists in the social sciences and guides survey developers of all levels through the design process. This design process is different from previously described processes in that it is inherently collaborative and relies on other experts in the field as well as potential survey respondents. In addition, the process front loads the task of collecting validity evidence by focusing heavily on item development. Taken together, the goal of this workshop is to help participants design better surveys; the workshop also demonstrates that Mark Twain’s lament, when applied to surveys, is often correct—but it certainly doesn’t have to be.
Learning Objectives
At the end of this virtual event, participants will be able to:
Describe how cognitive processes and motivation guide the way people understand and respond to survey questions
Recognize the elements of a survey
State several design principles and demonstrate how to use a systematic process as the framework for survey design
Identify poorly written survey items and other design pitfalls with assistance of a survey review checklist
Define the purpose of expert validation, cognitive interviews, and pilot testing
Suggested Readings
Guest Faculty
Dr. Anthony R. Artino, Jr. (@MedEdDoc) is tenured professor and Interim Associate Dean for Evaluation and Educational Research at The George Washington University School of Medicine and Health Sciences in Washington, DC. He received his PhD in educational psychology from the University of Connecticut and served 23 years in the U.S. Navy, retiring as Captain in 2020. Dr. Artino is a member of several editorial review boards, and he has published widely in medical and health professions education.Â
Co-FacilitatorÂ
Dr. Teresa Chan (@TChanMD) is an associate professor in the Division of Emergency Medicine, Department of Medicine at McMaster University. She is the assistant dean for McMaster Faculty of Health Sciences Program for Faculty Development (@MacPFD). She is an avid scholar in health professions education and works with the MERIT group (@MERIT_McMaster), and conducts research and scholarship within this area.Â