Contents lists available at ScienceDirect

Evaluation and Program Planning

1. Introduction and background

The right of people with disabilities to be full participants in their communities includes the informed choice to be involved in research and program evaluation both as members of methodological design teams and as study participants (Johnson, 2009; Watson, Feiler, & Tarleton, 2014). However, people with disabilities, particularly people with intellectual and developmental disabilities (IDD), have often been excluded from meaningful participation in research studies(Feldman, Bosett, Collet, & Burnham‐Riosa, 2014), due to inclusion criteria and consent processes required by institutional review boards (IRB), lack of accommodations to enable participation, and screenings to assess for capacity to consent that excludepotential participants (Cameron & Murphy, 2006; McDonald et al., 2009; McDonald & Patka, 2012). From an ethics standpoint, researchers and program evaluators have grappled with balancing the right to self-determination for people with IDD to participate with issues including recruitment strategies that may be perceived as coercive, concerns about whether some potential participants are able to comprehend information, and capacity to consent (McDonald & Kidney, 2012; McDonald & Patka, 2012). While respectful deliberation of potential ethical considerations for including people with IDD in studies is likely warranted, paternalistic views that influence researchers to exclude people with IDD are being challenged, and researchers in the field are calling for "methodologies that equate respect for autonomy with the right to take risks" (McDonald, Kidney, & Patka, 2013, p. 217). Furthermore, studies of people with IDD have demonstrated that they have a desire to participate, to share their opinions, and to be assured that the results of their participation in the study will improve their lives and others with IDD (Kidney & McDonald, 2014; McDonald et al., 2013).

A key component for full participation of people with IDD in research and program evaluation is the provision of supports or accommodations that help them to understand the study, to make a decision about participation, and to fully participate in the study itself. While there is quite a bit of literature on adapting consent processes to enable participation of people with IDD (for example, Inclusive Research Network, 2010; Kidney & McDonald, 2014), there isn't as much literature on the adaptation of research measures and their administration that allow for greater understanding and participation of people with IDD in research and program evaluation studies. Data collection instruments that are not understood by participants with IDD are also of concern to researchers and program evaluators who are assessing the efficacy of interventions. If there is question as to whether some participants understood evaluation items, then the validity of the instruments and the results are in question. Instruments that predominantly rely on either written or oral communication may be inaccessible to some people with IDD (Boxall & Ralph, 2009). An emerging strategy in research with various populations, including people with IDD, is to use video vignettes for evaluative purposes. This article will present an overview of video vignettes as a research and program evaluation methodology and highlight the use of video vignettes in Leadership Empowerment for Abuse Prevention (LEAP), a community-based abuse prevention program for people with IDD.

1.1. Video vignettes as research methodology

Vignettes are hypothetical narratives designed to illustrate, elicit, or teach (Alexander & Becker, 1978, as cited in Oremus, Xie, & Gaebel, 2016; Jobe & Glidden, 2008). They are used in research to approximate real-life situations and to provide windows into the actual behavior of study participants and the populations they represent. Vignettes have been used to study a wide variety of topics including: doctor-patient interactions (Hillen, van Vliet, de Haes, & Smets, 2013) stigma associated with disability (Barg, Armstrong, Hetz, & Latimer, 2010), and mental health first aid (Davies, Wardlaw, Morriss, & Glazebrook, 2016). The utility of vignettes is derived from the ability to expose participants to situations that would be otherwise difficult or impossible to create (Jobe & Glidden, 2008), as well as the greater ease with which researchers can manipulate key elements of those situations. The purpose of using vignettes is to evoke responses from participants that mimic how they would respond in real-world situations (Stacey et al., 2014).

Both the advantages and disadvantages of vignettes stem from their close proximity to, but not exact replication of, real-life experience. By simulating situations, researchers avoid ethical challenges and protect participants from risk. For example, researchers often use vignettes as proxies to investigate participants' attitudes and behaviors (Oremus et al., 2016) and, in this manner, social desirability bias as well as the social and psychological risk of a study is reduced (Desautels & Jacob, 2012; Jobe & Glidden, 2008). At the same time, because vignettes are controlled and constructed versions of real life, their use poses threats to both the internal validity and generalizability of research (Martinez et al., 2014). The staged and reduced aspects of vignettes can result in diminished ecological validity as participants recognize the scenario depicted is less urgent than real life (Hillen et al., 2013).

By removing ethical concerns and feasibility challenges, vignettes expand research design choices. Vignettes also facilitate the use of experimental design (Overmeer, Boersma, Main, & Linton, 2009) and permit wider participation in research than would be feasible using direct observation of events (Jobe & Glidden, 2008). In part, this flexibility is derived from the standardized nature of vignettes – whether written or audiovisual in form – a vignette is self-contained and therefore portable across distance. Lastly, by engaging participants in hypothetical narratives, vignettes reduce participant risk (Hillen et al., 2013).

Several elements of vignettes, and video vignettes in particular, make them advantageous for research with people with IDD. Vignettes create social distance and allow discussion of topics that may otherwise seem invasive and overly sensitive, such as abuse prevention especially for people with communication challenges (Jobe & Glidden, 2008). Additionally, when combined with video technology, vignettes reduce barriers associated with written and verbal communication skills. Boxall and Ralph (2009) suggest that creative use of visual images is an effective way to engage those with IDD and other populations who otherwise are excluded by the research community. In summary, video vignettes provide a common, and standardized, experience to which participants can respond without relying heavily on written or verbal communication skills.

1.2. Vignette development and deployment strategies

The development of vignettes is an area of increasing scholarship. Since vignettes are designed to approximate real-life situations, researchers have underscored the importance of gathering input and feedback from experts in the field as well as those with direct experience to increase content validity (Martinez et al., 2014; Oremus et al., 2016; Stacey et al., 2014). These different sources of input ensure that the vignettes incorporate both first-hand/internal ("emic") and observer/external ("etic") perspectives (Oremus et al., 2016). Later in the development and refinement process, pilot testing with stakeholders is useful to establish face validity. Other common elements in vignette development include identifying key elements of the phenomenon under study, selecting a scenario or scenarios to serve as context for the vignette(s), drafting scripts, and determining whether and how to manipulate the vignette(s) to achieve a basis of comparison (Hillen et al., 2013; Martinez et al., 2014).

Vignettes are used to assess different categories of outcomes. Hillen et al. (2013) found that vignettes are used to measure differences in evaluative outcomes (such as preferences for certain styles of communication), cognitive outcomes (e.g., retention of content), affective outcomes (distress or other physiological responses), and behavioral outcomes (how would you respond if…?). These categories can be combined, as when Overmeer et al. (2009) used a pre/post design to assess both behavior and cognitive outcomes associated with a physical therapist continuing education course in psychosocial sensitivity. Vignettes are also considered a valid and relatively feasible approach to measuring compliance with professional guidelines and other normative behaviors (Martinez et al., 2014).

Given that video vignettes have shown promise for measuring outcomes with various populations in intervention studies, we will now demonstrate how they can be used in community-based program evaluation through a case study of LEAP, an abuse prevention program for people with IDD. Over the course of an 18 month period, LEAP evaluators pilot-tested and refined instrumentation to measure the efficacy of the intervention. This refinement culminated in the use of video vignettes as a primary mechanism to measure participant outcomes.

2. Case study: leadership for empowerment and abuse prevention

2.1. Description of LEAP

People with disabilities are at an increased risk of violence when compared with peers in the general population (Beadle‐Brown, Mansell, Cambridge, Milne, & Whelton, 2010; Harrell, 2012; Hughes et al., 2012). Further, having IDD puts a person at an even greater risk, with some prevalence estimates for experiencing a form of abuse during one's lifetime at a high of 90% for women and up to 86% for men (Hughes, Lund, Gabrielli, Powers, & Curry, 2011; Sullivan & Knutson, 2000). While awareness, prevention, and intervention programs have been developed to address the risks that people with disabilities face, a recent review of the literature on violence prevention programs for people with disabilities concluded that there is "little clear guidance for practitioners…faced with selecting such interventions" and there is an "urgent need in this area for research of higher quality" (Mikton, Maguire, & Shakespeare, 2014, p.16).

The dearth of existing curricula for people with IDD about abuse prevention and how to identify and promote healthy relationships, especially for those with more moderate to severe disabilities, was the impetus for the creation of the LEAP curriculum. In the past decade, several violence/abuse prevention curricula have been developed with a focus on people with disabilities (e.g., Our Whole Lives, Curriculum on Abuse Prevention and Empowerment, SAFE, Respect Yourself, S.T.A.R.S., ESCAPE-DD). However, most are designed for people with mild IDD and require extensive time commitments over multiple months.

The LEAP curriculum is a four module training (90 min per module) and is based on primary prevention principles (Armstead et al., 2017). The curriculum provides introductory information and supports taking an action to identify and avoid potentially dangerous relationships. Each session reinforces the concepts taught in the previous session and reiterates a power statement, "I am strong. My feelings are important. I deserve to feel safe. I deserve respect." To date, over 450 people have participated in the training. Table 1 lists key concepts taught in each session.

Since people with IDD learn best when ideas and concepts are reinforced repeatedly, the training relies heavily on repetition (Archer & Hughes, 2010). It also incorporates techniques of Universal Design for Learning, a set of principles for curriculum development that give all individuals equal opportunities to learn (CAST, 2015). Accommodations are available for participants with significant disabilities who do not communicate verbally. An Implementation Manual (How-To Guide for Trainers) complements the curriculum and establishes protocols for trainer implementation fidelity. The guide provides scripts for each area of content, specific instructions for how the training is to be delivered, provides direction on participant engagement, and describes the main points to emphasize. It encourages consistency in training across sites and trainers and supports a unique feature of the LEAP training program, the use of a training approach that includes two trainers, one with and the other without a disability.

As part of their preparation, trainers complete 16 hours of training, both as part of a train-the-trainers day as well as by conducting real-life practice training sessions. Trainers are also instructed by violence prevention experts on how to respond when participants disclose that they are victims of abuse (as this has been reported by trainers in over 50 % of the LEAP sessions). In the development of the curriculum and in the training of trainers, it was essential to consult with experts to make sure that appropriate actions were taken.

2.2. Development of the LEAP vignettes

Reviews of literature have documented the lack of rigorously evaluated programs and studies to address abuse against people with disabilities (Barger, Wacker, Macy, & Parish, 2009; Dryden, Desmarais, & Arsenault, 2017; Mikton et al., 2014). As highlighted earlier, this may be largely due to the complexity of the target population and the challenges of developing appropriate instrumentation and methodologies (Dryden et al., 2017).

Several recent articles have identified promising evaluative practices for prevention programs for people with IDD (see Barger et al., 2009; Doughty & Kane, 2010; Lund, 2011; Mahoney & Poling, 2011; McEachern, 2012) and a few studies have increased the rigor of their evaluations significantly (see Dryden et al., 2017; Hickson, Khemka, Golden, & Chatzistyli, 2015; Robinson-Whelen et al., 2014). However, the majority of the studies have limited their focus to the prevention of sexual abuse of women with IDD; had small sample sizes, excluded people with more significant disabilities, and conducted limited follow-up (Barger et al., 2009; Doughty & Kane, 2010; Hickson et al., 2015; Lund, 2011; Mahoney & Poling, 2011; McEachern, 2012). The LEAP research sought to address this gap in the literature.

LEAP evaluation measures were focused in two areas: 1) developing protocols and monitoring tools for the LEAP to assess trainer fidelity of implementation and 2) evaluating participant outcomes. For establishing a trainer fidelity of implementation protocol, we adapted a four-step process developed by Fixsen, Naoom, Blasé, Friedman, and Wallace (2005)) which included: 1) identifying the critical components of LEAP; 2) identifying steps during the implementation process that co-trainers must follow when presenting the curriculum to participants; 3) developing an observational protocol that reflects these components and instructional steps; and 4) testing and refining the protocol to examine feasibility, usability, and reliability (across observers). Third-party observers piloted the implementation fidelity protocols in 60 LEAP sessions (15 series of 4 sessions). Results from initial sessions showed that multiple trainers had challenges with fidelity. These trainers were provided with coaching and technical assistance. As trainers became more familiar with the protocol, they made significant progress in achieving fidelity to the LEAP curriculum as measured by the fidelity checklist and observation.

For measuring participant outcomes, a "paper-pencil" pre-and post-test instrument was drafted based on the core components of the LEAP curriculum. The instrument was administered in a one-on-one interview format in over 60 LEAP sessions. As part of this process, seven versions of the pre- and post-test tool were tested, each version incorporating recommended modifications from previous piloting. While trainers felt that participants understood key concepts and were engaging with the content during the LEAP training sessions, many attendees had trouble successfully completing the pre- and post-tests and reported anxiety and concern with the paper-and-pencil-based, "testing" format.

Incorporating what was learned from piloting the multiple paper-and-pencil versions of the pre- and post-test tool, the project team developed an additional measure to evaluate the extent to which participants were generalizing information from LEAP trainings. The tool, administered by an evaluator three months after training, was vignette-based and presented in video format on an iPad. Each vignette, based on core components of the curriculum, portrays a scenario to participants and they determine if it illustrates a "healthy" or "unhealthy" relationship/situation. In addition, participants are asked to explain why they made this determination and, if it was "unhealthy," what should be done next. As advocated by experts (see Martinez et al., 2014; Oremus et al., 2016; Stacey et al., 2014), the vignettes were scripted out based on the core components of the curriculum and reviewed by disability professionals, experts in the violence prevention field, and by stakeholder advisors, who were people with disabilities. The reviews addressed content validity and sensitivity to discomfiting subject matter. Once the videos were recorded, they were once again vetted. Additionally, each vignette was independently ranked by three reviewers by level of complexity so a minimum of two straightforward, moderate, and complex items were included in the final instrument. The final video-based instrument was piloted with over 30 LEAP participants. The storyline for each vignette appears in Table 2 below.

3. Lessons learned

3.1. Evaluators' feedback about vignettes

The administration of the video vignettes by the evaluation team provided insight into what was successful and challenging with the vignettes and their implementation. Each of the video vignettes consisted of a scenario that featured actors with and without disabilities illustrating a healthy or unhealthy relationship. The vignette evaluation was administered individually to each participant and lasted approximately 15−20 min. Each of the six vignettes used in the evaluation was approximately 1 min and was played twice. After each vignette, participants were asked to respond by communicating (e.g., orally, sign, gesture) "yes" it is a healthy relationship or "no" it is not a healthy relationship. Participants were then asked to elaborate on their responses by answering "why" the relationship in the vignette was healthy or unhealthy and what action could be taken if the relationship was unhealthy. In general, participants who used alternative communication devices took longer than 15 to 20 min to respond to the open-ended questions.

Successes of the video vignette evaluation included the ability of the evaluator to adjust the accessibility of the vignette based on participant's needs and provide visual (i.e., video, & captioning), auditory (i.e., narration), or tactile engagement (i.e., touching the play button). The format was successfully administered with participants with concomitant visual, hearing, and mobility impairments. The video vignette format also engaged and sustained the participants attention during the evaluation process. Challenges with the video vignette evaluation revealed participants were sometimes confused by the nature of the relationships in the vignette. The videos were revised when a pattern of error responses demonstrated that a LEAP concept was not clearly illustrated in the vignette or that extraneous details confused respondents that were not the focus of the vignette.

The following examples illustrate the nature of the modifications that were made based on feedback from piloting. In one vignette, a bathroom door that was left ajar was a concern to respondents instead of the nature of the relationship being depicted between the personal care attendant and the person with the disability. We edited the door out of the vignette. In another instance, participants were concerned with a person who had a cellphone at the dinner table (they felt it was rude and therefore unhealthy), rather than focusing on the relationship depicted in the vignette. In an additional example, a vignette showed a physically ill actor being cared for by a personal care attendant. The question was posed if it was a "healthy" or "unhealthy" relationship. The majority of participants responded that it was "unhealthy," due to the actor's illness rather than it was "healthy" as shown by the personal attendant's care and concern. The revised vignettes focused on LEAP concept illustration while still portraying some of the ambiguity found in real life situations.

Another challenge that was encountered when implementing the LEAP vignettes was the heterogeneity in levels of comprehension among the participants. There were observable, significant differences in the understanding of respondents based on their support needs. In order to acknowledge these differences and account for them in our analysis, we developed a participant demographic information sheet that we collected from agencies and family members to record people's level of disability and other demographics. Table 3 outlines the items collected in the demographic sheet.

From an implementation standpoint, while the one-on-one administration of the tool proved to be critical in understanding the accommodation needs of participants and in probing for understanding, it is a relatively resource-intensive model of data collection. As highlighted earlier, the vignettes take approximately 15−20 min to administer and were staffed at a one-to-one ratio. However, if we are striving to better understand outcomes of interventions for complex populations, we needed to acknowledge and plan for adequate time and resources for the most efficacious data collection designs.

3.2. Recommendations for others

Through the development and piloting of the LEAP video vignettes, several key strategies emerged that may offer utility to other researchers or program evaluators who are engaging people with IDD in their work. The initial process of working with curriculum developers to identify the key concepts within the modules to form the basis for each vignette storyline was vital for ensuring that each storyline addressed the most salient curriculum topics. Additionally, while time consuming, the extensive piloting of the vignettes yielded feedback that significantly improved the accessibility of the tool and helped to strip out extraneous details in the vignettes that led to confusion. Finally, having both closed and open-ended questions as response items provided program evaluators with much more nuanced data to determine the degree of comprehension and to pinpoint specific instructional elements that were particularly challenging for respondents and how their thinking pattern influenced their answer.

4. Conclusion

Many people with IDD desire to be involved in research as participants and research teams should consider ways to maximize participation of people with IDD in their studies. Particularly for sensitive topics such as abuse prevention, the use of video vignettes for program evaluation provides an inclusive, accessible, and effective method for participants to demonstrate their learning compared to traditional paper-and-pencil test formats. Paying careful attention to designing video vignettes that are both clearly connected to the core concepts of a curriculum and that are void of extraneous details that may confuse research participants are two of the primary recommendations for others who intend to design their program evaluations to be more inclusive of people with IDD.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

The funder is the US Department of Justice – Office on Violence Against Woman Grant number: OVW-2017-11909.

Appendix A. Supplementary data

Supplementary material related to this article can be found, in the online version, at doi:https://doi.org/10.1016/j.evalprogplan.2019.101774.

References

Parthenia Dinora, pH.D., has worked in the disability field for over 24 years and currently serves as the Associate Director for the Partnership for People with Disabilities at Virginia Commonwealth University, Virginia's University Center for Excellence on Intellectual and Developmental Disabilities (IDD). Her research interests/scholarship focus on interventions that support community living for people with IDD and quality assurance for IDD services and supports. She also has conducted several studies and published research on the factors that influence decision making for parents who have young children diagnosed with Autism.

Andrew Schoeneman serves as Assistant Professor and Chair of Nonprofit Studies in the School for Professional and Continuing Studies at the University of Richmond. His scholarship is in the areas of nonprofit organizations and the relationships between agencies and the communities they serve. Through examination of legal aid programs, Muslim nonprofits, disability rights agencies, and the nonprofit sector itself, he explores how organizations build rapport and power alongside affected populations. Before taking on an academic role he spent 15 years in the fields of community organizing, affordable housing, and racial reconciliation.

Molly Dellinger-Wray, M.S. Ed. is part of the Partnership for People with Disabilities, a University Center for Excellence in developmental disabilities. She is endorsed as a Positive Behavior Support Facilitator and a certified trainer of Person Centered Thinking. She directs three projects that assist children and adults with disabilities with issues surrounding health, violence, abuse, sexual assault, and neglect.

Elizabeth "Liz" Paula Cramer, PhD, LCSW, ACSW, is a Professor in the Virginia Commonwealth University School of Social Work. She teaches courses in direct practice, diversity and social justice, interpersonal violence, and a doctoral level course on social work education and teaching. Her primary research/scholarship areas are domestic violence, particularly abuse of persons with disabilities, and educational strategies to impact social work students' attitudes toward, knowledge of, and skills with LGBTQ populations. She has worked in the domestic and sexual violence field for over 35 years, including as a direct service provider, administrator, researcher, board member, and educator.

Jack Brandt, M.S. is the Disability Policy Specialist at the Partnership for People with Disabilities at Virginia Commonwealth University, a position he has held since 2010. Mr. Brandt was a Disability Rights Advocate at the Virginia Office for Protection and Advocacy from 2008 to 2010. Mr. Brandt was appointed as a member of the President's Committee for People with Intellectual Disabilities in 2010 and was later appointed Chairman in 2016. In 2016, he was elected as Co-Chair of the Association of University Centers on Disabilities' (AUCD) Council on Leadership and Advocacy, a position in which he still serves.

Allison D'Aguilar PhD serves as a research assistant on the LEAP project for the Partnership for People with Disabilities. Her research interests include quality of life issues across the life span including health and wellbeing, financial accessibility and capability, and aging for adults with intellectual and developmental disabilities. Her research approach includes community based research and participatory action research. Previously, Allison served as a special educator in the public school setting for twenty years.

https://doi.org/10.1016/j.evalprogplan.2019.101774
Received 7 February 2019; Received in revised form 19 August 2019; Accepted 14 December 2019
Available online 20 December 2019
0149-7189/ © 2019 Elsevier Ltd. All rights reserved.

^⁎ Corresponding author.
E-mail addresses: padinora@vcu.edu (P. Dinora), aschoene@richmond.edu (A. Schoeneman), mdwray@vcu.edu (M. Dellinger-Wray), ecramer@vcu.edu (E.P. Cramer), brandtj@vcu.edu (J. Brandt), daguilara@vcu.edu (A. D'Aguilar).

Evaluation and Program Planning 79 (2020) 101774