Expert evaluators' naturalistic decision making during evaluations of HIV/AIDS health education programs

The goal of this research was to understand expert evaluators naturalistic decision making during evaluations of HIV/AIDS health education programs. The focus was to understand the following aspects: (1) the critical incidents that evaluators encountered; (2) the factors that influenced evaluators naturalistic decision making; and (3) how evaluators made decisions. A multiple case study research design, the Critical Decision Method, phenomenography, and narrative analysis were used to guide the methodological design and analysis of this study. Two one-hour interviews with seven participants (with ten to thirty years of evaluation experience) were conducted to develop narratives, decision making timelines, and models that captured evaluators decision making processes. Kleins (1997) and Kundins (2008, 2010) research on naturalistic decision making provided a framework for this study.The major findings in this study are threefold. First, the evaluators encountered critical incidents at various stages in the evaluation life cycle, some of which acted as barriers and others that served as facilitators for performing evaluations. These barriers included challenges in the areas of communication, community entry, coordination, contextual understanding, and evaluation design. Evaluators encountered events that undermined their ability to implement an evaluation, including data, budgetary, and time constraints, as well as resistance to evaluation. They also encountered events that facilitated their evaluations, such as successful stakeholder collaboration and sufficient time. Second, evaluators naturalistic decision making was shaped by context, evaluation, human, and real-world decision making factors. Third, evaluators relied on situation assessments to understand the situations framing the evaluations. Their philosophical assumptions, interpersonal skills, self-determination, and use of satisficing assisted evaluators during the decision-making process. Although expert evaluators attempted to anticipate barriers when implementing evaluations, they could not anticipate all the barriers they encountered. However, they adapted their evaluation strategies in response to these unexpected barriers. This reveals an opportunity to study evaluators as innovators, which may provide an insight into how evaluators innovate when they encounter unexpected barriers. Evaluators also established participatory relationships with stakeholders and engaged them in the evaluations. This is another opportunity to extend this research, thereby exploring how expert evaluators establish relationships with stakeholders.