At best, evaluation can make sense of the complex processes underlying programmes by formulating plausible explanations ex-post. In order to answer that question, realist evaluators aim to identify the underlying generative mechanisms that explain ‘how’ the outcomes were caused and the influence of context. h�b``�d``�����:�A��b�, Bs�30��4,�C�f`8��’zF6r�Ѣ��қ^$���u($�(� ���� a*�D�d�E iI ��+� ���x�ə����0�,�{�F�O���e��1x0��N�gx͞ ��A�!�����A�6��N ���U�ޟ � �w �~0o endstream endobj 38 0 obj <>>> endobj 39 0 obj <>/ExtGState<>/Font<>/ProcSet[/PDF/Text/ImageC]/XObject<>>>/Rotate 0/TrimBox[0.0 0.0 595.276 841.89]/Type/Page>> endobj 40 0 obj <>stream 0000032789 00000 n The complete realist question is: “What works, for whom, in what respects, to what extent, in what contexts, and how?”. The context-mechanism-outcome (CMO) configuration is used as the main structure for realist analysis. In essence, “middle range” refers to the degree of abstraction and can refer to programme theory, generative mechanisms, CMOs, formal theories etc. It explains when a realist impact evaluation may be most Develop planning documents for the evaluation or M&E system, 8. Review evaluation (do meta-evaluation), 2. These configurations or explanatory pathways describe how various contexts (C) work with underlying mechanisms (M) to produce particular outcomes (O). Realist impact evaluation draws on the realism school of philosophy. Allow sufficient time for assessing the interactions between intervention, actors and context. These will then be linked according to the realist evaluation CMO formula in which findings on context and mechanisms will used to explain outcomes observed. 'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+"://platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs"); Select a content type to filter search results: A special thanks to this page's contributors. Journal of Evaluation in Clinical Practice 13(6):935-941. http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2753.2006.00790.x/pdf, Dickinson, H. (2006). évaluation réaliste, santé rurale, santé mondiale, enfants dâge scolaire. This provided a potential explanation in the form of a causal pathway between the management practices and organisational commitment. H��W]s�H}�W�Gi��|5��TR�cg�5�L"���xk We initially selected ‘organisational commitment’ and ‘trust’ as proximal outcomes of human resource management, because our initial literature review indicated that these outputs are often found to explain the effect of high commitment management. ww�{U~�iM����;L�l6PL#�JVRI�6���Zh��1� �����I�:@� �C������+6[]I�#���s��ن��6ф�"��PE���J>7���-]�'9��9"���k#U���2sX���������(y(����$�TM7/I�$|�������,!l����(�} 0000008175 00000 n Typically, MRTs develop over time based on the accumulation of insights acquired through a series of studies allowing gradual specification of the realist findings. An impact evaluation approach based on contribution analysis, with the addition of processes for expert review and community review of evidence and conclusions. The realist philosophy positions itself between positivist and constructivist paradigms and seeks to answer the question ‘What works for whom, in what circumstances and why?’ 0000001684 00000 n The second CMO configuration can be summarised as ‘keeping up standards of excellence through organisational culture’. Realist trials and the testing of context-mechanism-outcome configurations: a response to Van Belle et al. 0000007537 00000 n Tackling health inequalities through partnership working: learning from a realist evaluation. mechanism-outcome (CMO) configuration and is the analytical unit on which realist evaluation is built. 0000095827 00000 n Realist evaluation is an emerging methodology that is suited to evaluating complex interventions such as SBE. PSS was found to be operating in the case of non-professional cadres like maintenance and administrative personnel. Recommended reading on the philosophical background and the general development of this approach: Pawson, R. & Tilley, N. (1997). Evaluation, 18, 192-212. That is, whenever a program is implemented, it is testing a theory about what ‘might cause change’, even though that theory may not be explicit. Realist approaches to evaluation assume that nothing works everywhere or for everyone: context really does make a difference to programme outcomes. Learn more in: Critical Realist Information Systems Research An approach to decision-making in evaluation that involves identifying the primary intended users and uses of an evaluation and then making all decisions in terms of the evaluation design and plan with reference to these. http://www.ramesesproject.org. Tweets by @BetterEval !function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)? This results in strong organisational commitment and trust. It draws on the … A realistic evaluation of fines for hospital discharges: Incorporating the history of programme evaluations in the analysis. H�\��N�@F�y����eF�D[Mz�O�> �m%��ҋ����M$�=dgv��N�ڬ7�]��f�G��B��?�ƻ�?t!�����o�6�zHRK�^N�?n¾O�ʥo6y��]ݵ��_'�Kl}���]}���.ݞ���}]�K���-�T��ѻtJ�ٴ6ߍ����x������2M���P7>����*�g�G{������-�v�泎I�#8�l0^�����|6� �99��s��� Y�%�?�M�Z�Nl�1c�)�~��%���A� ��B���L��p_���}��=��"�����&šj"*j��Tx*=�JO���S��Tx*=�ʳ+ή�Tx�X?�f�/sr9]�ύ�ʭ3�o?5������z�����C?8�›|0 :r�> endstream endobj 41 0 obj <> endobj 42 0 obj <> endobj 43 0 obj [/ICCBased 59 0 R] endobj 44 0 obj <> endobj 45 0 obj <> endobj 46 0 obj <> endobj 47 0 obj <>stream In this paper, we use a realist evaluation to assess a community-based music initiative designed and implemented to support the well-being of disadvan-taged young people in Scotland. There are hundreds of members and they are very good about providing advice and support. In a recent article in BMC Medicine, Wong and … As with any evaluation, the choice of data collection, and analysis methods and tools should be guided by the types of data that are needed to answer the evaluation questions, or more specifically, to test the initial programme theory in all its dimensions. 37 0 obj <> endobj xref 37 28 0000000016 00000 n 0000003244 00000 n Contexts may relate to the sub-groups for whom outcomes were generated and/or to other stakeholders, processes of implementation, organisational, socio-economic, cultural and political conditions. Rather, the refined programme theory will be tested subsequently in a different context in a next study. The difference between realist and other kinds of programme theory-based evaluation approaches is that a realist programme theory specifies what mechanisms will generate the outcomes and what features of the context will affect whether or not those mechanisms operate. American Journal of Evaluation 22: 317-324. http://aje.sagepub.com/content/22/3/317.full.pdf+html. Login Login and comment as BetterEvaluation member or simply fill out the fields below. In addition: I am based in Australia and am happy to reply to quick questions on methods and methodology. The context-mechanism-outcome (CMO) configuration is used as the main structure for realist analysis. Realist evaluation is method-neutral (i.e., it does not impose the use of particular methods). The actors and the interventions are considered to be embedded in a social reality that influences how the intervention is implemented and how actors respond to it (or not). 0000001143 00000 n Introducing the main constructs employed in RE: program theory, middle-range theory; and context-mechanism-outcome (CMO) configuration. I heartily endorse Alice's suggestion of joining the Rameses discussion list. Evans, D. and Killoran, A. 0000069202 00000 n Health Soc Care Community 14:375-383. http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2524.2006.00650.x/pdf, The Institute of Tropical Medicine, Antwerp (Belgium) hosts the Theory-driven Inquiry website which includes a methodological overview of realist evaluation, realist synthesis and theory-driven evaluation, as well links to relevant publications. You can join the discussion list through www.ramesesproject.org. Elicit-ing, refining and testing CMO configurations allows a deeper and more detailed understanding of for whom, in what circumstances and why the programme works. Synthesise data from a single evaluation. L. Manzano-Santaella applied realist evaluation to a policy in the English NHS that imposed fines for hospital discharges. This study used realist evaluation methodology informed by a realist synthesis . The study started with a case study of a well-performing hospital, where the management style and its effects were assessed on the basis of the initial middle range theory. (2010b). We present an example of a realist study of existing practices within the health care system in Ghana; it is not an evaluation of an intervention per se. The evaluations done so far have not been realist and have not assessed CMO configurations. Evaluation 14: 19-28. (2010) present a realist case study of human resource management in Ghana and discuss the main methodological challenges of realist evaluation, Marchal et al. Realist Impact Evaluation: An introduction. A particular type of case study used to jointly develop an agreed narrative of how an innovation was developed, including key contributors and processes, to inform future innovation efforts. Marchal, B., Van Belle, S., Van Olmen, J., Hoerée, T. & Kegels, G. 2012. Often the case study design is used, whereby case selection is typically purposive, as the cases should enable ‘testing’ of the initial programme theory in all its dimensions. A realist evaluation yields information that indicates how the intervention works (i.e., generative mechanism) and the conditions that are needed for a particular mechanism to work (i.e., specification of contexts) and, thus, it is likely to be more useful to policymakers than other types of evaluation. The initial programme theory may be based on previous research, knowledge, experience, and the assumptions of the intervention designers about how the intervention will work. Qualitative data are coded and appropriate methods for analysing quantitative data applied. 0000003279 00000 n All kinds of theory in realist evaluation – programme theory, theories about particular mechanisms, CMOs, and formal theory– are most useful if developed at a middle level of abstraction. (outcomes) on the basis of reciprocity (mechanism) –even in hospitals with limited margins of freedom regarding recruitment, salary scales, promotion and firing (context). To construct CMOs, the researchers explored the coded data to identify instances where NCA feedback was reported to have been used by providers. 0000098000 00000 n Evaluation 17: 21-36. April 2013; Evaluation 19(2):141-154; DOI: 10.1177/1356389013485194. Declined as CMO configurations constructed as part of a realist synthesis or realist evaluation, the program theory identifies and describes how mechanisms (e.g., actors’ ideas or choices) led to the outcomes for change in a given context [ 30, 48 ]. An impact evaluation approach suitable for retrospectively identifying emergent impacts by collecting evidence of what has changed  and, then, working backwards, determining whether and how an intervention has contributed to these changes. In this way, the effectiveness of the programme is understood, with an explanation of why the outcomes developed as they did and how the programme was able to react to underlying mechanisms and in what contexts. Recommend content, collaborate, share, ask, tell us what you like, suggest an improvement, or just say hi! For more details, we refer to Marchal et al. Their HRM bundles combine employment security, adequate compensation and training. Realist evaluations attempt to answer questions such as what works, for whom, in which circumstances, and why.