Systematic reviews aim to bridge the research-policy divide by assessing the range of existing evidence on a particular topic, and presenting the information in an accessible format. For example, UNICEF defines impact as "The longer term results of a program — technical, economic, socio-cultural, institutional, environmental or other — whether intended or unintended.
The assumption is that as they have been selected to receive the intervention in the future they are similar to the treatment group, and therefore comparable in terms of outcome variables of interest.
An evaluation which looks at the impact of an intervention on final welfare outcomes, rather than only at project outputs, Evaluation of the impact of political a process evaluation which focuses on implementation; An evaluation carried out some time five to ten years after the intervention has been completed so as to allow time for impact to appear; and An evaluation considering all interventions within a given sector or geographical area.
Different designs require different estimation methods to measure changes in well-being from the counterfactual. How well is the program or technology delivered?
It is safe to say that if an inadequate design yields bias, the stakeholders who are largely responsible for the funding of the program will be the ones most concerned; the results of the evaluation help the stakeholders decide whether or not to continue funding the program because the final decision lies with the funders and the sponsors.
Impact evaluation designs are identified by the type of methods used to generate the counterfactual and can be broadly classified into three categories — experimental, quasi-experimental and non-experimental designs — that vary in feasibility, cost, involvement during design or after implementation phase of the intervention, and degree of selection bias.
In formative research the major questions and methodologies are: Perhaps the most important basic distinction in evaluation types is that between formative and summative evaluation.
This means that "those individuals that were from the intervention group whose outcome data are missing cannot be assumed to have the same outcome-relevant characteristics as those from the control group whose outcome data are missing" Rossi et al.
An evaluation which looks at the impact of an intervention on final welfare outcomes, rather than only at project outputs, or a process evaluation which focuses on implementation; An evaluation carried out some time five to ten years after the intervention has been completed so as to allow time for impact to appear; and An evaluation considering all interventions within a given sector or geographical area.
The estimate of program effect is then based on the difference between the groups on a suitable outcome measure Rossi et al. There are five key principles relating to internal validity study design and external validity generalizability which rigorous impact evaluations should address: ITT therefore provides a lower-bound estimate of impact, but is arguably of greater policy relevance than TOT in the analysis of voluntary programs.
I decided I would see what my library says about politics: If either of these conditions is absent from the design, there is potential for bias in the estimates of program effect" Rossi et al. The IEG of the World Bank has systematically assessed and summarized the experience of ten impact evaluation of development programs in various sectors carried out over the past 20 years.
Furthermore, it is possible that program participants are disadvantaged if the bias is in such a way that it contributes to making an ineffective or harmful program seem effective. Evaluation utilizes many of the same methodologies used in traditional social research, but because evaluation takes place within a political and organizational context, it requires group skills, management ability, political dexterity, sensitivity to multiple stakeholders and other skills that social research in general does not rely on as much.
What is missing from the term 'impact' evaluation is the way 'impact' shows up long-term.
The difference-in-difference or double difference estimator calculates the difference in the change in the outcome over time for treatment and comparison groups, thus utilizing data collected at baseline for both groups and a second round of data collected at end-line, after implementation of the intervention, which may be years later.
As the term suggests, they emphasize the central importance of the evaluation participants, especially clients and users of the program or technology. Confounding factors are therefore alternate explanations for an observed possibly spurious relationship between intervention and outcome.
Other forms of bias[ edit ] There are other factors that can be responsible for bias in the results of an impact assessment. The problems are complex and the methodologies needed will and should be varied.
Scientific-experimental models are probably the most historically dominant evaluation strategies. It would also be legitimate to include the Logical Framework or "Logframe" model developed at U.
Does that mean that donor concerns for embedding the political institutions of accountability, transparency, participation or inclusion, for instance - that always take a long time to mature - should be abandoned? Client-centered and stakeholder approaches are examples of participant-oriented models, as are consumer-oriented evaluation systems.
Propensity score matching PSM uses a statistical model to calculate the probability of participating on the basis of a set of observable characteristics and matches participants and non-participants with similar probability scores. Matching involves comparing program participants with non-participants based on observed selection characteristics.
Estimation methods[ edit ] Estimation methods broadly follow evaluation designs. Where is the problem and how big or serious is it? Regression discontinuity design exploits a decision rule as to who does and does not get the intervention to compare outcomes for those just either side of this cut-off.
Interfering events[ edit ] Interfering events are similar to secular trends; in this case it is the short-term events that can produce changes that may introduce bias into estimates of program effect, such as a power outage disrupting communications or hampering the delivery of food supplements may interfere with a nutrition program Rossi et al.
This may be because of participant self-selection, or it may be because of program placement placement bias.
Unlike other forms of evaluation, they permit the attribution of observed changes in outcomes to the program being evaluated by following experimental and quasi-experimental designs". Self-selection occurs where, for example, more able or organized individuals or communities, who are more likely to have better outcomes of interest, are also more likely to participate in the intervention.
Understand context, including the social, political and economic setting of the intervention. Estimation methods[ edit ] Estimation methods broadly follow evaluation designs. Group comparisons that have not been formed through randomization are known as non-equivalent comparison designs Rossi et al.
What was the effectiveness of the program or technology?Impact Evaluation of Social Programs: A Policy Perspective John Blomquist, Senior Economist To adequately review the political aspects of evaluation, it is necessary to discuss the omponents and techniques in some detail.c An impact evaluation is an assessment of.
The political logic suggests creating conditions that are easy to meet but symbolically important so that the “impact” should be small and the “impact” of the conditionality could be interpreted as the welfare loss needed to politically sustain the program. Impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones.
In contrast to outcome monitoring, which examines whether targets have been achieved, impact evaluation is structured to answer the question: how would outcomes such as participants' well-being have. Evaluation has always been political. How can evaluators work with the politics of evaluation?
difference diffusion of innovation evaluation methods evaluation models evaluation use Focus Groups formative Group Interview impact impacts implementation inputs IRB judgement knowledge logic models merit monitoring needs assessment outcomes.
Evaluation Models, Approaches, and Designs BACKGROUND question guiding this kind of evaluation is, “What impact did the training bistroriviere.com 7/22/ PM Page Professional evaluation: Social impact and political conse-quences.
Thousand Oaks, CA: Sage. The political logic suggests creating conditions that are easy to meet but symbolically important so that the “impact” should be small and the “impact” of the conditionality could be interpreted as the welfare loss needed to politically sustain the program.Download