Submitted by kwilkerson on
  • Wizard Page

Step 2: Building evidence of effectiveness

SCALER Step 2: Overview

If an intervention does not have evidence that it is effective, your organization can draw on Step 2 of the SCALER to build that evidence before scaling. This step consists of a process—grounded in standards of rigorous research and evaluation—that you can use to gain knowledge about an intervention’s implementation and impact.

This process begins with clearly defining the intervention and the path it takes to affect outcomes. Once the intervention is implemented consistently and has suggestive evidence that participants’ outcomes improve after receiving the intervention, the process moves to designing and implementing an impact evaluation. This requires executing an evaluation that meets standards for research rigor and demonstrates the intervention’s effectiveness. If the evaluation shows evidence of effectiveness, the intervention is ready to scale. If it does not, you will tweak the intervention in ways to increase its likelihood of being effective, and the evidence-building process begins again.

Define intervention

  • For an organization to implement an intervention consistently and reliably produce beneficial outcomes for participants, the intervention must be (1) clearly defined, (2) implemented as part of the organization’s regular routine, and (3) and implemented according to the defined program model.
  • Additionally, any modifications the organization makes to the intervention model should be informed by research.
Start

FAQ

We recommend designating one person or role to be the SCALER owner, such as the intervention’s program or data manager. Because personnel at all levels are involved in intervention scaling, the SCALER owner will want to involve others including:

  • Executive directors
  • Program managers or frontline supervisors
  • Measurement, learning, and evaluation personnel (data managers or internal evaluation specialists)
  • Administrative personnel (human resources, development, or purchasing personnel)
Completing Step 2 (building evidence) requires enough knowledge and understanding of the following aspects of the intervention and your organization:

  • How the intervention was defined and implemented at the organization, which includes the intervention’s core activities and the resources needed to carry out these activities
  • The organization’s support for conducting an evaluation of the intervention
  • The availability of organizational resources and personnel to support an evaluation
  • Familiarity with how research should be conducted in a manner that meets standards for rigor
Organizations can take one of several approaches:

  • Involved personnel could complete the checklist on their own, discuss responses, and come to agreement on scores at a meeting organized by the SCALER owner.
  • Several personnel could fill out the checklist together during a meeting.
  • The SCALER owner could gather the necessary information and complete the checklist.
The printable Step 2 Results report summarizes whether evidence of effectiveness exists for your intervention. With this summary, you can:

  • Share the results with key stakeholders, organizational leaders, and funders.
  • Use the results to identify areas that need further attention, resources, or support as you work on building evidence for your intervention and prioritize next steps.
  • To gauge your progress, complete Step 2 multiple times as you prepare and continue your efforts to build evidence for your intervention’s effectiveness.

FAQ

A well-articulated TOC establishes the inputs, activities, outputs, and outcomes needed for the intervention to succeed. Identifying these components in a TOC provides a clear sense of what is needed to produce outputs and what needs to be measured to determine that the intervention improves participant outcomes.
It is critical that all stakeholders, including organizational leaders and personnel involved in implementing the intervention, support research and evaluation efforts before they are designed and begin. This helps ensure everyone involved in the evaluation will contribute accurate and reliable information about the intervention. If stakeholders are not willing and able to participate in an evaluation, the evaluation will likely not be feasible or as useful as possible, even if it is conducted.
Planning for and allocating sufficient resources to evaluation is important. Not allocating sufficient funding to evaluation activities could lead to less rigorous or poorly implemented research that does not meet the standards of rigor for showing an intervention is effective. Visit the SCALER Resources for more information on the evaluation-related costs to consider and how costs can vary depending on the scope of the evaluation.
A third-party evaluator should be engaged to conduct a rigorous evaluation of intervention effectiveness. A third-party evaluator should be objective and not have a vested interest in whether the intervention produces beneficial impacts for participants. When the evaluation is being implemented, organizational personnel and evaluators should work together to make sure the research tasks are carried out as planned.
Three reasons might explain why evaluation efforts do not show evidence of effectiveness:
  • Method failure occurs when a study’s design or implementation was flawed (for example, the sample size was too small to show statistically significant impacts). If this happens, your organization should conduct a new study that addresses the flaw (for example, by increasing the sample size).
  • Implementation failure occurs when an intervention is not implemented as intended (that is, without fidelity to the model). If this happens, your organization should take steps to improve implementation fidelity before conducting a new study.
  • Theory failure occurs when the intervention is not actually linked to the desired outcome. If this causal link does not exist, revisit the intervention or key components.

Use this checklist as you seek to build evidence of effectiveness for your intervention. Under “Define intervention,” “Lay evaluation groundwork,” “Prepare for evaluation,” and “Conduct evaluation and assess effectiveness,” select the items that reflect the status of your efforts in building evidence.

Ideally, you would make sure all items in a section are complete before moving to the next section. Ultimately, should the intervention be shown to be effective, you would get ready to scale the intervention (see Step 3 of the SCALER). Should the evidence not indicate effectiveness, the organization should continue to build evidence.

Define intervention
This means there is a clear description of the human, financial, and physical resources (inputs) and the intervention’s core components (activities) that are necessary to change participants’ outcomes.
This means the organization implements the inputs and activities identified in the TOC as part of a regular organizational routine, and personnel implementing the intervention are proficient and skilled and have managerial and administrative support.
This means the intervention is delivered according to the defined program model.
This means modifications made to an intervention in the hopes of improving participants’ outcomes are based on other existing research and evidence.
Lay evaluation groundwork
This means, for example, stakeholders understand the value of using a TOC to improve the design and implementation of an intervention and, subsequently, generating estimates of the intervention’s impact.
This means there is a consensus among stakeholders on the outcomes the intervention is trying to influence and the goals for the extent to which outcomes should be improved.
This means the stakeholders agree on the intervention’s TOC, implementation goals for the intervention (outputs and outcomes), research questions for the evaluation, and what constitutes evidence of effectiveness.
This means that stakeholders, including organizational leaders and personnel involved in implementing the intervention, support the evaluation efforts before they begin.
Prepare for evaluation
This means there are resources to cover costs associated with implementing the evaluation such as contracting with third-party evaluators, purchasing and preparing systems to record data that will be analyzed for the evaluation, paying fees for other experts in research design and implementation, and accounting for personnel time in collecting data for the evaluation.
This means, for example, that organization personnel help third-party evaluators understand the inputs and activities identified in the intervention’s TOC, work collaboratively with the evaluators during the evaluation’s implementation, and provide insights to support the evaluation’s data analysis.
Conduct evaluation and assess effectiveness
This means having a well-designed and well-implemented impact study conducted by an independent evaluator (see Step 1 for standards to identify effective interventions).
This means the intervention is shown to be effective for improving participant outcomes.
This means the evidence for the intervention’s effectiveness is applicable to the locations where or populations for which your organization plans to scale its intervention.

 
Back
Next

SCALER Step 2: Results

Your organization should continue its efforts to build evidence of your intervention’s effectiveness that also aligns to your plans for scaling.

Before your organization invests resources to scale an intervention, the intervention should have demonstrated evidence of effectiveness. To build such evidence, the intervention’s model must be defined and stable, and informed by research, and your organization must implement the intervention with fidelity to that model. Once that condition has been met, your organization can then lay the groundwork and prepare for conducting an evaluation of your intervention. After laying the groundwork and preparing for evaluation, your organization can find and work with a third-party evaluator to carry out a rigorous research study that demonstrates the intervention’s effectiveness and aligns to your plans for scaling the intervention. Use the guidance in the results below to identify the areas that need to be addressed to build evidence. View the SCALER Resources for more information that can help your evidence-building efforts and return to Step 2 of the SCALER to document any progress you make.

  • Define interventioncontinue making progress
  • Lay evaluation groundworkcontinue making progress
  • Prepare for evaluationcontinue making progress
  • Conduct evaluation and
    assess effectivenesscontinue making progress

What do your results mean?

Return to edit

Define intervention

  • The intervention does not have a defined theory of change (TOC)
    Begin working with others at your organization to develop a defined TOC for your intervention. A TOC defines an intervention by demonstrating how the intervention’s resources and activities are structured with the intent of changing participant knowledge, attitudes, behaviors, or conditions in ways that improve outcomes. Developing a TOC with this level of detail generally begins with thinking about the desired outcomes that the intervention will affect and then identifying all the assumptions about how the intervention might do so. Articulating the assumptions for how the outcomes will be affected is critical for the intervention’s success because it helps establish the inputs, activities, outputs, and outcomes needed for the intervention to succeed. View the SCALER Resources for more information on how to develop a TOC for your intervention.

  • The intervention is not yet mature and stable
    The inputs and activities identified in the TOC must be implemented and supported as part of a regular organizational routine. Conducting an evaluation before an intervention is mature and stable might not provide an accurate assessment of the intervention’s potential to improve participant outcomes. If your intervention is having difficulties with achieving maturity and stability, consider using formative and outcome evaluations to gather the information needed to refine the intervention. During a formative evaluation, organizations typically seek to determine if an intervention is feasible before fully implementing it. For an outcome evaluation, organizations monitor whether the intervention is achieving the desired outcomes and might decide that additional or different inputs are needed, or that the intervention’s activities need to be tweaked.

  • The intervention is not yet implemented with fidelity as described in the TOC
    Implementing an intervention with fidelity as described in the TOC is important because it can affect how well the intervention can succeed in improving participant outcomes. Your organization should investigate the underlying issues with implementation fidelity, such as aspects of service delivery adhering to the intervention model as intended.

  • The intervention, including any modifications, is not informed by research
    Your organization may seek to adapt an intervention in the hopes of improving upon participant outcomes. It might also make adaptations due to external contextual factors—for instance, funding might become available to engage a new population, and an organization can seek to adapt its intervention to apply for and use that funding. Ideally, these adaptations would be informed by other evaluation studies to illustrate how the adapted intervention can produce the desired outcomes for participants.

You have completed Step 2 of the SCALER. Based on your results, your organization needs to continue its efforts to build evidence of effectiveness for your intervention. Ideally, you would return to Step 2 to update your results as you make progress in this area and until such evidence is built before proceeding to Step 3 of the SCALER (to determine your intervention’s and organization’s readiness to scale).

Back
Print Results
Home

Need to finish later?

Update or view your checklists in the future with your unique link:

return home view SCALER resources