Developing and Using Program Logic: A Guide

Download as PDF

On this page

1. Introduction
2. What is program logic?
3. Why develop a program logic model?
4. When to develop program logic
5. Developing program logic
6. How can program logic be used?
7. Conclusion
8. References

Introduction

NSW Health is committed to the development of evidence-based policies and programs and the ongoing monitoring, review and evaluation of existing programs in line with NSW Government requirements. The NSW Treasury Policy and Guidelines: Evaluation (TPG22-22)¹ sets out mandatory requirements, recommendations and guidance for NSW General Government Sector agencies and other government entities to plan for and conduct the evaluation of policies, projects, regulations and programs.

This guide complements the NSW Treasury Evaluation Policy and Guidelines and the associated workbooks and resources. It aims to support NSW Health staff in the development of program logic and its use in informing population health program planning, implementation, monitoring and evaluation. It promotes a planned and structured approach to developing program logic and includes information on:

• the meaning and purpose of program logic
• when and how to develop program logic
• how program logic can be used, with a particular focus on monitoring and evaluation.

What is program logic?

A program logic model is a schematic representation that describes how an initiative* is intended to work by linking activities with outputs and short, medium and longer-term outcomes. Program logic aims to show the intended causal links for a program. Several different terms are used to describe program logic, such as program theory, logic model, theory of change, results chain and intervention logic.

* In this guide the word ‘program’ is used interchangeably with ‘initiative’. The NSW Treasury Policy and Guidelines: Evaluation (TPG22-22) define an initiative as a program, policy, strategy, service, project, or any series of related events. Initiatives can vary in size and structure; from a small initiative at a single location, a series of related events delivered over a period, or whole-of-government reforms with many components delivered by different agencies or governments.¹

Why develop a program logic model?

Using a program logic approach to describe a program has many benefits. For example:

• Having an agreed program logic model supports a systematic and integrated approach to program planning, implementation, monitoring and evaluation.²
• A program logic model tells the story of how the program is proposed to work. By clarifying activities and intended outcomes, a program logic model illustrates the change processes underlying a program.^1-6
• Program logic makes program assumptions explicit and enables testing of how these assumptions are supported by evidence.^1,5
• Program logic is a useful tool for engaging stakeholders in program planning and evaluation, and clearly communicating with stakeholder audiences about program concepts.² A program logic model agreed with key stakeholders can facilitate common language about the program and build a shared understanding of how it will work.^4-6
• Program logic provides a framework for identifying measures and indicators that track the program’s implementation and performance as well as areas where evaluation will be most important.^1,3,6

When to develop program logic

Ideally, program logic should be developed in the program planning stage. This allows stakeholders to articulate the desired program outcomes and clarify how the intervention will achieve these. Note that program logic does not replace a program plan, but rather informs it; a program plan generally has more detailed steps and tasks.⁴

The program logic may be reviewed and refined at different times, including during implementation and as part of planning a program evaluation.

Program logic can also be developed for an existing program, although this may be more difficult, particularly where the program is complex or has multiple unrelated components. Fitting a program logic model onto an existing program can enable stakeholders to consider whether the outputs and outcomes identified through the program logic match what the program is delivering, and amend the program implementation accordingly.⁴

Developing program logic

Getting started

Developing program logic is a participatory and iterative exercise. It involves review of the program to identify aims, objectives, activities and intended outcomes, and refining and assembling these statements into a causal chain that shows how the activities are assumed to contribute to short, medium and longer-term outcomes.¹ During this stage it is important to closely examine and question the assumptions underlying the program components and causal chain so that any unintended or unforeseen consequences can be anticipated, and outcomes can be fairly attributed to the program.

The process of developing program logic should involve working with a range of stakeholders to draw on their understanding of the program and its outcomes. Engaging stakeholders also has the benefit of encouraging ownership of the final program logic model.⁶ Decisions about which stakeholders to involve and the nature of their involvement will depend on how the program logic will be used. For instance, if the program logic is intended to develop an understanding of what is needed to make a program work, it is important to involve program clients and partner agencies; if the program logic will be used to design an evaluation, program staff and management involvement is important.⁶

Program logic development is often undertaken in a workshop format to engage relevant stakeholders. Alternatively, or in addition to a workshop, structured interviews may be conducted with stakeholders to elicit their understanding of the problem being addressed by the program, its causes and consequences, and how the program will contribute to addressing the problem.⁶

Developing a program logic model

There are many ways to develop program logic. The BetterEvaluation website lists several approaches, including articulating ‘mental models’ by talking with key informants individually or in groups about how they understand an intervention works; SWOT analysis to assess the Strengths, Weaknesses, Opportunities and Threats of a program to determine how it might best be implemented; or ‘backcasting’.⁷

Backcasting is a useful approach that involves identifying the long-term outcomes of a program and subsequently working backwards to identify the necessary steps required to achieve these outcomes. The benefit of backcasting, compared to approaches involving forecasting, is that it allows stakeholders to consider what is needed to create the future, rather than thinking about what is currently happening and trying to predict the future.⁷

Suggested steps for developing a program logic model using the backcasting approach are outlined below.

Step 1: Develop an outcomes hierarchy

An outcomes hierarchy, sometimes referred to as an outcomes chain, shows the assumed cause-and-effect relationships between program outcomes, in the short, medium and longer-term. The outcomes hierarchy is the centrepiece of the program logic as it provides a basis for thinking about how the program needs to function to achieve the desired outcome.⁵

Funnell and Rogers suggest a five-step process for developing an outcomes hierarchy:⁵

• Prepare a list of possible outcomes. It may be helpful to think of outcomes in relation to the problem that the program aims to address, including the causes and consequences of the problem. For example, if the problem is consumption of unhealthy food among children, the intended outcomes might include “increased fruit and vegetable intake” and “reduced consumption of energy dense food”. If one of the causes of unhealthy food consumption is lack of availability of healthy food, then another intended outcome might be “increased healthier choices in food outlets”. If a known consequence of unhealthy food consumption is overweight, the intended outcomes might include reduction in overweight. All important outcomes – including any unintended outcomes that can be predicted – should be documented.
• Cluster the outcomes. Outcomes that are related (e.g. all those referring to desired changes in program participants’ knowledge) should be grouped and given a simple working title (e.g. “children’s knowledge about healthy food increases”).
• Arrange the outcomes in a chain of ‘if-then’ statements. The if-then approach is a useful way of ensuring logical thinking when ordering outcomes in a chain from short-term impacts to long-term outcomes (e.g. “if there are healthier choices in food outlets, then children will be more likely to purchase healthy food”). It may be helpful to identify the highest and lowest levels of outcome first to anchor the remainder of the chain. The lowest level in the outcomes chain is the first point at which the program has some sort of effect on the target group, such as ensuring participation in the program. The highest levels are the ultimate health outcomes desired, which often relate to reducing a problem and its consequences.
• Identify any feedback loops (i.e. where a higher level outcome affects (or ‘feeds back’ into) a lower level one). This can be important when participation in a program (a lower level outcome) is affected by the success of the program in achieving anticipated outcomes (higher level outcomes) (e.g. a program that improves the health of clients may consequently attract more clients).
• Validate the outcomes hierarchy with key stakeholders.

An example of an outcomes hierarchy is presented at Figure 1. Note that the number of steps in outcomes hierarchies can vary. Ultimately, the outcomes should correspond to program aims and objectives.

Figure 1. Example of an outcomes hierarchy for the NSW implementation of the Healthy Workers Initiative

Step 2: Identify the deliverables

The next step in developing a program logic model is to consider the program deliverables. These include:

• Outputs – the products, goods or services that need to be provided to program participants in order for the short- term outcomes to be achieved.
• Activities – the essential actions required to produce program outputs.

Step 3: Identify assumptions and review the program logic

Once the model is complete, the logic underlying the activities, outcomes and causal links should be reviewed, and any assumptions identified. Assumptions include beliefs about the program, how it will work, and program participants (e.g. how they learn, how they behave, their motivations).⁵

Figure 2 summarises the steps in constructing a program logic using the backcasting approach.

Figure 2. Steps in constructing a program logic using the backcasting approach

Representing program logic

An effective program logic model should:⁵

• present a coherent causal model that explains how the program contributes to the outcomes
• be logical, so that the direction of expected change is clearly depicted and the sequential progression is plausible
• communicate clearly by focusing on key elements, using design features (e.g. symmetry, alignment) and ensuring readability.

Program logic can be represented in several ways including logframe (a matrix that maps program aims, objectives, activities and outputs against relevant indicators, data sources and assumptions); realist matrix (a table describing program resources, how they interact with the ‘object’ being changed, contextual variables, and anticipated outcomes); and the pipeline model.^6,7

The pipeline model (also known as a ‘results chain’) is commonly used for health programs. It depicts a program logic as a linear process with inputs and activities at the front (left) and outcomes at the end (right) (Figure 3).^6,7

A simple example of a pipeline logic model is presented at Figure 4. Two examples that link specific activities to specific outputs and outcomes are presented at Figures 5 and 6.

Other examples of program logic models can be viewed in published reports,^9,11,12 and the Treasury Evaluation Policy and Guidelines and related workbooks.^1,3 The NSW Treasury examples include ‘benefits’ (improvements in social welfare) and how the program aligns with ‘State Outcomes’ as further categories of longer-term outcomes.

How can program logic be used?

There are several ways in which program logic can support action throughout the program planning, implementation, monitoring and evaluation cycle.^5,6 For example:

• Used as a planning tool, program logic can clarify the path to get from where a program is to where stakeholders want it to be, including how investments are linked to activities to achieve the desired results.
• Program logic provides a simple, clear graphic representation that helps communicate the intent of a program to stakeholders, including program staff and funders.
• Program logic can inform the development of a detailed management plan to guide program implementation and to help monitor operations, processes and functions.
• A program logic model can facilitate effective evaluation by helping to establish what to evaluate, determine key evaluation questions, and identify relevant information to address those questions.

The NSW Health Guide to Measuring Value provides specific guidance about using program logic to support measurement across the quadruple aim of value-based healthcare at NSW Health.¹³

Using program logic to plan a program evaluation

Monitoring is a continuous and systematic process of collecting and analysing information about the implementation and outcomes of a program.1 Monitoring processes can be designed to report on performance, inform ongoing improvement and help identify trends and issues or gaps for further examination through evaluation.

Program logic can support program monitoring in the following ways

• Determining what data to collect. In the early stages of program planning a program logic can be used to anticipate what will happen and establish data collection processes to track changes as the program is implemented. For existing programs, existing data sources can be mapped onto the program logic then used to identify data gaps and priorities for collecting additional data.⁶
• Identifying key measures and indicators to track progress. A program logic can be used to identify key measures or indicators that should be monitored at each stage of the program’s life.¹ Indicators of the quality and quantity of inputs and activities can support effective program management.⁶ Indicators for short to medium-term outcomes can help to track early progress in achieving results. This can be particularly important when the intended impacts are longer-term and information about intermediate outcomes is needed to inform decisions.⁶
• Providing a program performance narrative. A program logic can help to make sense of data that has been collected and explain what has happened. It can provide a framework for reporting program performance including how the program makes contributions. This can be useful for communicating about the program to partners, participants and policymakers.⁶

Using program logic to plan a program evaluation

Evaluation will usually investigate a broader range of information on program performance and context than is feasible to monitor on an ongoing basis, as well as address specific questions regarding the performance of a program.¹ Program logic can support planning for an evaluation in the following ways:

• Determining what to evaluate. Program logic can help to identify the most important aspects of a program to be evaluated.⁵ This includes not only the intended outcomes of a program, but also aspects of program implementation that are critical to the achievement of these as described in the links between components of the logic model.⁵
• Identifying key evaluation questions. Evaluation questions serve to focus an evaluation and provide direction for the collection and analysis of data.¹⁴ Key evaluation questions should ‘fall out’ of the main components of a program logic model and the assumptions underpinning the components.¹⁵ This helps to convert very broad questions of interest (e.g. “Was the program effective?”) into more specific questions relating to particular elements of the causal pathway (e.g. “How effective was the program in helping smokers to initiate quit attempts?”).⁵ Table 1 lists some examples of evaluation questions drawn from program logic components. In cases where program stakeholders have diverse and disparate information needs (or ‘shopping lists’), program logic is particularly useful for formulating sensible sets of hierarchical questions.⁵

Examples of evaluation questions drawn from program logic components

Program activity

Training sessions delivered to staff.

Process

• How many training sessions were delivered?
• How many staff members participated in the training?
• How satisfied were participants with the training?

Anticipated short-term outcome

Quit attempts initiated

Outcome

• What proportion of clients initiated quit attempts?
• Were there differences in quit attempts across client groups?

Anticipated long-term outcome

Reduced smoking rate

Outcome

• How have smoking rates changed over time?
• In which population groups have smoking rates changed?

Identifying information needed to answer evaluation questions. Program logic can help in identifying the most appropriate information to answer evaluation questions by assisting in defining what constitutes program ‘success’. Success may be described in terms of the attributes of a program activity, output or outcome (e.g. quality, quantity, reach, timeliness, cost), and/or how it compares with agreed standards or targets.⁵ For example, an anticipated program outcome of “workers make sustained healthy lifestyle behaviour change” may be described in relation to attributes of the workers (e.g. age, gender), the specific changes in behaviour desired (e.g. physical activity, diet), and how ‘sustained’ is defined (e.g. number of months post-intervention). In this example success might be interpreted as “increased vegetable intake at 12-months post-intervention compared with baseline among male workers aged 30 to 45 years”. The success criteria will determine what information is relevant and useful for answering the evaluation questions, which in turn will inform the selection of data collection sources, methods and instruments.⁵

Helping to decide when to collect data. Because a program logic model depicts the expected sequence of activities, outputs and outcomes, it can inform decisions about the appropriate time to assess processes, shorter-term and longer-term outcomes. For example, program logic can help to identify critical preconditions for achieving outcomes, so that an outcome evaluation is not undertaken until it is clear that the preconditions have been met.⁵

Providing a mechanism for ensuring acceptability among stakeholders. A program logic model can help to ensure program stakeholders’ views concerning specific issues are kept in perspective in an evaluation by clarifying how these views relate to the overall program.⁵

Conclusion

This guide aims to support the development of program logic models and their use for monitoring and evaluation.

There are many ways to develop program logic models. This guide suggests an analytical and consultative approach that may be helpful for NSW Health staff involved in population health program planning, implementation, monitoring and evaluation. Ideally program logic should be developed in the program planning stage, although it can also be developed for an existing program.

A program logic model can support action throughout the program planning, implementation, monitoring and evaluation cycle. Program logic can be particularly useful for facilitating a shared understanding of how a program is proposed to work and what it is expected to achieve. This can help identify measures and indicators to monitor program implementation and performance and help determine what to evaluate, key evaluation questions, and information to address those questions.

References

1. NSW Treasury. NSW Treasury Policy and Guidelines: Evaluation (TPG22-22). Sydney: NSW Treasury; 2023.
2. WK Kellogg Foundation. Logic Model Development Guide. Battle Creek, Michigan: WK Kellogg Foundation; 2004.
3. NSW Treasury. Evaluation Workbook I. Foundations of evaluation. Sydney: NSW Treasury; 2022.
4. National Centre for Sustainability. Evaluation Toolbox: Program Logic. Swinburne University of Technology
5. Funnell SC, Rogers PJ. Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Hoboken: Wiley; 2011.
6. BetterEvaluation. Describe the theory of change.
7. BetterEvaluation. Develop theory of change/programme theory.
8. St George A, King L. Evaluation Framework for NSW Implementation of Healthy Workers Initiative (Revised June 2011). Sydney: Physical Activity Nutrition and Obesity Research Group, University of Sydney; 2011.
9. Wiggers J, Wolfenden L, Campbell E, Gillham K, Bell C, Sutherland R, et al. Good for Kids, Good for Life, 2006- 2010: Evaluation Report. Sydney: NSW Ministry of Health; 2013.
10. Meijer D, Campbell-Lloyd S. NSW School Vaccination Program. Health Protection Report, NSW: Communicable Diseases. December 2014.
11. Centre for Oral Health Strategy NSW. Smoking Cessation Brief Intervention at the Chairside: The Role of Public Oral Health Services. Sydney: NSW Ministry of Health; 2013.
12. Cultural and Indigenous Research Centre Australia. Culture Health Communities Activity Challenge Evaluation. Sydney: NSW Ministry of Health; 2014.
13. NSW Health. NSW Health Guide to Measuring Value. Sydney: NSW Ministry of Health; 2023.
14. Owen JM. Program evaluation: forms and approaches. 3rd edition. New York: The Guilford Press; 2007.
15. Roughley A. Developing and Using Program Logic in Natural Resource Management: User Guide. Canberra: Commonwealth of Australia; 2009.