Download as PDF
NSW Health is committed to the development of evidence-based policies and programs and the ongoing monitoring, review and evaluation of existing programs in line with NSW Government requirements. The NSW Treasury Policy and Guidelines: Evaluation (TPG22-22)1 sets out mandatory requirements, recommendations and guidance for NSW General Government Sector agencies and other government entities to plan for and conduct the evaluation of policies, projects, regulations and programs.
This guide complements the NSW Treasury Evaluation Policy and Guidelines and the associated workbooks and resources. It aims to support NSW Health staff in the development of program logic and its use in informing population health program planning, implementation, monitoring and evaluation. It promotes a planned and structured approach to developing program logic and includes information on:
A program logic model is a schematic representation that describes how an initiative* is intended to work by linking activities with outputs and short, medium and longer-term outcomes. Program logic aims to show the intended causal links for a program. Several different terms are used to describe program logic, such as program theory, logic model, theory of change, results chain and intervention logic.
* In this guide the word ‘program’ is used interchangeably with ‘initiative’. The NSW Treasury Policy and Guidelines: Evaluation (TPG22-22) define an initiative as a program, policy, strategy, service, project, or any series of related events. Initiatives can vary in size and structure; from a small initiative at a single location, a series of related events delivered over a period, or whole-of-government reforms with many components delivered by different agencies or governments.1
Using a program logic approach to describe a program has many benefits. For example:
Ideally, program logic should be developed in the program planning stage. This allows stakeholders to articulate the desired program outcomes and clarify how the intervention will achieve these. Note that program logic does not replace a program plan, but rather informs it; a program plan generally has more detailed steps and tasks.4
The program logic may be reviewed and refined at different times, including during implementation and as part of planning a program evaluation.
Program logic can also be developed for an existing program, although this may be more difficult, particularly where the program is complex or has multiple unrelated components. Fitting a program logic model onto an existing program can enable stakeholders to consider whether the outputs and outcomes identified through the program logic match what the program is delivering, and amend the program implementation accordingly.4
Developing program logic is a participatory and iterative exercise. It involves review of the program to identify aims, objectives, activities and intended outcomes, and refining and assembling these statements into a causal chain that shows how the activities are assumed to contribute to short, medium and longer-term outcomes.1 During this stage it is important to closely examine and question the assumptions underlying the program components and causal chain so that any unintended or unforeseen consequences can be anticipated, and outcomes can be fairly attributed to the program.
The process of developing program logic should involve working with a range of stakeholders to draw on their understanding of the program and its outcomes. Engaging stakeholders also has the benefit of encouraging ownership of the final program logic model.6 Decisions about which stakeholders to involve and the nature of their involvement will depend on how the program logic will be used. For instance, if the program logic is intended to develop an understanding of what is needed to make a program work, it is important to involve program clients and partner agencies; if the program logic will be used to design an evaluation, program staff and management involvement is important.6
Program logic development is often undertaken in a workshop format to engage relevant stakeholders. Alternatively, or in addition to a workshop, structured interviews may be conducted with stakeholders to elicit their understanding of the problem being addressed by the program, its causes and consequences, and how the program will contribute to addressing the problem.6
There are many ways to develop program logic. The BetterEvaluation website lists several approaches, including articulating ‘mental models’ by talking with key informants individually or in groups about how they understand an intervention works; SWOT analysis to assess the Strengths, Weaknesses, Opportunities and Threats of a program to determine how it might best be implemented; or ‘backcasting’.7
Backcasting is a useful approach that involves identifying the long-term outcomes of a program and subsequently working backwards to identify the necessary steps required to achieve these outcomes. The benefit of backcasting, compared to approaches involving forecasting, is that it allows stakeholders to consider what is needed to create the future, rather than thinking about what is currently happening and trying to predict the future.7
Suggested steps for developing a program logic model using the backcasting approach are outlined below.
An outcomes hierarchy, sometimes referred to as an outcomes chain, shows the assumed cause-and-effect relationships between program outcomes, in the short, medium and longer-term. The outcomes hierarchy is the centrepiece of the program logic as it provides a basis for thinking about how the program needs to function to achieve the desired outcome.5
Funnell and Rogers suggest a five-step process for developing an outcomes hierarchy:5
An example of an outcomes hierarchy is presented at Figure 1. Note that the number of steps in outcomes hierarchies can vary. Ultimately, the outcomes should correspond to program aims and objectives.
Figure 1. Example of an outcomes hierarchy for the NSW implementation of the Healthy Workers Initiative
The next step in developing a program logic model is to consider the program deliverables. These include:
Once the model is complete, the logic underlying the activities, outcomes and causal links should be reviewed, and any assumptions identified. Assumptions include beliefs about the program, how it will work, and program participants (e.g. how they learn, how they behave, their motivations).5
Figure 2 summarises the steps in constructing a program logic using the backcasting approach.
Figure 2. Steps in constructing a program logic using the backcasting approach
An effective program logic model should:5
Program logic can be represented in several ways including logframe (a matrix that maps program aims, objectives, activities and outputs against relevant indicators, data sources and assumptions); realist matrix (a table describing program resources, how they interact with the ‘object’ being changed, contextual variables, and anticipated outcomes); and the pipeline model.6,7
The pipeline model (also known as a ‘results chain’) is commonly used for health programs. It depicts a program logic as a linear process with inputs and activities at the front (left) and outcomes at the end (right) (Figure 3).6,7
A simple example of a pipeline logic model is presented at Figure 4. Two examples that link specific activities to specific outputs and outcomes are presented at Figures 5 and 6.
Other examples of program logic models can be viewed in published reports,9,11,12 and the Treasury Evaluation Policy and Guidelines and related workbooks.1,3 The NSW Treasury examples include ‘benefits’ (improvements in social welfare) and how the program aligns with ‘State Outcomes’ as further categories of longer-term outcomes.
There are several ways in which program logic can support action throughout the program planning, implementation, monitoring and evaluation cycle.5,6 For example:
The NSW Health Guide to Measuring Value provides specific guidance about using program logic to support measurement across the quadruple aim of value-based healthcare at NSW Health.13
Monitoring is a continuous and systematic process of collecting and analysing information about the implementation and outcomes of a program.1 Monitoring processes can be designed to report on performance, inform ongoing improvement and help identify trends and issues or gaps for further examination through evaluation.
Program logic can support program monitoring in the following ways
Evaluation will usually investigate a broader range of information on program performance and context than is feasible to monitor on an ongoing basis, as well as address specific questions regarding the performance of a program.1 Program logic can support planning for an evaluation in the following ways:
Training sessions delivered to staff.
Quit attempts initiated
Reduced smoking rate
Identifying information needed to answer evaluation questions. Program logic can help in identifying the most appropriate information to answer evaluation questions by assisting in defining what constitutes program ‘success’. Success may be described in terms of the attributes of a program activity, output or outcome (e.g. quality, quantity, reach, timeliness, cost), and/or how it compares with agreed standards or targets.5 For example, an anticipated program outcome of “workers make sustained healthy lifestyle behaviour change” may be described in relation to attributes of the workers (e.g. age, gender), the specific changes in behaviour desired (e.g. physical activity, diet), and how ‘sustained’ is defined (e.g. number of months post-intervention). In this example success might be interpreted as “increased vegetable intake at 12-months post-intervention compared with baseline among male workers aged 30 to 45 years”. The success criteria will determine what information is relevant and useful for answering the evaluation questions, which in turn will inform the selection of data collection sources, methods and instruments.5
Helping to decide when to collect data. Because a program logic model depicts the expected sequence of activities, outputs and outcomes, it can inform decisions about the appropriate time to assess processes, shorter-term and longer-term outcomes. For example, program logic can help to identify critical preconditions for achieving outcomes, so that an outcome evaluation is not undertaken until it is clear that the preconditions have been met.5
Providing a mechanism for ensuring acceptability among stakeholders. A program logic model can help to ensure program stakeholders’ views concerning specific issues are kept in perspective in an evaluation by clarifying how these views relate to the overall program.5
This guide aims to support the development of program logic models and their use for monitoring and evaluation.
There are many ways to develop program logic models. This guide suggests an analytical and consultative approach that may be helpful for NSW Health staff involved in population health program planning, implementation, monitoring and evaluation. Ideally program logic should be developed in the program planning stage, although it can also be developed for an existing program.
A program logic model can support action throughout the program planning, implementation, monitoring and evaluation cycle. Program logic can be particularly useful for facilitating a shared understanding of how a program is proposed to work and what it is expected to achieve. This can help identify measures and indicators to monitor program implementation and performance and help determine what to evaluate, key evaluation questions, and information to address those questions.