Research Impact Assessment: The Framework to Assess the Impact from Translational health research (FAIT)

Transcript of: Research Impact Assessment: The Framework t​o Assess the Impact from Translational health research​​

Andrew Searles: I'm Andrew Searles and [I’m with] my colleague Shanthi Ramanathan and we're both from the Hunter Medical Research Institute up in Newcastle.

So what we're going to do is take you through the methodology that we're using for the impact assessment. The methodology is based on the Framework to Assess the Impact from Translational health research – we call it the FAIT Framework. That Framework is actually based on three proven and relatively straightforward methodologies. The first methodology that we're using is based on the payback model, so for those of you who have done impact assessment before you'll know that the payback model is actually the world's most commonly used technique for doing an impact assessment, however it is actually reasonably difficult to implement so we've simplified it, and the way that we've simplified it is to actually introduce metrics and broad domains of benefit that we'll report the impacts under.

The second methodology that we use is an economic analysis and the reason that we've included an economic analysis, apart from the fact that we're health economists, is to enable us to report return on investment, and this is really important so that we can actually explain what the value has been from the investment into health and medical research.

Because these quantitative techniques don't always tell the full story about impact or translation and impact assessment, the third methodology that we use is a narrative of translation and how that translation has led to impact. This allows us to tell those more complex stories of how research products, research outcomes, have actually translated to be used by the end user and then generate impact.

Underpinning these three methodologies is a program logic model, and the program logic model that we use is probably a little bit different to those that you've seen before and, so if we go to the next slide, and I'll just show you what the program logic looks like. Now this is intentionally blank so we haven't left anything out here. The way that we've developed this program logic model takes an economic lens. It starts with identifying what is the need or the demand and then it looks to see what are the researchers, what are the research activities and products doing to address that demand or that need, and in economics if you generate products the way that you form impact is that somebody needs to utilise or use those products and so you'll see here that we've identified end users and a pathway to adoption and we put that in before we start identifying the kind of impacts that are generated.

When we go through this logic model the first thing I'd like to tell you is it is not rocket science, it's actually reasonably straightforward to do and as you do it for your project, if it's taken more than two hours then something's wrong and let us know because it should really be a really quite short and straightforward exercise.

The benefit of this logic model is it helps describe the translational pathway: that is the pathway to impact from your research. The approach that we take identifies broad metrics described as process metrics, output metrics and impact metrics, and importantly the kind of measures that we're using include both your standard academic measures that you would be used to using such as peer review journals and conferences but they also include metrics for other end users and those end users will include patients, the health service, the government but also the economy, so there is a range of impact metrics that we're including.

In terms of the project that we are going to provide or use as an example to build up this logic model, we're not selecting one from COVID because that would be unfair to participants in the room if we selected one project. We've deliberately selected a non-COVID project but it's a real project, it's one that Shanthi and I have been working on and it's from the area of stroke rehabilitation. The project that we're going to use as the example is a clinical trial and it was a trial that was conducted to examine whether task-specific arm training for people who have had a stroke and lost use of their arm was effective and cost effective.

So, at the outset what we're taking you through today, it's applicable to both the stroke project but the lessons and the principles will be equally applicable to each of the projects that you're working on.

So let's make a start by identifying what is the need for your research, and as an outset if I could just identify that the need for your research is actually not the research question. It should be phrased in terms of the problem that you're trying to address. If your research is a success the impacts should directly address the needs or the problem that you've identified in your program logic model.

A couple of key questions that you might want to ask: firstly, what is the problem that you're trying to solve? Is there any existing literature or evidence about the problem and/or the need that you're trying to address? Who, or what if it's a health service, is being affected? And what's the nature of the problem? And you can go in to describe what the nature of that problem is. And, if possible, describe the size and the magnitude of the problem and whether it's a priority for the health service.

So, let's take these broad principles and apply them to the exemplar stroke project that I mentioned before. As an example the need for the stroke project that I identified before was that only 50 percent of stroke survivors regained functional use of their affected arm, and those who didn't were unable to perform activities of daily living, they had reduced participation in all spheres of life including employment, they had reduced quality of life and what was needed was better patient outcomes in terms of increasing the functional use of their affected arm, and it so happens on this project an additional need was identified and that was that the evidence through Cochrane reviews of how to improve arm function using task-specific training was actually very low so one of the needs was to actually raise the level of evidence or knowledge in this space. And at this point I'd now like to hand over to my colleague Shanthi Ramanathan who is actually going to talk to you now about the aims.

Shanthi Ramanathan: Thank you Andrew. In this next section we'll need to look at the aims for your research, which is just articulating what your project's going to do to actually address the needs that you identified earlier.

The first thing to make sure is that your aims actually relate back to those needs and achieving those aims should then result in the meeting of those identified needs. A good tip to remember is that your aims are not your hypothesis, if you happen to be doing a clinical trial. And I just wanted to also point out that your aims are meant to be aspirational, so you might not meet all your aims. Some of the aims that you will list will relate directly to your research, where some of them will only result if your research is translated, so only achievable potentially in the longer term, and remembering that to achieve these aims will mean that you are addressing the needs that you have actually articulated earlier.

Let's have a look at the exemplar project to actually have a look at how these are expressed. If you look at the first aim it's the obvious one that relates very much to the reason for this clinical trial, and that's to increase the amount of use and motor control of the affected arm after stroke. That's pretty straightforward. The second one is around the sustainability of that improvement over time. And then we start to move into those more downstream aims. So because we're interested in the improvements to the patient we look at things like the increased participation, the improved quality of life, and we're also interested in scaling up this in the future, if it's successful, so we also want to understand the barriers and enablers to uptaking this and how much it's going to cost to implement versus the cost of what would have been usual care, because that is going to be valuable information for decision makers and clinicians that are wanting to actually implement task-specific training. And finally, if we want to adopt the task-specific training into the guidelines so it becomes recognised as best practice, and those guidelines are actually audited by the Stroke Foundation every two years, so then you will get to understand the proportion of services that are actually delivering your task-specific training.

The next thing that we're going to look at, after you have established what your aims are, are the activities for your research. A good tip is that these should be articulated in your project proposal or, if you are lucky enough, a project plan that you might have developed, and they again relate directly back to your aims. You can either have one or two activities per aim or it could be that one activity relates back to several different aims. I find that a good trick to use is to express those activities using verbs, things like 'submit' and 'recruit' are action words that just help to keep you on track.

Let's look at how we apply that to the exemplar project. What we have here is what you would normally see as a common list of activities for a clinical trial. You usually start with your ethics approval and registration, you work through all of the different aspects of applying the trial, including recruitment of participants and actually delivering the intervention, and you end up with the analysis of the data and the findings and potentially disseminating those a bit further. Now, should your project be selected for the deep dive that Liz had mentioned earlier, these activities will also help inform the cost of the research and these lists of activities can also serve as a simple project plan. So, if you think about any one of those activities and you nominate the time when that will occur, like you would in a normal Gantt chart, and where applicable some targets for each of these activities, so for instance if you're training the therapist or recruiting participants in this case you might have had to train 10 research assistants and recruit maybe 300 participants, then what you have there is a set of process metrics, an action with a time it needs to occur in, and then a target of what you need to actually achieve.

And that takes us to the end of the activities and now I'll hand back to Andrew to take you to the next stage of the program logic model.

Andrew Searles: Terrific, thank you Shanthi. And so Shanthi's just been taking you through what we've actually called the process metrics that are typically associated with the activities that you're going to do. I'm now going to start talking about the output metrics.

In this section what we're asking you to do is to specify the key outputs or the products that are produced from your research activities. Some key points to consider: firstly, an output is something that somebody else could use or benefit from that's generated from your research. For every listed activity on those previous slides that Shanthi was just talking about there should be an output or a product and it might be that some activities have several outputs or products. For each output you should be able to also nominate who the end user is going to be and this is really important and I'm going to come back to this point in a second but let's just take what we know about the output metrics and apply them to our exemplar project. So from our stroke project, and I'm not going to go through all these in detail, but what I want you to notice here with the example outputs that we've included here, some of them are some of the things such as, for example, that you've registered your clinical trial, that you've developed training packages, that you have identified and documented your clinical trial outcomes. These are some of the outputs that you might not normally be thinking of in terms of a research process. Some of the standard academic ones are also included in here which is peer-reviewed publications, conference presentations, policy briefs and submissions to, for example, to stroke guidelines. We've also put in here that it might be relevant for you to identify cost effectiveness and also barriers and enablers to uptake or translation of your research products.

In terms of the outputs, just noting that we're looking at both academic and non-academic outputs. And so now we're going to have a look at end users. So the end users are the people who are going to utilise your outputs or products, they're going to benefit in some way from your research. And two broad groups of end users you might like to consider: firstly, the implementers. Those are the people or the groups who will need to implement your research outputs or your research products. And then there's also the beneficiaries, and these are the people who are going to benefit from the research project. So this could be patient groups but depending on what your particular research outcomes might be it might be the health service that actually benefits.

A couple of key points to consider: firstly, the definition of end user is broad. It includes other researchers so it might be that one of your research outputs is actually passed to other colleagues, other researchers working in the same area. End users could include the health service, other providers, patients, communities and also, importantly, it could include industry so if you're commercialising your output your end user might be pharma or devices, and it also could be groups such as government. What we suggest is, think about your program and who needs to be involved so that the findings are used, and these points two and three here are really critical. We recommend to people that you engage with your end users very early in your research process so that they know what's likely to come down the research pipeline because they might be able to give you some insight into what does this product need to look like for us to be able to use it.

So keeping that in mind let's go and have a look at the end users for our exemplar project from stroke. Again I'm not going to list these but we've broken the exemplar project down into 'implementers', which in this instance could include therapists, health providers etc and into 'beneficiaries', patients, carers, general public etc. So with the end users and giving you some examples there I'd now like to hand back to Shanthi who's now going to talk about the next step in the logic model.

Shanthi Ramanathan: Thank you Andrew. I can't believe how fast we're skating through this. We've come to now the pathway to adoption of the actual logic model and this is actually a new section that we've added to the FAIT logic model because in working and applying it to researchers we found that that's something that a lot of people have not traditionally done and it's there to help you think about the translational activities that you need to undertake in order to maximise the potential that your research will translate and have impact.

Some of the key points you need to consider is how will you get the findings and outputs from your research to those who can use it, what is the actual pathway and the activities on that pathway for this to occur, and who else might need to be involved for that successful adoption?

Now let's have a look and go back to the exemplar project to have a look at what this looks like. As Andrew mentioned there's just too many things here for me to actually tease out so I'll just look at a couple to just walk you through what happens. Because this is a project around implementing an intervention called task-specific training these training manuals that were developed have got much greater use beyond the project but to be able to do that people need to have access to this and so being able to make that actual resource available online, for instance on a website, that can be downloaded, will help to actually increase the uptake of that intervention. Similarly, taking that time to actually contact occupational therapy schools to inform them of the outcomes of the project and where they could download the manuals might also help to get them to think about including it in the curriculum for occupational therapy students so that they're actually trained in task-specific training by the time they get their graduate certificate.

The second example here I wanted to look at very quickly was just the policy makers, so in this particular project there was direct engagement with policy makers and it's great to see in the C19 program that this is already happening via the emergency workstream where they're already linked, researchers are directly linked with these policy makers. In your situation I think it's the opportunity that you have with the translation workstream to create these evidence briefs from your work and directly target policy makers, is an excellent example of some translational activities that you can include in your logic model.

So now we get to what I like to call the end of the rainbow. We've been on this journey of developing that pathway and we get to the end where we hope to find that pot of gold. At this point, because we're doing this prospectively, it is an imaginary pot of gold, but these are the things that we're hoping to achieve at the end of your work in terms of your research if you follow that pathway. These impacts are really benefits and consequences of your research being translated.

Because the logic model underpins FAIT we try and express the impacts using those payback domains of benefit. Now they're generally five domains that are commonly listed in payback and I've added here the capacity and capability building which is a common one that most projects generate impact in. You can also add other customised domains depending on your project. For example if your project has community engagement as a vital component of your research then you can include, for instance, an engagement domain where you actually think of the impacts of that engagement in your project.

Now let's just quickly have a look at the intended impacts for our exemplar project. Just a quick note here: impacts don't all happen immediately. Some can occur during the project, some can happen soon after, and some are actually much further downstream. It is useful in addition to those domains that I talked about to also think about these timeframes when you're working out your impacts, what you might expect to see in the short term, the intermediate and the longer term. Under knowledge advancement here we have key findings from the research in terms of task-specific training, the barriers and enablers to uptaking, and the cost effectiveness of actually delivering the intervention, but I've also got here some outputs and products and their usage as key impacts in this domain. Under the domain of capacity building we have the trained therapist but there's also some training that's occurred in the conduct of clinical trials, such as with your research assistants, your coordinators and your PhD students which has built that capability which will continue long past your project. Ideally they would like to also be able to evidence that task-specific training has been included in the stroke guidelines as a policy impact as well.

Under the adoption, under practice impacts we would hope that task-specific training is actually adopted beyond this trial and this is one of those longer-term impacts that might not happen straight away but it's certainly something we would like to track and it links very much to that inclusion, as I said, in the guidelines in this case because the audit will tell us if that's actually occurred.

In the domain under community and patient impacts, which really relate very much back to those needs that we identified earlier, we would want to see the improved recovery of those stroke survivors, greater participation in everyday activities, potentially improved employment, their quality of life, and also even some impacts around the carers and reducing that carer burden.

And finally in the domain around economic impacts, if all of these things actually occur then we potentially might be able to evidence some reduction in health systems cost from a reduction in rehabilitation costs, increased productivity for those people returning potentially to work as the survivors but also the carers, and then a potential reduction in disability support that's being paid out by NDIS or other private insurers.

So that in a nutshell has brought us to the end of the pathway and what you see here is that earlier template that Andrew had showed you now populated with the data that we've just walked you through, from the needs all the way to the impacts.

Looking at this it looks like a big piece of work but I can assure you that when we did this piece of work it took us about an hour and 15 minutes. It was a face-to-face session with the researchers, and can I encourage you when you're doing this to actually have your team together, to have someone facilitate the discussion and someone scribing because I think that is the best way that you can get through this activity. And as Andrew said, if it takes more than two hours please let us know because it really isn't such a difficult activity if you walk through the steps that we have spoken about. ​​​