Transcript of Translational Research Grants Scheme - webinar on research impact assessment
Stephanie Blows: My name's Stephanie Blows and I look after the TRGS program here in the Office for Health and Medical Research.
I'd just like to start by welcoming everyone. We've had a lot of interest in this webinar, which is really excellent. I'm not sure how many people we've got attending but there's a lot of you, so welcome and thanks for your interest. I'd also like to welcome our presenters today.
So as you can see from the agenda this has been a collaborative piece of work between the Office for Health and Medical Research, the Centre for Epidemiology and Evidence here at the Ministry, and also the Hunter Medical Research Institute, and I'll talk through the agenda in a moment.
So today we're going to be talking about research impact and how we measure that in TRGS. Research impact is a big focus for the NSW Government at the moment and it arises from Treasury requirements that we document the value of our programs and Dr Elizabeth Whittaker will be talking a little bit more about that in a minute. But basically for, especially for, our big flagship programs such as TRGS it's really important that we can demonstrate the clinical and economic value of the program and the research that we're doing and it's also important for TRGS recipients and researchers to be able to demonstrate the value and the cost benefits of the intervention that you're studying. It's important for implementation locally but also scaling to other areas in NSW.
So today on the agenda we've got, as I said, Dr Elizabeth Whittaker talking about some of the context for the research impact assessment work and then we've got Professor Andrew Searles from HMRI who's going to be talking about some of the principles behind research impact, the framework that we're using to do the work, program logic, and some of the principles of economics analysis. Then Brigitte Fienberg is going to talk about impact metrics and reporting that are specific to TRGS.
So, I'd just like to reassure you at this point that although a lot of this information might be new to you, what we require from you as a TRGS recipient will remain very similar to what we currently require. You'll still have the same progress and reporting schedules for your research and the same financial reporting requirements. We will be developing some new templates to better capture some of the metrics around research impact but the majority of the metrics that we'll be using are very similar to what you're currently reporting. Andrew Searles will be talking about program logic and we will require all TRGS recipients to complete a program logic, if you haven't already. We'll give you a template for that and, as I said, Andrew will sort of step you through how to complete those today. So, the new template will basically be very similar to what you what you're already completing, just set up so we can better quantify the impact that your research is having, and we'll send out the new template to you within the next couple of weeks. We'll also, you'll also see an increased focus on research impact at the application stage for new TRGS rounds moving forwards.
Tony, I think you're online, Tony Penna, did you want to add anything?
Tony Penna: Only just to reiterate that this is going to be a work in progress for all of us, it'll only improve over time, and we'd welcome your feedback as you do start going down this pathway. So, this is very much a two-way approach to things. We're going to inform and educate with what we're currently doing but we will evolve it with your assistance, so over to you guys.
Stephanie Blows: Okay, thanks Tony. We'll hand over to Dr Elizabeth Whittaker then from the Centre for Epidemiology and Evidence to talk about the context for research impact assessment.Elizabeth Whittaker: To help set the scene on broader government context that has led to us investing in this research impact assessment approach, firstly you might be aware that NSW Treasury is in the process of leading a number of government-wide reforms that all agencies must comply with. So, across a number of these reforms Treasury has indicated that there's going to be a much greater importance being placed particularly on economic evidence as they really are intending to use this economic evidence as a primary source of information to inform their resource allocation decision making.
So, to give you a couple of examples, Treasury is currently revising their Evaluation Guidelines and these guidelines specify how and when government agencies must evaluate their funded initiatives and this revision is now taking them on a much greater focus towards economic evaluations. Another example we're seeing in this space is that Treasury is currently implementing the NSW Government Evidence Bank. So, for those of you not familiar with the Evidence Bank, this is essentially for new government initiatives over $10 million and it captures key economic information from business cases and their evaluations to allow comparisons across initiatives. So originally Treasury had proposed that only cost benefit analysis or CBAs from these business cases and evaluations would be reported and therefore used in funding discussions, but fortunately we've now been successful in obtaining agreement from Treasury to allow Health to include evidence from multiple forms of economic analyses such as cost effectiveness and cost consequence analyses, and this is really important because it will give us greater flexibility to report on those outcomes which are important to our key stakeholders but may not be able to be monetised in a CBA.
So while this is a great outcome that we can draw from a range of economic methods now and we can use the one most suitable for the initiative we're looking at, there does remain this overarching requirement to produce some form of economic evidence from our funding schemes. So, to do this we have partnered with the Hunter Medical Research Institute who have been working closely with us to apply to our funding schemes their research impact assessment methodology, which you will hear shortly from Andrew Searles has an economic component. And to the best of our knowledge this is the first time an approach like this has been applied at the funding scheme level anywhere in the world. So really what we're doing here is a really exciting innovation for the measurement of health and medical research impact.
The final point I would like to make is that our goal from investing in this approach is to continue to secure and attract research funding which will presumably become more difficult as we enter a budget repair phase. But we do feel confident that with the research impact assessment approach that Andrew is about to outline that we will be able to demonstrate high level of accountability for these funds to executives in both Health and Treasury because we will be able to clearly measure how the funded research has made a real difference to clinical practice and NSW communities over time.
So, with that as the background I'm now going to hand over to Andrew Searles to provide some more detail on what is a research impact assessment.
Andrew Searles: Many thanks Liz and to start off what I was just going to do is just to talk a little bit about the way that we used to understand the value that we extracted from health and medical research so I'm using the abbreviation of HMR throughout this presentation. So, before research impact assessment the way that we typically used health and medical research was to look at theacademic achievements or specifically sometimes it was the trial outcomes. Now these are important, critically important, but those outcomes on their own don't necessarily capture the impacts that are important to other members of the community such as patients, governments, industry and so on. So, a research impact assessment is an opportunity to capture a broad basket of evidence around the range of facts that are going to be generated from health and medical research. We can apply research impact analysis to a research project, to a research program, but also to a funding stream and as I've said earlier these are much broader than the kind of evaluations that you might have typically built into a research trial, for example.
There are two core components of a research impact analysis, so, firstly, it's understanding what are the impacts. So, in a broad sense these are the returns or the benefits that have been made possible by the funded research. As I said, they do include trial outcomes and the academic achievements. They also though should include benefits that are meaningful to others in the community, whether that be patients, health services, industry, government and also importantly to the broader economy.
A lot of what we do in health and medical research has direct economic consequence but quite often we don't capture that. The other component of a research impact analysis is to look at cost, and to summarise what cost looks like there are two broad cost buckets. Firstly, the cost of undertaking the research which is typically research and development, and secondly there will be a cost of using the research outputs and what we mean by this, if you come up with a new model of care or a new device, something that we introduce into health services, there may be a cost to the users of that model of care that also need to be captured when we're doing an economic analysis or a research impact assessment.
And so, I'm going to talk a little bit about how to measure research impact. So, there are many research impact frameworks. Typically, when we did our literature review before we came up with the FAIT Framework which is the Framework to Assess the Impact from Translational health research, most of the impact frameworks that we found were based on a retrospective analysis and that is they would look at what was achieved at the end of the research program and go back over the five years, for example, or when funding might have started to see what actually happened, and the second point that we found from our review of the frameworks is that most of them did not include an economic analysis, and as we've been saying in this presentation, including an economic analysis is important to understand value, and this is becoming more and more important in NSW, particularly with the lens that NSW Treasury is putting over funding streams in this state.
A quick few points on the FAIT Framework. So, it is a means to capture a wider range of impacts generated from health and medical research. It draws on an economic evaluation methodology, a cost benefit or a cost consequence analysis and you would remember that Liz specifically mentioned those two techniques earlier on in this presentation. The framework that we use is based on both quantitative and qualitative methods and that's because even though we are basically quantitative researchers we understand that they don't always capture some of the intricacies or the difficult pathways that sometimes research takes to translate, so it's a combination of quantitative and qualitative. The FAIT Framework is recommended to be applied prospectively from the time that the research is being conceptualised and that's when we would typically recommend that you start to put together a program logic, which I'm going to go to in a couple of slides, but the FAIT Framework can be applied retrospectively, and we've done a number of retrospective assessments already.
The Framework reinforces a basic economic principle and when we speak with colleagues in our own network what we emphasise to them is that even though you might come up with fantastic research evidence of something that's effective and cost effective, if we do not get translation and the utilisation of that research evidence, the impact will be zero. So, in the framework that NSW Treasury are looking at, what we say to our research colleagues is that if we don't have translation utilisation it's going to appear that we're all cost and little if any benefit. So, when applied prospectively the FAIT Framework can be used to encourage researchers to conceptualise how they're going to translate and how their research evidence will be utilised, and hence there is an opportunity in a prospective application to incorporate nudges to both support translation and utilisation.
So just a brief overview of the methods that go into the FAIT Framework. Can I say at the outset there is no rocket science involved in this. We have used three existing methodologies that have been well used. The first one that we use is based on the payback methodology, so for people who have done impact assessments before you'll know that payback is, around the globe, is a very well used technique to measure impact. It's typically done retrospectively at the end of the research and the way that they measure research impact is often by forming an expert panel. So, when, and we worked with the researchers from Brunell where payback came from, we thought that this was a very expensive and time-consuming way to do an impact analysis so we tweaked those methods. We used domains of benefits such as knowledge generation, clinical care, policies and programs, the economy, community and health as broad buckets of the kind of impacts we were looking for and instead of using an expert panel we used metrics, and this was the ability for us then to start introducing process, output and impact metrics across the life of a research project.
The second methodology that we use is an economic analysis, so I won't go into detail of what the analyses are because there's a range of them that we can use but they're included to give us the ability to estimate the return on investment from funded health and medical research. And, as I said earlier, because the quantitative measures don't necessarily capture the complexity of translation and utilisation, we've also included a qualitative or a narrative within the Framework. But one of the key things that undertakes or underpins the techniques that we've got here, these three methods, is a modified program logic model. So, I'm sure everybody in this room will have used a program logic before. We tweaked the standard way of looking at program logic and we put an economic lens over it. Again, we suggest to be prospectively, both the logic model be prospectively designed and applied, and by using this logic model we can identify the process output and the impact metrics. And I'm just going to show you, in very broad terms, what the program logic looks like for research impact assessment, in very broad terms. At the end of this webinar we're going to provide a link of where myself and one of my colleagues, Dr Shanthi Ramanathan from HMRI, will take you through each of the steps of putting together a program logic like this. Can I say at the outset it is not a big task. It takes generally between 45 minutes and an hour at the most. If it's taking more than two hours something is wrong and that's when we say get some help and some advice, they're fairly straightforward to put together.
The way that we have viewed the program logic from an economic lens is to start with demand or a need. What is the problem that your research is addressing? Describe it, quantify it, is it common or rare, what does the needs analysis and the evidence synthesis show, and, most importantly, is this a priority for the health system? The aims and activities of what your research is delivering to address that need. So, this is, in fact, in effect what you're supplying. For every activity, and some examples of activity are on the slide and because we're making these slides available I'm not going to read them out word for word, but it might be, for example, that your research is going to be looking at developing a new model of care.
You might be undertaking some engagement with your end users, you might have a capacity building stream throughout your research, and when I talk about capacity building an example of that might be that you've got a postdoc program or you're employing PhDs to work on your particular project. These are all activities and every activity should have a particular product or an output. So, it might be that you've completed and analysed some trials, you might have new evidence on a new model of care, you might have undertaken capacity building that would be evidenced in terms of completed postdoc projects, completed PhDs etc. The very last point I've got on the slide there under key outputs is actually a business case for the model of care and I'm going to mention this a couple of times in the presentation because we're trying to also to give you some takeaway messages around the way that you can build in some economic tools or policy tools to also help your translation and having a business case is one of those tools. For every product from your research you should also be able to identify end users.
So, end users can be patients, the community, health services, decision makers or budget holders within the health service. It might be other researchers and we mentioned this because we are fully aware that the translational pipeline, or the research pipeline for that matter, is not linear. It actually is quite complex and it might be that your research outputs are actually going to be used by another research team. We also include, under end users, funders, people in policy and generally government departments. The additional column that we've got there is to get people thinking about the pathway to adoption: how do or how will your research outputs get to those people who are going to use them or utilise them? This is around translation: what does this pathway look like and also you might ask the question who else do we need, for example, on our grant to help successful adoption or adaption of our research findings? For example, it might be a patient peak body.
In going through this form of program logic, and as I've said there'll be a web link to a more detailed run through of how to actually complete this for a research project, we can then start to tease out impacts. Now we all know that impacts from health and medical research don't necessarily happen in the year that the project completes. Sometimes they could take a decade or more, so the way that we look at impacts is to break them into short term and we will build them into various domains of benefit and I've mentioned those domains of benefit earlier. So, it might be that you have for a short term a documentation of a new model of care or you've contributed to a guideline or policy advice, for example.
The intermediate impact would be that there's been an observable or an evidence-based change in practice, that a policy has shifted, that you've started to record some economic impacts, so for example employment on your research project. The final impacts though that we're looking for from health and medical research are typically around those for improved outcomes for patients. So this could be reported, for example, by quality of life or other clinical outcomes but we're also looking at those broader baskets of impacts that could impact on health services, businesses and the broader economy. So you also, when you're looking at the pathway to adoption, and I'm just going to mention it here, you might also be considering some of the translational frameworks, so for example you might be looking, depending on whether it's appropriate for your research, the RE-AIM Framework or for example the knowledge to action framework.
So, these are all of the kind of things that you'd be thinking about around the pathway to adoption. And this slide shows a summary of the detailed program logic that we've collaboratively put together for the Translational Grants Scheme and we've done this from the perspective of NSW Health, from LHDs but also from TRGS recipients. So, again this is going to be provided to you after the presentation so I'm not going to read it through, but it starts with identifying the need, it has some high level aims of the TRGS scheme and these aims relate both to those who are administering the scheme, so in this particular flow diagram, sorry in this particular program logic, you can see under activities I've got activities for the TRGS scheme and then underneath I've got activities for TRGS recipients, so this is a common group of aims for both of those groups.
We've separated them for both those administering the TRGS scheme and the recipients of TRGS. We go to the end users, there's an overlap of end users between those administering the scheme and those receiving TRGS funding. When we get to the path of adoption I've put a little bit more detail there for the TRGS recipients and there are some takeaway messages from this and, again this will be provided to everybody who is in this webinar and those who would like to download it. So, what we're suggesting is a pathway to adoption and this evidence comes out of the literature, that you have an upfront plan for translation and implementation scale and impact. That you understand what the engagement is going to look like with your key stakeholders; this might be patients, could be decision makers within health services. That you've got a pathway for translation, implementation, and scale up activities that are appropriate to the type of research that you're doing and again here, this is where you might be thinking about the translational frameworks that I mentioned on the previous slide.
Points five and six are quite important and again I think there's an important takeaway message here and this is around building in evidence of value from your research, so evidence of cost efficiency or affordability and if your research is proven to be cost efficient, that you think about how you're going to be building up a compelling evidence-based business case. Both points five and six are components of an economic analysis so I'm going to come to a little bit more detail on that in a future slide.
So, in terms of the impact, I've just given some examples here from the TRGS scheme and I've mixed in here the kind of impacts that we'd be looking for from both the TRGS scheme overall. So from the TRGS scheme overall they'd be looking at the inclusion of value as business as usual in health and medical research in this state, but for TRGS recipients we'd be looking at, for example in the short term, we're looking still at some of your traditional academic achievements, might be publications, presentations, academic, sorry, input to policies or to other broad domains of impact, we've got some there listed under clinical care, policies and programs, but importantly you can see we've got a basket of impact metrics there for the NSW economy, so this could be around, for example, more efficient and effective healthcare.
This is really important to think about with your research. With many of the clinicians that we work with they come up with models of care that are actually cost saving or cost avoiding so that's a benefit, it's an impact, it's important that we capture that. It might be that you're coming up with a new device or a new technique and it could be that there'll be revenue and jobs from that particular stream of work. There could also be jobs associated with your TRGS-funded research. So, for example, you might put on a research assistant, you might create a position for a postdoc; these are the kind of things that we want to start capturing because these are the kind of outputs that NSW Treasury will be interested in when we start looking at the benefits to the NSW economy.
The last basket of impacts that we've got there, of course those the broader ones around patient and population health under the domain of NSW community and health outcomes. So the way that we've put this program logic together also supports an economic analysis and so the activities of undertaking the research program, so activities for those administering the TRGS scheme, there will be costs associated with establishment and administration of the scheme, but the TRGS recipients are also going to have costs and effectively this is the cost of the research and development on the TRGS projects.
In terms of utilisation, and I've got these costs here listed under the pathway to adoption, so for the TRGS administrators this will be the cost of utilising the TRGS scheme: this is administration costs but also the ongoing monitoring and evaluation that will take place so that we can build in continual improvement into the TRGS scheme.
For the TRGS recipients there'll be a cost of affecting the research, that is how there might be a cost for other people to use your research outputs. There might be a patient-related cost or maybe the health service needs to contribute to the cost of implementing your model of care. An example that I often give to people is, for example, if you were to come up with a new drug, a new pharmaceutical, there would be an additional cost both to the Pharmaceutical Benefits Scheme in terms of providing those medications to Australian patients but there also could be a co-payment that's involved with that as well. So, there in the purplish colour, they're the cost side of the equation of research impact assessment that we're looking at. Under the impact column is where we start to list the benefits and where possible we will monetise these benefits but quite often in health and medical research it's not possible to do that monetisation and so and with Treasury's approval we're actually now able to list those benefits in their natural units. Many of them though, can I suggest, can be monetised so cost savings from a model of care that will improve the efficiency of the way that we deliver health services, we can usually estimate what downstream costs avoided would be. If you're capacity building on your grant we can actually look at the increase of productivity if you upskill somebody who has a master's degree to a PhD, we can estimate the value of that to the NSW economy, so we can monetise many of those benefits but clearly not all of them.
The very final slide is just around economic analysis, so I'm just going to make a couple of points here because all the way through talking about research impact assessment many of the points I've been making are actually how we can embed economics into both research impact assessment of the TRGS scheme but also the individual TRGS projects. So adding an economic lens to an evaluation introduces the concept of measuring and valuing the resources to do research and in an economic analysis we'll compare these costs to the measures of impacts. So we've taken you through those steps already. I've already given some simple comparison of costs and impacts on a previous slide. One of the points I would like to make here is that doing these kind of economic analyses are very doable and even though I don't get brownie points from my economist colleagues I just like to point out that the economics with many of these grants is not rocket science and you can build them in, you can build the economics in from the very beginning. The inclusion of an economic perspective is important information to decision makers so one of the take-home messages from this webinar is to consider an economic evaluation as part of your TRGS application, and remember that the inclusion of an economic analysis can also contribute to translation by helping you make the business case if you have come up with effective and cost-effective research evidence that the economics can help you with a business case to persuade decision makers that it will be a good decision to implement that within the health care system.
So, with that I would like to thank you Brigitte for turning the slides over for me and I would now like to, in fact, pass over to you Brigitte to talk about metrics and reporting.
Brigitte Fienberg: Thanks Andrew. So, yes, you're probably now wondering what does this all mean for me and so the main change will be to our reporting, however this is only going to be in a minor way, as many of you probably know that our current progress reports and final reports already align with the FAIT Framework quite conveniently. So, the main change will be likely to the structure of our template and this is just to better capture some of the key impact metrics which we've got listed here that cover, I guess, five key domains which Andrew had mentioned: so that's knowledge generation, capability building, policy and practice, patient health and population outcomes, and of course economic outcomes.
And so, I guess the main thing to keep in mind for these impact metrics is that they will directly relate to your TRGS project so we're interested only in metrics that relate to your particular project. So for knowledge generation we're interested in any new research findings, we're interested in the number of peer-reviewed publications and the impact factor, so that would relate to the strength of the publication, the inclusion of findings in any meta-analyses or other reviews, the number of citations, the number of presentations on the project, so that could be sharing new knowledge at a conference, it could be training a particular audience in a new technique at a workshop, or it could be providing a recommendation to Ministry or Local Health District decision makers in a meeting.
In terms of capability building we're looking for the number of collaborations on the TRGS project and what their contributions would be, the number of training and professional development activities by the research team members, the number of research students that are supported in the project, so these could be, as Andrew mentioned, PhD students, masters students and those that are supported through TRGS funding directly, through other NSW Health funding, but also through other external organisations.
For policy and practice we're looking for instances where research findings have impacted policy and/or clinical practice, usually they interrelate, within the LHD, other LHDs, or statewide. And so, we're looking, as Andrew had said before, for TRGS projects that propose say a new model of care, an intervention, a new technology, have they been adopted within the LHD or have they been scaled across LHDs or statewide.
For patient health and population outcomes we are interested in improved health outcomes so these are the, I guess, the direct or the projected benefits to patients. So, for example, if your new model of care has reduced the probability of a life-threatening event occurring then we're interested in that. If your TRGS project proposes a new intervention which is saving a number of lives then we like to see that. So we are interested in, I guess, our TRGS recipients quantifying the impact but also providing us quality qualitative information as well around these impacts. In terms of reduced health inequities for priority populations we're interested in this and we're interested in knowing who the priority populations are and how have you reduced their health inequities through your TRGS project. So, have you improved access to a particular treatment or have you improved health outcomes in any way?
For economic outcomes we're looking for the number of research positions that have been funded on the project, so that could be through TRGS funding or it could also be through external sources. A number of other grants that have been obtained through the project, so that could be NHMRC or MRFF grants and we're really interested to know whether those grants could have been obtained without TRGS. That's really important for us. We're also wanting to collect data around cost savings, as Andrew had mentioned just before, to the health system, so this could be improved health system efficiencies, it could be through reduced health service utilisation and so we are really looking for our TRGS recipients to include an economic analysis within their projects and this is very favourable for funding as well, so we like to see it right from the application stage and to see how that analysis is going throughout the reporting phase. And we're also interested in understanding the costings for whether it's a model of care, an intervention that's being proposed, or a program and how that compares to standard care or standard delivery.
The final metric is the number of commercialisable findings, so some outputs we're looking for are patents, agreements with commercial partners and any revenue generated, so whether that's through training or resources or consultancies. So I understand that many of the metrics that relate to policy and practice, patient health and population outcomes and economic outcomes may not be realised until the final reporting stage, but, or even many years after as Andrew had mentioned before, but that's okay and we are very interested in understanding the pathway to impact. So as we do with our usual progress reports, we'll be looking for progress against regular milestones and also the implementation plan which is submitted at the application stage so we'll still be looking for those key translation activities.
So, in terms of reporting the next thing that we're going to do is to send a reporting template to you and we're looking to do that before the end of the month. So the way that will work, the template should be suitable for both progress and final reports, so it means that recipients will simply need to update their previous progress report when they're at each reporting stage, so hopefully this should make things a bit easier both for the recipient and for OHMR.
Return to Online presentations
