In a two-part series, guest blogger Jasmine Kestemont, Managing Partner at Innovion, shares her findings from a cost-benefit analysis for CDISC data standards implementation during clinical trial setup. In this article, Kestemont reviews study setup activities from protocol to Define.xml creation, highlighting that any level of standardization has a direct and immediate impact on the effort required for trial setup.
Typically, a return on investment analysis looks at specific components and evaluates the cost to build something to replace the activity and the amount of time it takes to complete that activity (with or without the implementation of the new component).
The below analysis includes a retrospective and prospective analysis based on real data as our industry has evolved through different levels of standardization and technology implementation. It provides insights into why we should see huge benefits in standardization, but do not see the same results when putting the effort into practice.
The scope of the analysis includes the activities related from the design of the study (protocol or protocol outline) until creation of SDTM and Define.xml. In the calculations “Effort” includes the study design discussions, EDC design and build, integration of electronic data streams, User Acceptance Testing, creation of Documentation and mapping or conversion to SDTM, including creation of define.xml, where data managers, database designers and/or programmers are involved.
Figure 1: Scope of activities included in assessment
The analysis compares 3 levels of standardization (50%, 80% and full standardization) against no standardization.
To evaluate the true cost of trial set-up, variable costs not associated with trial set-up effort (such as EDC hosting) were excluded, and hourly rates across cost proposals were harmonized to exclude regional variability. Effort calculators were built to model the cost of a trial. To compare trial set-up effort of the different levels of standardization, the following parameters were fixed:
- # subjects
- # sites
- Trial Duration
For each level of standardization the effort calculator was run to see the effect on complex trials (multiple or complex efficacy endpoints, phase II/III trials in complex disease areas) and on standard trials (phase I trials, non-interventional or well-defined endpoints). These results were plotted in a graph.
Three main conclusions were drawn from the data
- Any level of standardization will result in a reduction of set-up effort and the impact of standardization is higher for complex trials (fig. 2).
This should not be surprising. Once standards are defined, the effort does not have to be repeated, provided there is a discipline within the team to adhere to standards.
As the absolute cost of a complex trial is assumed to be higher than the absolute cost of a standard trial, the opportunity to achieve a benefit is higher in complex trials than in standard trials. Often more people are involved and simply not needing to have a discussion about something in a large team meeting creates a bigger benefit than not having the discussion in a small team (calculating man hours).
Figure 2: Conclusion 1
- The more you invest in standards, the bigger the impact (fig. 3)
So while a basic level of standardization will already give immediate benefits, the reality is that when developing standards, most teams will start with defining “the obvious”. The more detailed the standards development becomes, the more likely that complex issues, such as endpoint definitions are defined. While the immediate gain means that fewer topics are rediscussed, one can also introduce efficiencies by changing training and validation procedures (validate once, use many times). Lastly the QC effort is significantly reduced and may even be automated if you can verify deliverables against pre-defined specifications.
Figure 3: Conclusion 2
- There is a finite benefit to be gained from standardization (fig. 4)
This part is often overlooked and is the reason why statements on the extreme benefits of standards are challenged. The purpose of doing trials is to gather insights through data collected, therefore, as long as a company wants to learn, it will have to introduce new science, new hypotheses. This implies that a library of standards is never static. Using standards to set-up trials will bring the discussion forward, but the discussion still needs to take place. There seems to be a “golden rule” in our industry that there is a 12 week period between final protocola and first subject recruited, and unfortunately this is where we lose momentum. Most humans work best with a deadline
Figure 4 : Conclusion 3
Any level of standardization will generate an immediate impact on the effort associated with trial set-up.
In part II of the cost-benefit analysis, we will incorporate the effort required for standardization and determine the break-even point of the cost of trial set-up with different levels of standardization against no standardization (fig. 5)
Figure 5: Investments included in Assessment to calculate the break-even point