In the second article of this two-part series, Jasmine Kestemont from Innovion delves deeper into the cost-benefit analysis, focusing on the upfront investment required to implement CDISC standards at study setup and the break-even points for three different levels of standardization.
In part one of this paper we discussed the benefits of implementing standards in the set-up phase and demonstrated the benefits of implementing standards in a single trial. We concluded that 2 parameters had the largest impact on efficiency gain – complexity of trial and level of standardization. We concluded that the higher the standardization level, the smaller the trial set-up effort and the absolute gain was higher for complex trials than standard trials.
The implementation of standards requires an upfront investment, and to no surprise, the higher the level of standardization, the higher the investment.
In part 2, we will look further in the investments required to develop standards and determine the break-even point.
Basic Standardization: A company that is new to implementing standards, will usually opt to implement the domains well described in the CDISC SDTM Implementation Guide first. We would recommend creating visualizations, add the associated controlled terminology and CDASH/SDTM annotations and define conventions, such as reference dates and basic computational algorithms. This level of standardization will use material already available and does not require any significant technology investment.
Figure 1: snapshot of a visualization of basic standardization
Advanced Standardization: The next level of the standardization effort will include the definition of regularly recurring disease area specific assessments and endpoints and the associated metadata.
The main goals and advantages of an advanced standardization are the re-use of library templates across studies and unambiguous communication between sponsors and vendors. Sponsors will be able to define study expectations in a common human and machine-readable format. In view of submissions and exploratory analysis, the consistency across trials will facilitate data pooling.
Figure 2: snapshot of advanced standardization
Full Standardization: The goal of this level of standardization is to provide full data specifications from data capture through storage and analysis (the latter not in scope of this article). While theoretically it would be possible to achieve this through common tools, such as excel, in practice it does require the investment of a metadata repository (MDR) to develop and maintain standards. Unless you have a dedicated study specifications team that fully understands the construct of an MDR, you would also need a user interface to allow study specific selection of data collection forms and the associated metadata.
Figure 3: snapshots of a metadata repository
Calculating the break-even point
Any level of standardization requires an upfront effort. In this section we will compare the effort of the investment and the gain against the same trials without standardization. The following parameters were considered to calculate the effort:
- developing standards
- governance and maintenance of standards
- technology investment
- process change and implementation
Benefits of standardization include:
- quality control, including user acceptance testing
- Basic Standardization vs No Standardization
Basic standardization requires a relatively small upfront investment, common tools such as MS Word will suffice. As it primarily focuses on the domains that are well-defined and thus stable, maintenance of the standards is limited. The largest gain in this process is facilitated communication and a reduction in review effort, but there is limited opportunity to automate any parts of the process.
Figure 4: oversight and quality control of an outsourced trial with basic standardization
Nevertheless, as already shown in part I (http://www.innovion.be/news-and-events/news), we do see a recurring benefit of any level of standardization, and this effort can easily be completed by anyone, new or experienced in the development of standards.
When comparing the effort and investment to set up a trial with basic standardization vs no standardization, one can see that, despite the upfront investment, the total budget for setting up 5 trials is cheaper with basic standardization than without standardization.
Figure 5: break-even point of basic standardization vs no standardization
- Advanced Standardization vs No Standardization
Building advanced standards, looking at the needs of your (or your Sponsor’s) trials will require a higher investment in skillsets, training and possibly process updates, on the other hand, the per trial benefit is larger. The need to invest in technology is limited (something as basic as MS Excel will do), provided the team is disciplined and adheres to the standards agreed upon.
Other than providing expectations to your partner which facilitates communication, there are 2 major advantages in set-up. Firstly, it is acceptable to perform user acceptance testing at standards level. For the forms UAT’ed, it is not required to repeat this effort at trial level, provided there is no change to the form. Secondly, it is possible to compare what was provided to the CRO and what has been received in a semi-automated way, thus reducing your quality control efforts and overall oversight.
Figure 6: oversight and quality control of an outsourced trial with advanced standardization
This translates to an overall larger effort in the definition of standards, but reduced effort in the per trial set-up. While the initial effort is larger, because of the gain per trial set-up, we again see that the overall budget for setting up 5 trials with advanced standardization is lower than setting up 5 trials without standardization.
Figure 7: break-even point of advanced standardization vs no standardization
- Full Standardization vs No Standardization
Finally, we evaluated the effort of full standardization, where the sponsor will provide exact specifications for the trial and will clearly define what to expect back from the CRO. To be able to do this, the sponsor will need to invest in technology (MDR and automation), process updates, monitoring and training to facilitate this. There are multiple technology partners with varying capabilities and associated costs. The below business case was developed using Formedix.
In the calculation, we collated all Standards Development efforts into 1 bucket (standards development), regardless of whether this was done for an individual trial. In practice our experience has been that the effort is concentrated in the first 2 trials and from the 3rd trial onwards there is a significant return on investment. This is because you can build most of your standards based on experience, but the details of trial specific components will only be built as the protocol becomes available. This effort would not go away in any of the other models, but in this model, the responsibility lies with the sponsor or their designated standards development partner. Important to note in this model is that the entire effort of trial set-up is front-loaded, possibly even before first engagement with the CRO.
There are 2 major benefits to this model. It is possible to generate the specifications in such a way that an automated build of the EDC is possible. If the system is properly validated, then the per trial level validation and user acceptance testing can be significantly reduced and the QC can be automated. In practice this means that you can have your EDC and SDTM/define.xml completely defined prior to FPFV, allowing the team to focus on trial content from that point forward.
Figure 8: trial oversight and quality control of an outsourced trial with full standardization
Despite the much higher upfront investment, we were able to achieve the break-even point after 6 trials in comparison to no standardization at all. This did not take into account the activities associated with an automated study build (we did not assess this part yet, but look forward to including this in a future analysis).
Fig 9: break-even point of advanced standardization vs no standardization
Calculating the return on investment
So how would you choose which level of standardization is the right level for your company? It is possible to move from one level of standardization to the next and any effort can be carried through. Figure 10 summarizes the long-term return on investment, beyond the initial break-even point. In summary, the more you standardize, the higher the quality, the higher the potential for automation and the higher the return on investment.
Fig 10: Return on Investment
Thank you to the Formedix team for an in-depth training on building a Metadata Repository and allowing us to use their tools to test and assess the impact and feasibility of full standardization.