EU guidance on statistical analyses in joint clinical assessments

The European Commission has published guidance on dealing with multiplicity issues and complementary analyses in Joint Clinical Assessments (JCA). The guidance, adopted on 10 June 2024, is aimed at reviewers and members of assessment teams.

The aim is to allow Member States to draw their own conclusions at the national level without favouring a particular approach, with the recommendations implicitly assuming that health technology developers (HTDs) provide appropriate analyses and information.

The guidance defines important terms such as multiplicity, subgroup analyses, and sensitivity analyses. It emphasises the importance of controlling first-order error in multiple statistical tests and explains different approaches for doing so. For subgroup analyses, the use of appropriate interaction tests is recommended, while for sensitivity analyses, the focus is on examining the robustness of the results.

Multiplicity Problems

Multiplicity arises from performing multiple statistical hypothesis tests, which increases the risk of false-positive results. The guidance distinguishes between:

• Multiplicity due to multiple endpoints
• Multiplicity due to interim analyses
• Multiplicity due to multiple treatment groups

To address multiplicity issues, various statistical methods to control the familywise error rate (FWER) are recommended.

Interim Analyses

Interim analyses are analyses of study data on efficacy that are conducted before the formal completion of a study. They can lead to multiple analyses for a specific endpoint before the end of the study and should be defined in advance (and their impact) if necessary. The following additional requirements apply, among others:

• They have ethical and scientific advantages but may also have methodological disadvantages, including a potential increase in type I error if not handled appropriately.
• The results can be used to make decisions about early study discontinuation. Reasons may include:
  • Clearly demonstrated superiority of the investigated treatment(s)
  • Confirmation that superiority is unlikely
  • Unacceptable side effects

Special reporting requirements apply to interim analyses, such as the indication of the cut-off date and the reporting of effects on the conduct of the study. In the case of premature study termination, the termination rule should be specified. If the final analysis is not yet available, the results of the last pre-specified interim analysis must be reported.

Complementary Analyses

Complementary analyses are defined as:

• Subgroup analyses: To identify differences in treatment effect between patient groups. The use of appropriate interaction tests is recommended.
• Sensitivity analyses: To investigate the robustness of the results to changes in the assumptions. Particularly important in the case of missing data.
• Post-hoc analyses: Unplanned analyses conducted after the study results are known. These should be clearly labelled as such and interpreted with caution.

Subgroup Analyses

Subgroup analyses are analyses performed to investigate whether the treatment effect differs in different subgroups of patients. These subgroups are typically defined by basic characteristics such as age, gender, or disease severity. The guideline emphasises that subgroup analyses should be interpreted with caution and recommends the use of appropriate interaction tests.

Sensitivity Analyses

Sensitivity analyses are conducted to examine the robustness of study results to changes in assumptions or methods. They are particularly important when there are uncertainties in the data, such as missing values. Sensitivity analyses can take various forms, such as the use of alternative statistical methods or the inclusion or exclusion of certain data points.

Detailed reporting requirements are a key aspect. These include:

• A clear description of the statistical methods used
• An indication of whether analyses were planned in advance
• The provision of all relevant results with corresponding statistics

Special requirements apply to multiple data cuts, interim analyses, subgroup analyses, and sensitivity analyses.

The guideline also addresses specific challenges in evidence syntheses, such as dealing with multiple endpoints or time points. It emphasises the importance of pre-specifying analyses and discusses ways to address multiplicity problems in meta-analyses.

Summary

Overall, the guideline aims to improve the quality and consistency of JCAs by providing clear guidelines for dealing with statistical challenges such as multiplicity, subgroup analyses, and sensitivity analyses. This should make it easier to assess the robustness and reliability of the results presented and thus enable more informed decisions to be made in the healthcare sector.

This information is important for companies planning clinical studies to prove the positive care effect of medical devices in order to have the positive care effect or the medical (clinical) benefit recognised by national authorities. In our experience, these requirements are also increasingly being made in this regulatory area, even if they are not yet to be found in the corresponding documents.

Reference and more details: https://health.ec.europa.eu/ document/ download/f2f00444-2427-4db9-8370-d984b7148653_ en?filename=hta_multiplicity_jca_guidance_en.pdf (delete blanks for link).

You want to use your data to ensure that the high regulatory requirements are met according to European regulations? Contact us for an initial consultation.

‍