Subgroup analysis is routinely conducted in drug development, in various settings; one key aspect is the regulatory requirement to demonstrate consistency of treatment effect across a pre-defined set of subgroups (e.g., ICHE5, E9, E17). This is performed as a risk-benefit assessment, aiming to identify the right patient population to treat - and, here, that set of subgroups is agreed with regulators prior to the trial conduct. Another key aspect is subgroup selection, where the aim is to estimate the effect in the most promising subpopulation (typically for planning another trial). The latter can either be done with respect to the same fixed set of pre-specified subgroups as mentioned earlier, or in a data driven fashion (e.g., biomarker subgroup detection).
There are well-known inherent statistical difficulties with all the above; with consistency, due to limited data in the subgroups, there is a high risk of false positives (random highs) as well as a low power to detect true differential effects (since trials are seldomly sized for it). In the subgroup selection setting, it is of key importance to provide an honest estimate discounted for the number of subgroups inspected, in order to not overstate the real effect. Even in a consistency assessment setting, there might well be a certain tendency to focus on the most deviating subgroup results, hence possibly introducing a bias although not formally a 'selection' problem.
The PSI Subgroup SIG is devoted to methodologies and aspects around these questions, with a focus on questions related to assessment of a fixed, pre-specified, set of subgroups. As such, the SIG has not yet explored the application to data-driven biomarker subgroup detection area (data mining), although some of the approaches investigated could be used for this purpose.
The work aims at providing as much guidance and clarity as possible on the inherent issues and possible approaches to analysis and subsequent detailed investigation of pre-specified subgroups in order to provide context for any subgroup findings.
Numerous methods have been suggested in the literature and ranges from interaction testing, permutation-based ordered statistics, Bayesian shrinkage, Bootstrap bias reduction, model averaging and graphical methods. One inherent difficulty, making the analysis less straightforward than it might first appear, is the fact that many subgroups will overlap. Also, non-trivial aspects arise from some subgroup factors being prognostic.
The PSI Subgroup SIG submitted a White Paper in May 2018 on some of these aspects, containing an overview of the inherent problems, recommendations for the planning stage, a novel permutation based approach for assessing expected deviations under a null assumption, and some simulation based conclusions where various methods were compared.
Due to the complexity not all available methods were initially studied (e.g., the Bayesian ones) and further work is being conducted. The aim is to provide an updated document later when these methods have been developed and evaluated.
Who we are:
Björn Bornkamp, Aaron Dane, Christine Fletcher, Ilya Lipkovich, Henrik Loft, Brian Millen, Heiko Goette, Necdet Gunsoy, Tom Parke, Arne Ring, Gerd Rosenkranz, Amy Spencer and David Svensson.
The SIG is currently lead by David Svensson.
(Up to May 2018, the lead was Aaron Dane).
How to get in touch firstname.lastname@example.org
Topics Discussed During Previous Meetings:
- 2018 APRIL: Update re progress of White Paper, remaining work for 2018. Ideas include further work on Bayesian shrinkage, Model averaging, Simulations under NULL when prognostic factors are present, SEAMOS development for non-linear models and Bootstrap Bias reduction.
- 2018 JUNE: David PSI presentation on some aspects of Shrinkage, multi-level hierarchical models, and model averaging. Key aspect: many variations exist, and some unknowns regarding performance.
- 2018 JULY: Simulation of RCT discussed with underlying prognostic predictive continouos variables - dichotomized into subgroup factors - some preliminary illustrations of methods listed under APRIL.
- 2018 SEPT: Further simulations on the performance of BIC model averaging.
- 2018 NOV: Visualisation using novel R package SubrPlots (e.g., UpSet graph), Amy Spencer presenting further work on SEAMOS (modification to increase power). Updates on PSI 2019 Subgroup Section
- 2019 JAN: Discussing content for SIG subgroup session at PSI, presentation by David Svensson on SEAMOS (some simulation results in a non-linear case with prognostic factors).
- 2019 MARCH: Review of novel R CRAN package 'subtee', containing functionality for Bootstrap-based bias-reduction and BIC-Model-Averaging Shrinkage. Some simulation examples were discussed, and it was decided that more such (across broader assumptions) will be the topic for another SIG meeting. Issues regarding hierarchical Bayesian shrinkage when subgroups are overlapping was also discussed, with particular attention to a paper by Varadhan and Wang.
- EFSPI WORKSHOP 20th MARCH 2019 on Recent Developments on Subgroups and Biomarkers (Gothenburg) saw contributions from the SIG (presentations by G. Rosenkranz, and I.Lipkovich & D.Svensson) with the focus on HTE/Individual Treatment Effects, but also general aspects of consistency across subgroups was discussed, including some regulatory contribution.
- MAY 2019: Discussion of resampling principles for creating NULL data (idea: generating a large number of replicates with marginal distribution and correlations as in observed data except that treatment hetereogeneity is taken out); discussings also regarding possibities of a new points-to-consider document for exploratory analyses ('reality check' aspects to consider when assessing a given datamining/subgroup detection analysis).
- JUNE (PSI LONDON 2019): SIG members presented an overview and some recommendations regarding subgroup detection (Necdet Gunsoy, Ilya Lipkovich & David Svensson) at the SIG session (chaired by Aaron Dane).
- AUGUST 2019: Further discussions and planning of compiling a points to consider document that would help upfront when planning/conducting/reviewing an exploratory subgroup analysis (the latter to be interpreted in a broad sense, including data-driven approaches). Discussions regarding the upcoming PSI webex and PSI Subgroup session for Barcelona 2020.