Staging and clean room: Constructs designed to facilitate transparency and reduce bias in comparative analyses of real-world data

Journal: Pharmacoepidemiology and Drug Safety

Authors: Paul Muntner, Rohini K Hernandez, Shia T Kent, James E Browning, David T Gilbertson, Kathleen E Hurwitz, Susan S Jick, Edward C Lai, Timothy L Lash, Keri L Monda, Kenneth J Rothman, Brian D Bradbury, M Alan Brookhart

NLM Citation: Muntner P, Hernandez RK, Kent ST, Browning JE, Gilbertson DT, Hurwitz KE, Jick SS, Lai EC, Lash TL, Monda KL, Rothman KJ, Bradbury BD, Brookhart MA. Staging and clean room: Constructs designed to facilitate transparency and reduce bias in comparative analyses of real-world data. Pharmacoepidemiol Drug Saf. 2024 Mar;33(3):e5770. doi: 10.1002/pds.5770. PMID: 38419140.

Abstract

Purpose: We describe constructs designed to protect the integrity of the results from comparative analyses using real-world data (RWD): staging and clean room.

Methods: Staging involves performing sequential preliminary analyses and evaluating the population size available and potential bias before conducting comparative analyses. A clean room involves restricted access to data and preliminary results, policies governing exploratory analyses and protocol deviations, and audit trail. These constructs are intended to allow decisions about protocol deviations, such as changes to design or model specification, to be made without knowledge of how they might affect subsequent analyses. We describe an example for implementing staging with a clean room.

Results: Stage 1 may involve selecting a data source, developing and registering a protocol, establishing a clean room, and applying inclusion/exclusion criteria. Stage 2 may involve attempting to achieve covariate balance, often through propensity score models. Stage 3 may involve evaluating the presence of residual confounding using negative control outcomes. After each stage, check points may be implemented when a team of statisticians, epidemiologists and clinicians masked to how their decisions may affect study outcomes, reviews the results. This review team may be tasked with making recommendations for protocol deviations to address study precision or bias. They may recommend proceeding to the next stage, conducting additional analyses to address bias, or terminating the study. Stage 4 may involve conducting the comparative analyses.

Conclusions: The staging and clean room constructs are intended to protect the integrity and enhance confidence in the results of analyses of RWD.

Keywords: comparative effectiveness; methods; real-world data; safety; study design.