Post-randomization Biomarker Effect Modification Analysis in an HIV Vaccine Clinical Trial

1 Introduction

A vaccine that prevents HIV-1 infection is critically needed for ending the global HIV-1 pandemic. A recent efficacy trial of a candidate HIV-1 vaccine – HVTN 505 – randomized 2,496 HIV-1 negative volunteers in 1:1 allocation to the DNA/rAd5 vaccine regimen or placebo [1] administered at months 0, 1, 3, 6. The primary objective compared the rate of HIV-1 infection from 6.5 to 24 months between the randomized treatment arms, with estimated cumulative incidence 4.62% (3.15%) in the vaccine (placebo) group, and cumulative incidence ratio 1.46 (95% CI 0.82 to 2.63, Wald test p = 0.20). Through a 2-phase sampling design, Janes et al. [2] studied HIV-1 Envelope-specific CD8+ T cell responses measured %by intracellular cytokine staining 2 weeks post last vaccination (Month 6.5) as a correlate of HIV-1 infection between 6.5 and 24 months, and found in vaccine recipients a strong inverse correlation between CD8+ T cell polyfunctionality score (PFS) and HIV-1 infection (p < 0.001). The surprising strength of the correlate [e.g., estimated cumulative risk 0.160 and 0.034 for vaccinated subgroups with PFS below and above the median response, compared to 0.070 for placebo recipients (Figure 4 of [2]), suggests the possibility that a qualitative interaction occurred, where some marker-defined vaccinated subgroups received partial protection from vaccination whereas others had their risk increased.

Janes et al.’s observation of intermediate risk of the placebo group between that of the biomarker response-defined vaccinated subgroups does not imply causal vaccine vs. placebo effect modification across these subgroups, because post-randomization selection bias could occur (e.g., due to an unmeasured genetic factor predictive of both the PFS and HIV infection). A direct assessment of a causal vaccine effect (eliminating possible selection bias) would compare risk for each vaccinated subgroup defined by biomarker response value to that of the placebo recipient subgroup who would have had the same biomarker response value if assigned vaccination, which essentially repeats the primary analysis of vaccine efficacy (which is valid based on the randomization) across the marker-defined subgroups. This “principal surrogate (PS) analysis” [3] is a sub-field of the general framework of principal stratification established by Frangakis and Rubin [3].

However, available methods for PS analysis make strong structural risk assumptions that do not hold in HVTN 505, and require study design augmentations such as close-out placebo vaccination to measure counterfactual biomarker response [4] that were not utilized in HVTN 505. Given the high threshold of evidence needed to convince scientists of a qualitative interaction, application of these methods would be inadequate – rather PS analysis with methods that rely on weaker assumptions are needed. Fortunately, many nonparametric and semiparametric principal stratification methods are available that do not require a design augmentation, many of which were developed for applications in another sub-field of principal stratification – “survivor average causal effect (SACE) analysis” (e.g., [5, 6])– which we define as principal stratification methods focused on a single principal stratum (e.g., always survivors). We describe how such methods designed for SACE analysis can be adapted to PS analysis for the special case of a binary intermediate outcome. This adaptation contributes novel PS methods, by (1) applying to clinical trials without design augmentations; (2) relaxing the strong assumption of no individual-level clinical treatment effects before the biomarker is measured; (3) avoiding strong structural placebo conditional risk assumptions; and (4) extending SACE methods to study designs that only measure the intermediate outcome in a subset of participants (e.g., two-phase/case-control studies), as in HVTN 505. The PFS marker studied by Janes et al. [2] is a summary measure of many different cellular expression patterns, which are binary (expressed vs. not expressed), motivating methods for a binary intermediate outcome.

2 Principal Surrogate (PS) Target Parameters and the HVTN 505 Application

A common objective of randomized clinical trials is to assess biomarker response endpoints as principal surrogate endpoints for the clinical endpoint of interest [3]. With Z denoting binary treatment assignment and S^τ denoting a biomarker measured at a fixed time point τ post-randomization and Y denoting the binary clinical endpoint of interest measured after τ, the PS estimand of interest is the “principal effects” or “causal effect predictiveness” (CEP) surface [7]: CEP(s₁, s₀) = h(risk₁(s₁, s₀), risk₀(s₁, s₀)), where risk_z(s₁, s₀) ≡ P(Y(z) = 1∣S^τ(1) = s₁, S^τ(0) = s₀, Y^τ(1) = Y^τ(0) = 0) for z = 0, 1. Here Y^τ(z) is the indicator that the clinical endpoint occurs by time τ, such that the population for inference is free of the endpoint under both treatment assignments when S^τ is measured. In HVTN 505, S^τ measures an immune response to vaccination and Y = 1 is subsequent HIV-1 infection, and it would be problematic to include individuals already infected with HIV-1, because their S^τ values would be affected by the natural immune response to HIV-1 infection, making the results uninterpretable. Therefore, CEP(s₁, s₀) contrasts the risk of infection between the vaccine and placebo arms among those who would be infection-free at time τ under either treatment assignment and have biomarker measures s₁ and s₀ under assignment to vaccine and placebo, respectively. The objective of PS analysis is inference about the CEP(s₁, s₀) parameters and contrasts in these parameters, which has been addressed using a variety of methodological approaches including estimated maximum likelihood [4, 7, 8], pseudo-score estimating equations [9], principal score methods [10, 11, 12], and Bayesian methods [13, 14, 15].

We consider PS methods assuming either “No Early Effect” (NEE), “No Early Harm” (NEH), or “No Early Benefit” (NEB) of treatment on Y by time τ, as defined below as A4, A4′, and A4″. For the PS methods assuming NEH or NEB, we focus on the special case that Z = 0 is a control condition such as placebo, and there is no variability of S^τ in subjects assigned to Z = 0, i.e., P(S^τ(0) = 0∣Y^τ(0) = 0) = 1. This “Constant Biomarker (CB)” case occurs in placebo-controlled preventive vaccine efficacy trials that only enroll subjects not previously infected with the pathogen under study and for which the intermediate response endpoint is a readout from a validated bioassay designed to only detect an immune response specific to the pathogen under study [7]; HVTN 505 is a typical example. For methods assuming NEE, we consider both the special Case CB and the general case where S^τ(0) varies and a monotonicity assumption holds (A5 below). These scenarios are chosen because they commonly occur in applications. Table 1 describes the interpretation of the CEP(s₁, s₀) parameters for HVTN 505. Because Case CB holds, CEP(1,0) and CEP(0, 0) are of interest. For a given binary immune response biomarker S^τ, VE(1) = CEP(1, 0) is vaccine efficacy for the subgroup with positive response if assigned vaccine and negative response if assigned placebo, and VE(0) = CEP(0, 0) is vaccine efficacy for the subgroup with negative response under both treatment assignments. Our scientific goal is to study whether vaccine efficacy differs between these two subgroups, i.e. μ ≡ VE(1) − VE(0) ≠ 0, and we also seek to assess evidence for a qualitative interaction, defined by VE(1) and VE(0) having opposite signs.

3 Assumptions and Identifiability of the PS Target Parameters CEP(s₁, s₀)

4 SACE Methods and their Translation to PS Analysis

5 Illustration of the Approach with IPW Extensions of Particular SACE Methods

For each assumption set scenario, we show how the PS estimation and inference works using an extension of Shepherd, Gilbert, and Dupont’s [24] SACE method, which in scenarios NEE-VB, NEE-CB simplifies to the Gilbert, Bosch, and Hudgens [22] (GBH) SACE method. The extension incorporates inverse probability weighting (IPW) to handle missing S^τ. We focus on semiparametric efficient estimators given the data (Z, R, Y^τ, S^τ, Y) not including baseline covariates W, which amount to sample means with or without IPW as needed. Moreover, we focus on the case that Y is binary and observed for all participants; Web Appendix C summarizes how the methods translate to Y = I(T ≤ t) with T subject to right-censoring before t. We use general estimating function notation so that users preferring to use more efficient estimators leveraging information in W (e.g., [28]) may substitute alternative estimating functions into the equations.

6 Simulation Study

We first simulated data sets under the assumptions of NEE-CB. For each of n independent individuals, we generated potential outcomes and then observed outcomes. First, (Y^τ(1), Y^τ(0)) was set to (0, 0) or (1, 1) with probabilities 0.8 and 0.2, such that A4 (NEE) holds. If Y^τ(1) = 1, then Y(0) and Y(1) were set to 1. If Y^τ(1) = 0, then (S^τ(1), S^τ(0)) was set to (0, 0) or (1, 0) with probabilities 0.4 and 0.6, such that Case CB holds.

To evaluate size and power of a test of H₀: CEP(1, 0) = CEP(0, 0) versus H₁ : CEP(1, 0) ≠ CEP(0, 0), data were simulated under 13 different values for CEP(1, 0) − CEP(0, 0): −0.6, −0.5, …, 0.6. Specifically, if Y^τ(1) = 0 and S^τ(1) = S^τ(0) = 0, then Y(1) was generated as Bernoulli with mean a (Bern(a)) and Y(0) as Bern(0.5). If Y^τ(1) = 0 and S^τ(1) = 1, S^τ(0) = 0, then Y(1) was Bern(b) and Y(0) was Bern(0.5). Thus CEP(1, 0) − CEP(0, 0) = b − a for contrast function h(x, y) = x − y. The values of a ranged from 0.7 to 0.1 by decrements of 0.05 and b increased from 0.1 to 0.7 by increments of 0.05. Under this parameterization risk₀(0, 0) = 0.5 = risk₀(1, 0), implying β₀ = 0.

To generate the observed data, Z was drawn from Bern(0.5), and the vector of observable random variables (Z, Y^τ, S^τ, Y) = (Z, Y^τ(Z), S^τ(Z), Y(Z)) was determined. Simulations were conducted with and without case-cohort sampling of S^τ. For the latter, membership in the random subcohort was determined by R drawn from Bern(ν). For individuals with Y^τ = 0, S^τ was observed for subcohort members (i.e., R = 1) and cases (i.e., Y = 1).

Data sets were generated for all 156 combinations of: n ∈ {200, 400, 800, 1600}; ν ∈ {0.1, 0.25, 1}; and CEP(1, 0) − CEP(0, 0) ∈ {−0.6, −0.5, …, 0.6}, except the 13 scenarios with n = 200 and ν = 0.1 were not studied because only 8 vaccine recipient uninfected controls are expected to have S^τ measured. All results are based on 2000 simulated data sets. Analyses used [l₀, u₀] ∈ {[0, 0], [−1, 1], [−2.5, 2.5]}. The true values for CEP(1, 0) and CEP(0, 0) were selected to include zero effect modification and gradients in effect modification magnitudes in both directions, with values (a, b) ∈ {(0.1, 0.7), (0.15, 0.65), (0.2, 0.6), (0.25, 0.55), (0.55, 0.25), (0.6, 0.2), (0.65, 0.15), (0.7, 0.1)} expressing qualitative interactions. For each simulated data set, the data analysis was done using the methods for the scenario NEE-CB assumption set. The null hypothesis H₀ was rejected if and only if the 95% EUI for μ = CEP(1, 0) − CEP(0, 0) calculated as described in Section 5 excluded 0. Power was estimated by the proportion of simulated data sets where H₀ was rejected (Figure 1).

Figure 1

Power to reject H₀: μ = CEP(1, 0) = CEP(0, 0) for the simulation study under Scenario NEE-CB. Solid black lines denote full cohort and dashed (dotted) lines denote case-cohort with 10% (25%) random subcohort.

Empirical type I error was close to the nominal level α = 0.05. Power increased with sample size and decreased as the interval [l₀, u₀] became wider and as the subcohort size decreased. Web Figure 1 (Web Appendix D) shows the average widths of the 95% EUIs for μ. The EUIs cover at approximately the nominal rate when l₀ = u₀, i.e., when μ is identifiable, and are conservative otherwise. The widths are relatively constant across values of μ and increase as the subcohort size decreases. Empirical coverage of the 95% EUIs are plotted in Web Figure 2. Web Figures 3 and 4 display bias of the μ̂ estimates and ratios of the empirical standard errors (ESE) to the average of the sandwich variance estimated standard errors (ASE), showing unbiasedness of both the point and standard error estimators.

Figure 2

For HVTN 505, ignorance intervals (solid lines) and 95% EUIs (dashed lines) for VE(0) = CEP(0, 0) and VE(1) = CEP(1, 0) and μ = VE(1) − VE(0) with binary Month 6.5 biomarker S^τ equal to 1 (0) if the vaccine recipient expresses (does not express) the CD8+ T-cell subset defined by expression pattern (S1τ)(−,−,−,−,+,+),(S2τ)(+,−,−,−,+,+), or (S3τ)(+,−,+,+,−,+). The sensitivity analysis allows β₀ and β₁ to vary over [l₀, u₀] = [0, 0], [−0.5, 0.5], or [−1, 1]; circles are point estimates assuming no selection bias. The top panel shows results for μ = VE(1) − VE(0) with the black and grey lines (left and right) results for the scenario NEE-CB and NEB-CB method, respectively. The bottom panel shows scenario NEE-CB results for VE(1) and VE(0).

A second simulation study was conducted, with data simulated under scenario NEH-CB such that A4 in scenario NEE-CB failed. Web Appendix D describes details and results, with data analysis by the methods under assumption-set NEH-CB (Web Figures 5–9).

7 Application: HVTN 505 HIV Vaccine Efficacy Trial

We apply the new PS methods to HVTN 505. The PFS biomarker described in the introduction [2] is a quantitative aggregate score derived from constituent qualitative biomarkers that are of interest for binary intermediate outcome PS analysis. The data from a vaccine recipient’s Month 6.5 blood sample are measurements of expression of 6 different functional markers (Granzyme-B, IL4, CD40L, TNFalpha, IL2, IFNgamma) in CD8+ T cells after stimulation with HIV-1 peptides. The Bayesian method COMPASS [35] provided, for each vaccine recipient and each of the 2⁶ = 64 possible cell subsets (i.e., combinations of functional markers), the probability the subset was expressed. A question of interest is whether the vaccine effect on HIV-1 infection varied by expression yes vs. no for any of the 64 subsets. Identification of specific cell subsets associated with vaccine protection or harm would give clues on cellular mechanisms of vaccine effect.

In this application Y^τ is the indicator of HIV-1 infection diagnosis between enrollment and the Month τ = 6.5 study visit, and Y is the indicator of this event by the Month 24 final follow-up visit; moreover S^τ is a binary biomarker measured from a Month 6.5 visit blood sample. We defined a vaccine recipient as expressing a given T cell subset (S^τ = 1) if the COMPASS posterior probability exceeded 0.9, and as not expressing the subset (S^τ = 0) if the posterior probability was below 0.1. We restricted to cell subsets with at least 25% of vaccine recipients expressing and at least 25% not expressing the subset, to focus on subsets with ample data support. This yielded three cell subsets for analysis, defined by (GranzymeB, IL4, CD40L, TNFalpha, IL2, IFNgamma) expression pattern of (1) (−, −, −, −, +, +), (2) (+, −, −, −, +, +), and (3) (+, −, +, +, −, +), where + or - indicate that cells do or do not have the function. Among uninfected vaccine recipients (i.e., sampled controls with Y = 0) with S^τ data, 71 of 110 (64.5%), 70 of 124 (56.5%), and 43 of 125 (34.4%) have S^τ = 1, respectively. Among infected vaccine recipients (i.e., cases with Y = 1) with S^τ data, 5 of 21 (23.8%), 3 of 25 (12.0%), and 1 of 25 (4.0%) have S^τ = 1. Given Case CB holds, for each biomarker our goal is inference on VE(0) = CEP(0, 0) and VE(1) = CEP(1, 0), using the vaccine efficacy contrast h(x, y) = 1−x/y (Table 1). For reasons given below, we implement the methods using the NEE-CB method and the NEB-CB method. We consider the meaning and plausibility for HVTN 505 of the critical assumptions in question needed for these two approaches – (A4, A6) and (A4″, A7′, A8), respectively, where A6, A7′, and A8 are needed for valid inference as described in Section 5.2. A4 (NEE) states that the vaccine has no individual-level effects on infection between enrollment and Month 6.5, which is defensible given that P̂(Y^τ(1) = 1) = 14/1251 = 0.011 and P̂(Y^τ(0) = 1) = 10/1245 = 0.0080 with Fisher’s exact test 2-sided p-value of p = 0.54 for a difference. Because 0.011 > 0.0080, we also consider the NEB-CB method that relaxes A4 to A4″ (no individual-level beneficial vaccine effects through Month 6.5).

A6 states that the biomarker has a higher frequency of response for at-risk vaccine than placebo recipients, which essentially always holds. Under Case CB as in HVTN 505 (with P(S^τ(0) = 1∣ Y^τ = 0) = 0), it obviously holds. A7′ states that there is a harmful vaccine effect (in expectations) through Month 6.5, which is consistent with the point estimates 0.011 vs. 0.0080 but is not supported by the hypothesis testing with p = 0.54. In rare event studies such as HVTN 505, A8 states that the fraction of vaccine recipients with positive biomarker response S^τ = 1 is not near zero, which holds. In sum, the method for scenario NEE-CB may be the most reasonable because it does not require Early Harm A7′, and we focus on its results; we also apply the NEB-CB method for comparison. The NEE-CB and NEB-CB methods are implemented as described in Sections 5.3 and 5.5, respectively.

Figure 2 shows the results in terms of ignorance intervals and 95% EUIs under each of the three ranges of sensitivity parameters specified in the simulations. To interpret the sensitivity analysis, note that the single sensitivity parameter β₀ for the NEE-CB method has interpretation as the log odds ratio that the biomarker response to vaccination S^τ(1) takes value 0 for a placebo recipient who is diagnosed with HIV-1 infection between Month 6.5 and 24 compared to a placebo recipient who is not diagnosed with HIV-1 infection through 24 months. Setting β₀ to 0, 0.5, and 1 specifies no, intermediate, and highest degree of selection bias, corresponding to the odds ratio e^β₀ varying over [1,1], [0.61,1.65], and [0.37,2.72]. The NEB-CB method uses the additional sensitivity parameter β₅ (defined in Section 5.5), which is the log odds ratio of infection for a placebo recipient, comparing the always early uninfected (AEU) subgroup (numerator) to the early harmed subgroup (denominator); thus it measures the degree to which vaccine-caused infection by Month 6.5 is prognostic for infection by Month 24 if assigned placebo. It was varied over the same range as β₀.

On the results, first note that the NEE-CB and NEB-CB methods give very similar results (top panel), with EUIs for the latter method only very slightly wider (and thus the bottom panel only shows results from the NEE-CB method). Web Appendix E (especially Web Figure 10) studies why the methods under scenarios NEE-CB and NEB-CB are essentially equivalent for HVTN 505, with extra simulations suggesting that a key part of the explanation is that P(Y^τ(1) = 1, Y^τ(0) = 0) is small. Second, the results are similar for the three markers, with evidence for effect modification across the expressed vs. not expressed subgroups based on the 95% EUIs for μ = VE(1) − VE(0) lying above zero even when allowing for the largest amount of potential selection bias (β = 1, Figure 2 upper-right panel, with lower EUI limit 0.12, 0.19, 0.18 for marker S1τ,S2τ,S3τ).

The bottom panel shows that VE(1) is estimated to be positive for each marker (ignorance intervals 0.70–0.85, 0.78–0.91, 0.85–0.96 and 95% EUIs 0.44–0.98, 0.54–1.0, 0.59–0.96, respectively), whereas VE(0) is estimated to be less than zero with wide estimated uncertainty intervals. Based on ignorance intervals these results support a qualitative interaction for each marker. However, because the upper 95% EUI limits for VE(0) extend well above 0, there is not compelling evidence for qualitative interactions.

Remarkably, the results support differential vaccine efficacy according to whether the vaccine induced CD8+ T cells with expression vs. not expression of specific cell subsets, suggesting that the vaccine may have conferred partial protection for individuals with certain identified expression signatures. This motivates follow-on basic science studies of cellular mechanisms of protection. The HIV vaccine field has “moved on” from the DNA/rAd5 type of HIV vaccine platform, no longer considering it. Thus these new results sound a note of caution to not prematurely abandon this platform, in suggesting that if a new version of the regimen could be invented that induces the specific cell subset responses in a much larger subgroup of vaccine recipients, then it could potentially confer high enough overall vaccine efficacy to confer worthwhile public health benefit.

8 Discussion

A sizable literature on nonparametric and semiparametric methods for inference on the survivor average causal effect (SACE) parameter has developed over the past 20 years. Motivated by the need for more robust principal surrogate (PS) analysis in HVTN 505, we described how these methods can be adapted to PS analysis for a binary intermediate variable. This provides new methods for PS analysis, with novel features summarized in the Introduction, all of which were needed for the HVTN 505 application, to appropriately account for the study design and available data and to relax questionable and untestable assumptions. The new methods have application to similar PS questions in other randomized clinical trials.

More robust PS assessment of a qualitative interaction in HVTN 505 was important given the appropriate skepticism that the DNA/rAd5 vaccine could have both beneficial and detrimental effects. The data analysis supported that DNA/rAd5 vaccinated subgroups defined by induction of CD8+ T cells with specific functions had beneficial vaccine efficacy, motivating further research to re-engineer the vaccine regimen to increase induction of these cells.

PS analysis can be improved by accounting for baseline covariates associated with principal strata and/or the final outcome [16, 36]. The specific PS methods described in Section 5 can account for baseline covariates by implementing any preferred covariate-adjusted estimator of the means risk₁, risk₀, p(0, 0), p(1, 0), p(1, 1). An appealing alternative approach that fully accounts for baseline covariates – principal score methods [10, 11, 12] –provide PS analysis under a different assumption, principal ignorability, and provide an alternative sensitivity analysis. In general, given that the key steps in estimating the CEP(s₁, s₀) parameters is estimation of the three SACES listed in Table 2, any SACE method that accounts for baseline covariates can be integrated into a new method of PS analysis.