A Multi-center Study on the Reproducibility of Drug-Response Assays in Mammalian Cell Lines

Abstract

Evidence that some high-impact biomedical results cannot be repeated has stimulated interest in practices that generate findable, accessible, interoperable, and reusable (FAIR) data. Multiple papers have identified specific examples of irreproducibility, but practical ways to make data more reproducible have not been widely studied. Here, five research centers in the NIH LINCS Program Consortium investigate the reproducibility of a prototypical perturbational assay: quantifying the responsiveness of cultured cells to anti-cancer drugs. Such assays are important for drug development, studying cellular networks, and patient stratification. While many experimental and computational factors impact intra- and inter-center reproducibility, the factors most difficult to identify and control are those with a strong dependency on biological context. These factors often vary in magnitude with the drug being analyzed and with growth conditions. We provide ways to identify such context-sensitive factors, thereby improving both the theory and practice of reproducible cell-based assays.

Keywords: cancer drugs; cell line; dose response; high[HYPHEN]throughput; microscopy; oncology; pharmacology; reproducibility; screening.

Figure 1.. Overview of Workflow

(A) Centerone defined the experimental protocol and established within-center reproducibility by assessment of technical (different wells, plates, same day) and biological (different days) replicates. Common stocks of drugs, cells, and media, as well as a standard experimental protocol, were distributed to each of the five data-generation centers. Center one explored the various technical and biological drivers of variability. This information was fed back to the other centers to refine their dose-response measurements. (B) Dose-response curves of MCF 10A treated with the MEK½ inhibitor Trametinib from a typical experiment showing technical and biological replicates. Technical replicates at the well (triplicate wells per plate) and plate (triplicate plates per experiment) levels make up biological replicates (repeats collected on different days in the same laboratory). The red triangles represent the average of the three biological replicates shown. Error bars represent SD of the mean. (C) Independent experiments performed in center one, and in all centers (averages of two or more biological replicates). Circles represent the original dataset, triangles represent data collected by a new technician 2 years after the initial data collection [data shown in (B)], and diamonds represent independently collected data in center one. Inter-center replicates (averages of one or more biological replicates) performed independently at each center. Error bars represent the standard deviation of the mean. See also Figure S1.

Figure 2.. Experimental Causes of Variability

(A) Dose-response curves of MCF 10A cells treated with four different drugs measured by image-based cell count or ATP content (CellTiter-Glo) on the same day by center one, which is equivalent to technical replicates. Note the GR₅₀ value for alpelisib as measured by CellTiterGlo was not defined. (B) Representative images of MCF 10A cells treated with vehicle control (DMSO) or 1 μM Palbociclib. Cells were stained with Hoechst and phalloidin. Images have been contrast adjusted. (C) Uneven growth of MCF 10A cells in a 384-well plate over the course of 3 days that demonstrates the presence of edge effects. In the heatmap, color represents the number of cells per well, as assessed by imaging. Plots show deviation from mean number (for the full plate based on the distance from the edge, by column, or by row). Error bars represent the standard deviation. Asterisks indicate the row or column differs significantly from all others. See also Figure S2.

Figure 3.. Technical Causes of Variability

(A) Dose-response curves of MCF 10A cells treated with Trametinib or Dasatinib fitted to either the extended dose range (up to 1 μM and 10 μM, respectively) or omitting the last order of magnitude. (B) Results of cell counting for MCF 10A cells treated with Dasatinib or Neratinib using two different image processing algorithms (denoted as A (red) and B (blue)) included in the Columbus image analysis software package. (C) Number of dead cells (LIVE/DEAD™ Fixable Red Dead Cell Stain positive) and nuclei (Hoechst positive) counted for MCF 10A cells treated with 3.16 μM Dasatinib or 1 μM Neratinib based on the two different algorithms (corresponding to the plots in C). See also Figures S3 and S4.

Figure 4.. Changes in Drug Response Related to the Underlying Biology

Left: Inhibition of MCF 10A growth (12-h instantaneous GR values) measured in a time-lapse, live-cell experiment involving treatment with multiple doses of Etoposide (top) or Neratinib (bottom). Different colors indicate different drug concentrations ranging from 1 nM (yellow) to 10 μM (blue). Right: Dose-response curves derived from 12-h GR values computed at 24 (red) and 48 h (blue) across three biological repeats. Etoposide displays only modest time-dependent effects (top) while neratinib appears to be more effective at inhibiting growth at early time points as compared to later time points (bottom). Error bars, SD. See also Figure S5.

Figure 5.. Technical and Biological Variability in Estimating GR Values and Metrics

(A) The kernel density estimate (KDE) of the standard error (SE) for measurement of GR values across technical (green curve) or biological (blue curve) replicates for all drugs and doses. The left panel depicts data from center one, scientist B (performed in 2018); the middle panel shows four sets of measurements from all scientists in center one (performed between 2016–2018); and the right panel all data from all centers. The distribution of technical error for Scientist B is duplicated in the middle and right panels as a black dotted line to facilitate comparison. Data for these distributions were derived from GR values for each dose and replicate, not GR metrics obtained from curve fitting. The number of GR value data points used to compute SE is detailed in STAR Methods. The number of SE data points that constitute each KDE is shown in the legend; for the left panel this is 192 SE data points (8 doses × 8 drugs × 3 biological repeats). The lower section of each panel depicts the error in GR value measurements across technical replicates (green) and biological replicates (blue) for each individual drug. (B) The range of SE in GR values compared to the SE in corresponding GR metrics (GR_max, area over the GR curve (GR AOC), and log₁₀GR₅₀) for all drugs. The black vertical line (A, lower plots, and B) is the mean technical error for a given drug and the red vertical line demarcates the 90^th percentile error across technical replicates (meaning that the error for 90% of GR values or GR metrics is below that value); a blue circle demarcates 90^th percentile error across biological replicates.

Figure 6.. Variability of the Response Measures across Centers

(A) Dose-response curves of MCF10A cells treated with eight drugs measured independently by the five centers (circles represent data from image-based assays and triangles from CellTiter-Glo assays). See Figure S6 for underlying replicates. Dotted black lines show the dose-response curve when all independent replicates were averaged. Error bars represent SD of the mean. (B) GR metrics describing the sensitivity of MCF 10A cells to eight drugs measured independently by five centers (circles represent data from image-based assays and triangles from CellTiter-Glo assays). The black line shows the mean sensitivity across all centers, and the gray area shows the standard error of the mean computed from the average of each center. For GR₅₀ and GR_max, error bars represent the standard deviation of the log₁₀(GR) values. Note that some data are shared between Figures 6 and S3.

Figure 7.. Best Practices for Dose-Response Measurement Experiments

(A) Summary of findings in this and related studied with respect to experimental and technical variability in dose response studies at the experimental design, materials, methods, and analysis stages; “*” indicates sources of variability that have been thoroughly investigated in a previous paper (Hafner et al., 2016). (B) Differences between precision, robustness, and reproducibility; see text for details.