Evaluation of log P, pKa, and log D predictions from the SAMPL7 blind challenge

Abstract

The Statistical Assessment of Modeling of Proteins and Ligands (SAMPL) challenges focuses the computational modeling community on areas in need of improvement for rational drug design. The SAMPL7 physical property challenge dealt with prediction of octanol-water partition coefficients and pK_a for 22 compounds. The dataset was composed of a series of N-acylsulfonamides and related bioisosteres. 17 research groups participated in the log P challenge, submitting 33 blind submissions total. For the pK_a challenge, 7 different groups participated, submitting 9 blind submissions in total. Overall, the accuracy of octanol-water log P predictions in the SAMPL7 challenge was lower than octanol-water log P predictions in SAMPL6, likely due to a more diverse dataset. Compared to the SAMPL6 pK_a challenge, accuracy remains unchanged in SAMPL7. Interestingly, here, though macroscopic pK_a values were often predicted with reasonable accuracy, there was dramatically more disagreement among participants as to which microscopic transitions produced these values (with methods often disagreeing even as to the sign of the free energy change associated with certain transitions), indicating far more work needs to be done on pK_a prediction methods.

Keywords: Free energy calculations; SAMPL; log P; pK a.

Fig. 1

Structures of the 22 molecules used for the SAMPL7 physical property blind prediction challenge. Log of the partition coefficient between n-octanol and water was determined via potentiometric titrations using a Sirius T3 instrument. pK_a values were determined by potentiometric titrations using a Sirius T3 instrument. Log of the distribution coefficient between n-octanol and aqueous buffer at pH 7.4 were determined via potentiometric titrations using a Sirius T3 instrument, except for compounds SM27, SM28, SM30-SM34, SM36-SM39 which had log D_7.4 values determined via shake-flask assay. PAMPA assay data includes effective permeability, membrane retention, and log of the apparent permeability coefficient. Permeabilities for compounds SM33, SM35, and SM39 were not determined. Compounds SM35, SM36 and SM37 are single cis configuration isomers. All other compounds are not chiral

Fig. 2

For each molecule in the SAMPL7 pK_a challenge we asked participants to predict the relative free energy between our selected neutral reference microstate and the rest of the enumerated microstates for that molecule. In this case, we asked for the relative state free energy including the proton free energy, which could also be called the reaction free energy for the microstate transition which has the reference state as the reactant and the alternate state as the product. Using SM43 as an example, participants were asked to predict the relative free energy between SM43_micro000 (our selected neutral microstate highlighted in yellow) and all of the other enumerated microstates (SM43_micro001–SM43_micro005) for a total of 5 relative state free energies ($\mathrm{\Delta }$G_BA, $\mathrm{\Delta }$G_CA, $\mathrm{\Delta }$G_DA, $\mathrm{\Delta }$G_EA, $\mathrm{\Delta }$G_FA). Some transitions involved a change in a single protonation state (e.g. the D–A transition of Figure 2) or tautomer (e.g. the C–A transition of Figure 2). A few cases involved a change of multiple protons (e.g. the F–A transition of Figure 2). All transitions were defined as away from the neutral reference state. Distinct microstates are defined as all tautomers of each charge state. For each relative free energy prediction reported, participants also submitted the formal charge after transitioning from the selected neutral state to the other state. For example, the reported charge state after transitioning from SM43_micro000 to SM43_micro001 would be − 1, SM43_micro000 to SM43_micro004 would be 0 (these are tautomers of each other), SM43_micro000 to SM43_micro005 would be + 1, and SM43_micro000 to SM43_micro003 would be + 2

Fig. 3

Using the microstate probability to convert microscopic pK_a predictions to macroscopic pK_a’s with the titration method pK_a’s. Blue and orange lines represent two states. Blue states have one more proton than the orange states, and thus a formal charge higher by + 1. The blue state has one tautomer and the orange state has 3, denoted by the dashed lines. The solid lines are the ensemble averaged state probability for each group with a given charge. The crossing point between two ensemble lines is the macroscopic pK_a

Fig. 4

Overall accuracy assessment for all methods participating in the SAMPL7 log P challenge shows that many methods did not exhibit statistically significant differences in performance and there was no single clear winner; however, empirical methods tended to perform better in general. Both root-mean-square error (RMSE) and mean absolute error (MAE) are shown, with error bars denoting 95% confidence intervals obtained by bootstrapping over challenge molecules. Empirical methods outperform the majority of the other methods. Methods that achieved a RMSE $\le$ 1.0 log P units were mainly empirical based, and some were QM-based physical methods. Submitted methods are listed in Table 1. The submission REF1 ChemAxon [80] was a reference method included after the blind challenge submission deadline, and NULL0 mean cLogP FDA is the null prediction method; all others refer to blind predictions

Fig. 5

Predicted vs. experimental value correlation plots of 5 best performing methods and one representative average method in the SAMPL7 log P challenge. Dark and light green shaded areas indicate 0.5 and 1.0 units of error. Error bars indicate standard error of the mean of predicted and experimental values. In some cases, log P SEM values are too small to be seen under the data points. The best-performing methods were made up of three empirical methods (ClassicalGSG DB3 [85], TFE MLR [87], Chemprop [88]) and two QM-based physical methods (COSMO-RS [89], TFE-NHLBI-TZVP-QM). Details of the methods can be found in "A shortlist of consistently well-performing methods in the pK_a challenge" sect. and performance statistics are available in 2. Method NES-1 (GAFF2/OPC3 G) was selected as the representative average method, which has a median RMSE

Fig. 6

Molecule-wise prediction accuracy in the log P challenge point to isoxazoles as poorly predicted, especially by MM-based physical methods. Molecules are labeled with their compound class as a reference. A The MAE calculated for each molecule as an average of all methods. B The MAE of each molecule separated by method category. C log P prediction error distribution for each molecule across all prediction methods

Fig. 7

Overall accuracy assessment for all methods participating in the SAMPL7 pK_a challenge shows that two methods, one a Physical (QM) method and one a QSPR/ML, performed better than other methods. Both root-mean-square error (RMSE) and mean absolute error (MAE) are shown, with error bars denoting 95% confidence intervals obtained by bootstrapping over challenge molecules. REF00_Chemaxon_Chemicalize [80] is a reference method that was included after the blind challenge submission deadline, and all other method names refer to blind predictions. Methods are listed out in Table 3 and statistics calculated for all methods are available in Table S3

Fig. 8

Overall correlation assessment for all methods participating in the SAMPL7 pK_a challenge shows that one Physical (QM) method and one QSPR/ML reference method exhibited modestly better performance than others. Pearson’s R² and Kendall’s Rank Correlation Coefficient Tau ($\tau$) are shown, with error bars denoting 95% confidence intervals obtained by bootstrapping over challenge molecules. Submission methods are listed out in Table 3. REF00_Chemaxon_Chemicalize [80] is a reference method that was included after the blind challenge submission deadline, and all other method names refer to blind predictions. Most methods have a statistically indistinguishable performance on ranking, however, for R², two methods (EC_RISM [92], REF_Chemaxon_Chemicalize), tend to have a greater ranking ability than the other methods. Evaluation statistics calculated for all methods are available in Table S3 of the Supplementary Information

Fig. 9

Predicted vs. experimental value correlation plots of 2 best performing methods and one representative average method in the SAMPL7 pK_a challenge. Dark and light green shaded areas indicate 0.5 and 1.0 units of error. Error bars indicate standard error of the mean of predicted and experimental values. Some SEM values are too small to be seen under the data points. Method DFT_M05-2X_SMD [94] was selected as the method with the median RMSE of all ranked methods analyzed in the challenge. Performance statistics of these methods is available in Table 4

Fig. 10

Molecule-wise prediction error distribution plots show the prediction accuracy for individual molecules across all prediction methods for the pK_a challenge. Molecules are labeled with their compound class as a reference. A The MAE of each molecule separated by method category suggests the most challenging molecules were different for each method category. It is difficult to draw statistically significant conclusions where there are large overlapping confidence intervals. The QM+LEC method category appears to be less accurate for the majority of the molecules compared to the other method categories. QSPR/ML methods performed better for isoxazoles (SM41-SM43) and 1,2,3-triazoles (SM44-SM46) compared to the other two method categories. Physical QM-based methods performed poorly for acylsulfonamides (SM26 and SM25). B Error distribution for each molecule over all prediction methods. SM25 has the most spread in pK_a prediction error

Fig. 11

Chemical transformations that lead to common sign disagreements among participants typically involve a protonated nitrogen in terminal nitrogen groups, 1,2,3-triazoles, and isoxazoles. Shown are some chemical transformations that repeatedly show up as having large disagreement on the sign of the relative free energy prediction, as seen in Fig. 13

Fig. 12

The average relative microstate free energy predicted per microstate and the distribution across predictions in the SAMPL7 pK_a challenge show how varied predictions were. Molecules are labeled with their compound class as a reference. A The average relative microstate free energy predicted per microstate. Error bars are the standard deviation of the relative microstate free energy predictions. A lower standard deviation indicates that predictions for a microstate generally agree, while a larger standard deviation means that predictions disagree. Predictions made for microstates such as SM25_micro001, SM26_micro002, SM28_micro001, SM43_micro003, SM46_micro003 widely disagree, while predictions for microstates such as SM26_micro004 are in agreement. B Distribution for each relative microstate free energy prediction over all prediction methods shows how prediction agreement among methods varied depending on the microstate

Fig. 13

The Shannon entropy (H) per microstate transition shows that participants disagree on many of the signs of the relative free energy predictions. Microstates with entropy values greater than 0 reflect increasing disagreement in the predicted sign. Microstates with an entropy of 0 are not shown here, but indicate that methods made predictions which had the same sign for the free energy change associated with a particular transition. About 44% of all microstates predictions disagreed with one another based on the sign, and the rest agreed. Roughly 5% of microstates strongly disagreed on the sign of predictions—meaning that predicted relative free energies were fairly evenly split between positive, neutral, and negative values. This indicates that these transitions were particularly challenging

Fig. 14

Structures of microstates where relative microstate free energy predictions disagree. Shown are some of the microstate transitions where participants predictions largely disagree with one another, based on Fig. 12. The average relative free energy prediction ($\mathrm{\Delta }$G) along with the standard deviation are listed under each transition

Fig. 15

Overall accuracy assessment for log D estimation. Both root-mean-square error (RMSE) and mean absolute error (MAE) are shown, with error bars denoting 95% confidence intervals obtained by bootstrapping over challenge molecules. REF00_ChemAxon [80] is a reference method and NULL0 is a null method that was included after the blind challenge submission deadline, and all other method names refer to blind predictions. Methods are listed out in Table 5 and statistics calculated for all methods are available in Table S4

Fig. 16

Predicted vs. experimental value correlation plots of all log D estimation methods in the SAMPL7 challenge. Dark and light green shaded areas indicate 0.5 and 1.0 units of error. Error bars indicate standard error of the mean of predicted and experimental values. Some SEM values are too small to be seen under the data points. Performance statistics of all methods is available in Table S4

Fig. 17

log D values from a combination of the best pK_a and log P are typically superior. Shown is the RMSE in calculated log D values, with error bars denoting 95% confidence intervals from bootstrapping over challenge molecules. This plot is similar to Fig. 4, except it includes some additional pK_a and log P combinations (for log D estimation). Method logP_experimental + EC_RISM combines the experimental log P with the top performing pK_a method (based on RMSE). Method logP_experimental + pKa_experimental combines the experimental log P and pK_a value. Method TFE MLR + EC_RISM combines the best performing (based on RMSE) log P and pK_a methods. Method TFE MLR + pKa_experimental combines the best performing (based on RMSE) log P method with the experimental pK_a. Method logP_experimental + DFT_M05-2X_SMD combines the experimental log P with an average performing pK_a method. Method NES-1 (GAFF2/OPC3) B + pKa_experimental combines a log P method with average performance with the experimental pK_a. All other methods are the same as in Fig. 4