Fudging the volcano-plot without dredging the data

Abstract

Selecting omic biomarkers using both their effect size and their differential status significance (i.e., selecting the “volcano-plot outer spray”) has long been equally biologically relevant and statistically troublesome. However, recent proposals are paving the way to resolving this dilemma.

Full text

Comment
https://doi.org/10.1038/s41467-024-45834-7
Fudging the volcano-plot without dredging
the data
Thomas Burger Check for updates
Selecting omic biomarkers using both their
effect size and their differential status sig-
nificance (i.e., selecting the “volcano-plot outer
spray”) has long been equally biologically rele-
vant and statistically troublesome. However,
recent proposals are paving the way to resolving
this dilemma.
In their recent Nature Communications article, Bayer et al. present the
tool CurveCurator1to select biomarkers according to their dose-
response profiles, with well-established statistical guarantees. To
conveniently blend the effect size and the significance of the dose-
response curve into a single relevance score, they revisit the so-called
fudge factor introduced in the SAM test2. Moreover, to overcome the
risk of involuntary data dredging inherent to “fudging”the differential
analysis3, they propose a new approach inspired by the target-decoy
competition framework (TDC4). The principle of TDC is to add coun-
terfactual amino acid sequences (termed decoys) to a (target) data-
base of real amino acid sequences, as to mimic erroneous matches in a
peptide identification task. Despite its original empirical-only justifi-
cations (peptide matches involving decoy sequences should be as
probable as mismatches involving target sequences), TDC has long
been used in mass spectrometry-based proteomics to validate peptide
identifications according to a False Discovery Rate (FDR5) threshold.
Accordingly, Bayer et al. claim FDR control guarantees regardless of
the fudge factor tuning. Several recent works in selective inference (a
subfield of high-dimensional statistics) have provided theoretical
support to their intuition6,7, which justify its generalization to a variety
of similar situations. Concretely, this comment asserts that essentially
any omics data analysis involving a volcano-plot is concerned –be it
transcriptomics, metabolomics, proteomics or any other; either at
bulk or single cell resolution. Therefore, elaborating on Bayer et al.
visionary proposal should lead to new user-tailored computational
omic tools, with sweeping consequences from the application
standpoint.
Issues pertaining to the fudge factor
While the fudge factor was originally introduced as a small positive
constant (denoted as s0) to improve the independence of the test
statistic variance and of the omic feature expression, its tuning to a
larger value has been observed to yielda user-defined weighting of the
significance and of the effect size. Concomitantly, the permutation-
based procedure of SAM test has sometimes been replaced by classical
p-value adjustment –as prescribed in the Benjamini-Hochberg (BH)
procedure for FDR control5. Applying simultaneously these two tricks
enhances volcano-plot interpretation: the biomarkers selected are
located in the outer spray of the volcano-plot, with selection
boundaries following hyperbolic contours (see Fig. 1). Unfortunately
doing so jeopardizes the statistical guarantees: briefly, a too large s0
value distorts the p-values as well as the subsequent adjusted p-values
calculated in the BH procedure. To cope with this, itis either necessary
to constrain the tuning of s0(at the cost of less flexible selection of the
outerspray)ortoreplaceBHprocedurebyanotherFDRcontrol
method that does not require any p-value adjustment. Although the
permutation-based procedure associated to SAM test is an option, it
does not strictly controls for the FDR (see Table 1). Bayer et al. have
thus explored another option inspired by TDC, which has emerged
nearly twenty years ago in proteomics in absence of p-values to assess
the significance of peptide identification.
Competition-based alternatives to control for the FDR
Although published a decade later, the most convincing theoretical
support of TDC to date has been knock-off filters (or KO)6,7.Inspiteof
minor discrepancies with TDC8, KO mathematically justifies TDC gen-
eral approach to FDR control, as well as its main computational steps.
Notably, it demonstrates that FDR can be controlled on a biomarker
selection task by thresholding a contrast of relevance scores, which
results from a pairwise competition between the real putative
Fig. 1 | A typical volcano-plot. Asignificance measure is depicted on the Y-axis
(here, -log10(p-value)) and an effect size is depicted on the X-axis (here, the loga-
rithmized fold-change). The blue lines represent the contours of the relevance
score and the points highlighted in red are those selected according to a knockoff
procedure.
nature communications (2024) 15:1392 | 1
1234567890():,;
1234567890():,;

biomarkers and other ones, fictionalized –respectively referred to as
decoys and knock-offs in the proteomic and statistic parlances. Intui-
tively, the proportion of fictionalized features selected should be a
decent proxy of the ratio of false discoveries [Nota Bene:InKOtheory,
this proportion is corrected by adding 1 to the ratio numerator to cope
for a bias issue. Although this bias is still investigated9,thissuggeststo
correct for Eq. 16 in1by adding 1 to the numerator too.], as long as the
decision is made symmetrically (i.e., their relevance score is attributed
regardless of their real/fictional status). However, despite conceptual
similarities, the problems solvable by TDC and KO differ: For the for-
mer, features are classically amino acid sequences; while for the latter,
a quantitative dataset describing biomolecular expression levels in
response to various experimental conditions is classically considered.
In this context, the TDC extension proposed in CurveCurator to pro-
cess quantitative dose-response curves constitutes a nice bridge
between the TDC and KO kingdoms.
Generalizing the CurveCurator approach
With this in mind, the pragmatic fallouts of Bayer et al. become strik-
ing. Any data analyst wishing to select omic biomarkers with a rele-
vance score picturing hyperbolic contours on a volcano plot (see Fig. 1)
can easily adapt CurveCurator approach to their own case, by follow-
ing the above procedure:
(1) Perform statistical tests to obtain a p-value for each putative
biomarker that assess the significance of its differential status,
(2) Likewise, compute the biomarker fold-change, as a measure of the
effect size, and construct the volcano-plot,
(3) Tune s0to blend the significance of the differential status and the
effect size into a single relevance score,
(4) Acknowledge the relevance score looks like a p-value even though
it may not be valid to use it as such, depending on the s0chosen,
(5) Rely on the KO framework (e.g., using the “knockoff”Rpackage
(https://cran.r-project.org/web/packages/knockoff/index.html)
as well as on the numerous tutorials available (https://web.
stanford.edu/group/candes/knockoffs/software/knockoffs/)to
control for the FDR on the biomarker selected according to the
relevance score, in a way similar to that of CurveCurator.
Different FDR control frameworks for different situations
An important and possibly troublesome feature of Fig. 1is that some
“unselected”black points are surrounded by “selected”red ones. In
other words, some putative biomarkers may not be retained while
other ones with smaller effect size and larger raw p-value are. This is a
classical drawback of competition-based FDR control methods: each
putative biomarker being retained or not does not only depend on its
features, but also on those of its fictionalized counterpart, which
generation is subject to randomness. Although this weakness can be
addressed too, it requires less straightforward tools10. Another still
open problem in KO theory lies in the KO/decoy generation, which can
be difficult depending on the dataset. With this respect, the approach
of CurveCurator is worthwhile. More generally, no method is perfect:
KO filters, like p-value adjustment or permutation-based control have
pros and cons (see Table 1). Therefore, depending on the data analyst
‘need, the preferred method should change. Considering this need for
multiple off-the-shelf tools, it is important to noticethat KO filters have
hardly spread beyond the theoretical community so far,and that their
applications to enhance data analysis in biology-centered investiga-
tions are still scarce, unfortunately. In this context, the seminal pro-
posal of Bayer et al. can be expected to foster the translation of these
fast-evolving theories into practical and efficient software with grow-
ing importance in biomarker discoveries, and they must be acknowl-
edged for this.
Thomas Burger
1
1
Univ.GrenobleAlpes,INSERM,CEA,UA13BGE,CNRS,CEA,FR2048
ProFI, 38000 Grenoble, France. e-mail: thomas.burger@cea.fr
Received: 21 December 2023; Accepted: 2 February 2024;
References
1. Bayer, F. P., Gander, M., Kuster, B. & The, M. CurveCurator: a recalibrated F-statistic to
assess, classify, and explore significance of dose–response curves. Nat. Commun. 14,
7902 (2023).
2. Tusher, V. G., Tibshirani, R. & Chu, G. Significance analysis of microarrays applied to the
ionizing radiation response. Proc. Natl Acad. Sci. 98,5116–5121 (2001).
3. Giai Gianetto, Q.,Couté, Y., Bruley, C.& Burger, T. Uses and misuses of thefudge factor in
quantitative discovery proteomics. Proteomics 16,1955–1960 (2016).
4. Elias, J. E. & Gygi,S. P. Target-decoy searchstrategy for increased confidence in large-scale
protein identificationsbymassspectrometry.Nat. Methods 4,207–214 (2007).
5. Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: a practical and powerful
approach to multiple testing. J. R. Stat. Soc.: Ser. B (Methodol.) 57,289–300 (1995).
6. Barber, R. F. & Candès, E. J. Controlling the falsediscovery rate via knockoffs. Ann. Stat. 43,
2055–2085 (2015).
7. Candès, E., Fan, Y., Janson, L. & Lv, J. Panning for gold:‘model-X’knockoffs for high
dimensional controlled variable selection. J. R. Stat. Soc. Ser. B: Stat. Methodol. 80,
551–577 (2018).
Table 1 | Pros and cons of the various approach to FDR control with respect to selecting biomarkers on the outer spray of the
volcano-plot
Approach to FDR control Advantages Disadvantages
P-value adjustment/
q-value
•Standard, easy to apply and computationally efficient. •Requires well-calibrated p-values.
•Issue with FC filtering3,11,12, either following hyperbolic contours
or not.
Empirical Bayes/null •Can cope for most of the drawbacks of the above methods (p-
value calibration and FC interaction).
•Requires the capability to tune the priors.
•Does not have frequentist interpretation, which may hinder objec-
tive significance assessment13.
Permutations •The multiple test correction is non-parametric.
•No calibration issue.
•Related works based on FDP bounding authorize double-
dipping14.
•Strictly speaking, does not control for the FDR; Instead , it provides a
probabilistic upper bound to the FDP15.
•The fudge factor should not be tuned in contradiction to the sta-
tistical guidelines2.
Knock-offs/decoys •Flexibility of the relevance score. •Instable w.r.t. KO generation10.
•Difficulty of assessing the KO generation (which can lead to overly
conservative FDR control).
Comment
nature communications (2024) 15:1392 | 2

8. Etourneau, L. & Burger, T. Challenging targets or describing mismatches? A comment on
common decoydistribution by Madej et al. J. Proteome Res. 21, 2840–2845 (2022).
9. Rajchert, A. & Keich, U. Controlling the false discovery rate via competition: Is the+ 1 nee-
ded? Stat. Probab. Lett. 197, 109819 (2023).
10. Nguyen, T. B., Chevalier, J. A., Thirion, B., & Arlot, S. (2020,November). Aggregation of
multipleknockoffs.In InternationalConferenceon Machine Learning(pp. 7283-7293). PMLR.
11. McCarthy, D. J. & Smyth, G. K. Testing significance relative to a fold-change threshold is a
TREAT. Bioinformatics 25,765–771 (2009).
12. Ebrahimpoor,M. & Goeman, J. J. Inflated false discovery rate due tovolcano plots:problem
and solutions. Brief. Bioinform. 22, bbab053 (2021).
13. Burger, T. CanOmics Biology Go Subjective because of Artificial Intelligence? A Comment
on “Challenges and Opportunities for Bayesian Statistics in Proteomics”by Crook et al. J.
Proteome Res. 21,1783–1786 (2022).
14. Enjalbert-Courrech, N. & Neuvial, P. Powerful and interpretable control of false discoveries
in two-group differential expression studies. Bioinformatics 38,5214–5221 (2022).
15. Hemerik, J. & Goeman,J. J. False discovery proportion estimation by permutations: con-
fidence for significance analysis of microarrays. J. R. Stat. Soc. Ser. B: Stat. Methodol. 80,
137–155 (2018).
Acknowledgements
This work was supported by grants from the French National Research Agency: ProFI project
(ANR-10-INBS-08), GRAL CBH project(ANR-17-EURE-0003) and MIAI@ GrenobleAlpes (ANR-19-
P3IA-0003).
Author contributions
Conceptualization (TB), bibliography (TB), analysis (TB), manuscript writing (TB).
Competing interests
The author declares no competing interests.
Additional information
Correspondence and requests for materials should be addressed to Thomas Burger.
Peer review information Nature Communications thanks the anonymous reviewer(s) for their
contribution to the peer review of this work.
Reprints and permissions information is available at
http://www.nature.com/reprints
Publisher’snoteSpringer Nature remains neutral with regard to jurisdictional claims in pub-
lished maps and institutional affiliations.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International
License,which permitsuse, sharing,adaptation, distributionand reproductionin any medium or
format,as long as you give appropriate credit to the original author(s) and thesource, provide a
link to the Creative Commons licence, and indicate if changes were made. The imagesor other
third party materialin this article are included in thearticle’s Creative Commons licence, unless
indicated otherwise in a credit line to the material. If material is not included in the article’s
Creative Commons licence and your intended use is not permitted by statutory regulation or
exceeds thepermitted use, you will need to obtain permission directly from the copyright
holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
© The Author(s)2024
Comment
nature communications (2024) 15:1392 | 3