Figure - available from: BMC Bioinformatics
This content is subject to copyright. Terms and conditions apply.
Overview of the regional splicing constraint model. A The per-site splicing substitution rate by reference allele and Sum SpliceAI score bin across autosomal protein-coding genes. The rate of no substitutions across all SpliceAI score bins for each reference allele is A > A = 0.9003, C > C = 0.8565, G > G = 0.8433, and T > T = 0.9347 B Calculating an Observed over Expected (O/E) ratio for a genomic region by counting the number of variants in that region from gnomAD and the number of expected variants with a given SpliceAI score. C The O/E score distribution. Smaller O/E scores indicate higher constraint against splicing, while larger O/E scores indicate lower constraints against splicing. (O/E plot truncated at -2000 to 2000 for visibility) D Representation of regional splicing constraint O/E scores across a hypothetical gene. The presence of gnomAD variants, in gray and the SpliceAI prediction for each position in the gene, in shades of red influences the splicing-specific observed and expected counts in a region. gnomAD variants with higher SpliceAI scores show evidence for more tolerance against splicing variation. In contrast, sites with a higher SpliceAI score and no gnomAD variant show evidence for less tolerance against splicing. Pathogenic splicing variants, in black, are commonly absent from gnomAD and have predictions of alternative splicing from SpliceAI. In this example, the regional constraint model identifies constraint signals at regions that harbor pathogenic splicing variants, such as at canonical splice regions (^) and cryptic splice regions (^^). All genomic positions in C without a SpliceAI score should be recognized as sites with a SpliceAI score < 0.1
Source publication
Background
Despite numerous molecular and computational advances, roughly half of patients with a rare disease remain undiagnosed after exome or genome sequencing. A particularly challenging barrier to diagnosis is identifying variants that cause deleterious alternative splicing at intronic or exonic loci outside of canonical donor or acceptor spli...
Citations
... Notably, our exclusion of mutations that generate cryptic splice sites further reduces the utility of SpliceAI for ESM evaluation in our database. This is consistent with recent findings where SpliceAI or similar tools were used to evaluate mutations located outside of the splice sites ( 69 ). ...
It is now widely accepted that aberrant splicing of constitutive exons is often caused by mutations affecting cis-acting splicing regulatory elements (SREs), but there is a misconception that all exons have an equal dependency on SREs and thus a similar vulnerability to aberrant splicing. We demonstrate that some exons are more likely to be affected by exonic splicing mutations (ESMs) due to an inherent vulnerability, which is context dependent and influenced by the strength of exon definition. We have developed VulExMap, a tool which is based on empirical data that can designate whether a constitutive exon is vulnerable. Using VulExMap, we find that only 25% of all exons can be categorized as vulnerable, whereas two-thirds of 359 previously reported ESMs in 75 disease genes are located in vulnerable exons. Because VulExMap analysis is based on empirical data on splicing of exons in their endogenous context, it includes all features important in determining the vulnerability. We believe that VulExMap will be an important tool when assessing the effect of exonic mutations by pinpointing whether they are located in exons vulnerable to ESMs.
... HAL [36] takes a distinct approach by training on a library of randomized sequences and their experimentally observed splicing patterns, while MMSplice [37] combines the training data from HAL with features derived from primary sequence and additional modules trained on annotated splice sites and clinical variants. Finally, ConSpliceML [38] is a metaclassifier that combines SQUIRLS and SpliceAI scores with a population-based constraint metric measuring the regional depletion of predicted splice-disruptive variants among apparently healthy adults in population databases. While these are, for the most part, general-purpose short-variant predictors, other tools have been purpose-built for more specialized contexts, e.g., synonymous variants and deep intronic variants [39][40][41]. ...
... Given the proliferation of splicing predictors and their utility in variant interpretation, it is important to understand their performance characteristics. Previous comparisons have suggested that overall, SpliceAI represents the state-of-the art, with several other algorithms including MMSplice, SQUIRLS, and ConSpliceML showing competitive or in some cases better performance [32,34,38,[42][43][44][45][46][47]. However, benchmarking efforts to date primarily relied upon curated sets of clinical variants [32,38,42,[44][45][46][47], which are strongly enriched for canonical splice site mutations [35,42,[47][48][49][50] likely due to the relative ease of their classification. ...
... Previous comparisons have suggested that overall, SpliceAI represents the state-of-the art, with several other algorithms including MMSplice, SQUIRLS, and ConSpliceML showing competitive or in some cases better performance [32,34,38,[42][43][44][45][46][47]. However, benchmarking efforts to date primarily relied upon curated sets of clinical variants [32,38,42,[44][45][46][47], which are strongly enriched for canonical splice site mutations [35,42,[47][48][49][50] likely due to the relative ease of their classification. This leaves open the question of how well these tools' performance may generalize, and whether certain tools may excel in particular contexts (e.g., for exonic cryptic splice activating mutations). ...
Background
Variants that disrupt mRNA splicing account for a sizable fraction of the pathogenic burden in many genetic disorders, but identifying splice-disruptive variants (SDVs) beyond the essential splice site dinucleotides remains difficult. Computational predictors are often discordant, compounding the challenge of variant interpretation. Because they are primarily validated using clinical variant sets heavily biased to known canonical splice site mutations, it remains unclear how well their performance generalizes.
Results
We benchmark eight widely used splicing effect prediction algorithms, leveraging massively parallel splicing assays (MPSAs) as a source of experimentally determined ground-truth. MPSAs simultaneously assay many variants to nominate candidate SDVs. We compare experimentally measured splicing outcomes with bioinformatic predictions for 3,616 variants in five genes. Algorithms’ concordance with MPSA measurements, and with each other, is lower for exonic than intronic variants, underscoring the difficulty of identifying missense or synonymous SDVs. Deep learning-based predictors trained on gene model annotations achieve the best overall performance at distinguishing disruptive and neutral variants, and controlling for overall call rate genome-wide, SpliceAI and Pangolin have superior sensitivity. Finally, our results highlight two practical considerations when scoring variants genome-wide: finding an optimal score cutoff, and the substantial variability introduced by differences in gene model annotation, and we suggest strategies for optimal splice effect prediction in the face of these issues.
Conclusion
SpliceAI and Pangolin show the best overall performance among predictors tested, however, improvements in splice effect prediction are still needed especially within exons.
... SpliceAI [ 10 ] is widely recognized as the most successful method of this kind, although its performance has been shown to vary across studies and datasets considered [ 5 ]. Recently, new models have been developed based on SpliceAI, either combining its predictions with other sources of information (such as genetic constraint for ConSpliceML [ 25 ] and PDIVAS [ 26 ] or tissue-specific splice site usage for AbSplice-DNA [ 27 ]) or creating an entir el y ne w model based on SpliceAI arc hitectur e . For example , Pangolin [ 28 ] uses splicing quantifications from multiple species and tissues to not only predict whether a position is a splice site (as SpliceAI does) but also to predict splice site usage (e.g., how m uc h a splice site is being used in a given tissue). ...
Background
The adoption of whole-genome sequencing in genetic screens has facilitated the detection of genetic variation in the intronic regions of genes, far from annotated splice sites. However, selecting an appropriate computational tool to discriminate functionally relevant genetic variants from those with no effect is challenging, particularly for deep intronic regions where independent benchmarks are scarce.
Results
In this study, we have provided an overview of the computational methods available and the extent to which they can be used to analyze deep intronic variation. We leveraged diverse datasets to extensively evaluate tool performance across different intronic regions, distinguishing between variants that are expected to disrupt splicing through different molecular mechanisms. Notably, we compared the performance of SpliceAI, a widely used sequence-based deep learning model, with that of more recent methods that extend its original implementation. We observed considerable differences in tool performance depending on the region considered, with variants generating cryptic splice sites being better predicted than those that potentially affect splicing regulatory elements. Finally, we devised a novel quantitative assessment of tool interpretability and found that tools providing mechanistic explanations of their predictions are often correct with respect to the ground - information, but the use of these tools results in decreased predictive power when compared to black box methods.
Conclusions
Our findings translate into practical recommendations for tool usage and provide a reference framework for applying prediction tools in deep intronic regions, enabling more informed decision-making by practitioners.
... The ConSplice feature was obtained from the scoreprecomputed bed file of the best_splicing_constraint_ model provided by the authors [20]. MaxEntScan prediction of the variant's effect on splicing was performed using the plugin module of VEP [21][22][23]. ...
... Predictors to be compared to PDIVAS were SpliceAI, Pangolin [24], ConSpliceML [20], MaxEntScan, and CADD-Splice [25], which were selected based on the following criteria: (1) the program or the precomputed score file is freely available, (2) the program can assess deep-intronic variants, and (3) the program is operated in a Linux environment and can be applied to large-scale variant analysis. SQUIRLS and SPiP also matched these criteria, but their developers recognized their lower performance on deep-intronic SAVs because of the limited number of training datasets [26,27]. ...
... Therefore, a more specific prediction was achieved by combining a deleterious prediction with a human splicing constraint metric from ConSplice, which models mutational constraints on splice-altering variants within the human population. The effectiveness of this approach in predicting deleterious splicing was demonstrated in the ConSpliceML predictor [20]. Likewise, we employed ConSplice for more specialized usage for deep-intronic SAVs. ...
Background
Deep-intronic variants that alter RNA splicing were ineffectively evaluated in the search for the cause of genetic diseases. Determination of such pathogenic variants from a vast number of deep-intronic variants (approximately 1,500,000 variants per individual) represents a technical challenge to researchers. Thus, we developed a Pathogenicity predictor for Deep-Intronic Variants causing Aberrant Splicing (PDIVAS) to easily detect pathogenic deep-intronic variants.
Results
PDIVAS was trained on an ensemble machine-learning algorithm to classify pathogenic and benign variants in a curated dataset. The dataset consists of manually curated pathogenic splice-altering variants (SAVs) and commonly observed benign variants within deep introns. Splicing features and a splicing constraint metric were used to maximize the predictive sensitivity and specificity, respectively. PDIVAS showed an average precision of 0.92 and a maximum MCC of 0.88 in classifying these variants, which were the best of the previous predictors. When PDIVAS was applied to genome sequencing analysis on a threshold with 95% sensitivity for reported pathogenic SAVs, an average of 27 pathogenic candidates were extracted per individual. Furthermore, the causative variants in simulated patient genomes were more efficiently prioritized than the previous predictors.
Conclusion
Incorporating PDIVAS into variant interpretation pipelines will enable efficient detection of disease-causing deep-intronic SAVs and contribute to improving the diagnostic yield. PDIVAS is publicly available at https://github.com/shiro-kur/PDIVAS.
Graphical abstract
... Given the proliferation of splicing predictors and their utility in variant interpretation, it is important to understand their performance characteristics. Previous comparisons have suggested that overall, SpliceAI represents the state-of-the art, with several other algorithms including MMSplice, SQUIRLS, and ConSpliceML showing competitive or in some cases better performance 30,32,[36][37][38][39][40][41][42] . However, benchmarking efforts to date primarily relied upon curated sets of clinical variants 30,36,37,[39][40][41][42] , which are strongly enriched for canonical splice site mutations 33,37,[42][43][44][45] likely due to the relative ease of their classification. ...
... Previous comparisons have suggested that overall, SpliceAI represents the state-of-the art, with several other algorithms including MMSplice, SQUIRLS, and ConSpliceML showing competitive or in some cases better performance 30,32,[36][37][38][39][40][41][42] . However, benchmarking efforts to date primarily relied upon curated sets of clinical variants 30,36,37,[39][40][41][42] , which are strongly enriched for canonical splice site mutations 33,37,[42][43][44][45] likely due to the relative ease of their classification. This leaves open the question of how well these tools' performance may generalize, and whether certain tools may excel in particular contexts (e.g., for exonic cryptic splice activating mutations). ...
... As we found, SpliceAI was often but not always the top performer in these past comparisons [37][38][39][40][41][42]93 . Together, these our results suggest opportunities for metaclassifiers to better calibrate existing predictors and to leverage each within its strongest domain 36,38 . ...
Background
Variants that disrupt mRNA splicing account for a sizable fraction of the pathogenic burden in many genetic disorders, but identifying splice-disruptive variants (SDVs) beyond the essential splice site dinucleotides remains difficult. Computational predictors are often discordant, compounding the challenge of variant interpretation. Because they are primarily validated using clinical variant sets heavily biased to known canonical splice site mutations, it remains unclear how well their performance generalizes.
Results
We benchmarked eight widely used splicing effect prediction algorithms, leveraging massively parallel splicing assays (MPSAs) as a source of experimentally determined ground-truth. MPSAs simultaneously assay many variants to nominate candidate SDVs. We compared experimentally measured splicing outcomes with bioinformatic predictions for 3,616 variants in five genes. Algorithms' concordance with MPSA measurements, and with each other, was lower for exonic than intronic variants, underscoring the difficulty of identifying missense or synonymous SDVs. Deep learning-based predictors trained on gene model annotations achieved the best overall performance at distinguishing disruptive and neutral variants. Controlling for overall call rate genome-wide, SpliceAI and Pangolin also showed superior overall sensitivity for identifying SDVs. Finally, our results highlight two practical considerations when scoring variants genome-wide: finding an optimal score cutoff, and the substantial variability introduced by differences in gene model annotation, and we suggest strategies for optimal splice effect prediction in the face of these issues.
Conclusion
SpliceAI and Pangolin showed the best overall performance among predictors tested, however, improvements in splice effect prediction are still needed especially within exons.
... The ConSplice feature was obtained from the score-precomputed bed file of the best_splicing_constraint_model provided by the authors [20]. MaxEntScan prediction of the variant's effect on splicing was performed using the plugin module of VEP [21][22][23]. ...
... Predictors to be compared to PDIVAS were SpliceAI, Pangolin [24], ConSpliceML [20], MaxEntScan, and CADD-Splice [25], which were selected based on the following criteria: 1) the program or the precomputed score file is freely available, 2) the program . CC-BY 4.0 International license It is made available under a perpetuity. ...
... Therefore, a more specific prediction was achieved by combining a deleterious prediction with a human splicing constraint metric from ConSplice, which models mutational constraints on splice-altering variants within the human population. The effectiveness of this approach in predicting deleterious splicing was demonstrated in the ConSpliceML predictor [20]. Likewise, we employed ConSplice is the author/funder, who has granted medRxiv a license to display the preprint in (which was not certified by peer review) preprint (Fig. 2h). ...
Deep-intronic variants often cause genetic diseases by altering RNA splicing. However, these pathogenic variants are overlooked in whole-genome sequencing analyses, because they are quite difficult to segregate from a vast number of benign variants (approximately 1,500,000 deep-intronic variants per individual). Therefore, we developed the Pathogenicity predictor for Deep-intronic Variants causing Aberrant Splicing (PDIVAS), an ensemble machine-learning model combining multiple splicing features and regional splicing constraint metrics. Using PDIVAS, around 27 pathogenic candidates were identified per individual with 95% sensitivity, and causative variants were more efficiently prioritized than previous predictors in simulated patient genome sequences. PDIVAS is available at https://github.com/shiro-kur/PDIVAS.
... SpliceAI [10] is widely recognized as the most successful method of this kind, although its performance has been shown to vary across studies and datasets considered [5]. Recently, new models have been developed based on SpliceAI, either combining its predictions with other sources of information (such as genetic constraint for ConSpliceML [24] or tissue-specific splice site usage for AbSplice-DNA [25]) or creating an entirely new model based on SpliceAI architecture. For example, Pangolin [26] uses splicing quantifications from multiple species and tissues to not only predict whether a position is a splice site (as SpliceAI does) but also to predict splice site usage (e.g., how much a splice site is being used in a given tissue). ...
The adoption of whole genome sequencing in genetic screens has facilitated the detection of genetic variation in the intronic regions of genes, far from annotated splice sites. However, selecting an appropriate computational tool to differentiate functionally relevant genetic variants from those with no effect is challenging, particularly for deep intronic regions where independent benchmarks are scarce.
In this study, we have provided an overview of the computational methods available and the extent to which they can be used to analyze deep intronic variation. We leveraged diverse datasets to extensively evaluate tool performance across different intronic regions, distinguishing between variants that are expected to disrupt splicing through different molecular mechanisms. Notably, we compared the performance of SpliceAI, a widely used sequence-based deep learning model, with that of more recent methods that extend its original implementation. We observed considerable differences in tool performance depending on the region considered, with variants generating cryptic splice sites being better predicted than those that affect splicing regulatory elements or the branchpoint region. Finally, we devised a novel quantitative assessment of tool interpretability and found that tools providing mechanistic explanations of their predictions are often correct with respect to the ground truth information, but the use of these tools results in decreased predictive power when compared to black box methods.
Our findings translate into practical recommendations for tool usage and provide a reference framework for applying prediction tools in deep intronic regions, enabling more informed decision-making by practitioners.
