No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts
Abstract
This study focuses on a recently introduced type of indicator measuring disruptiveness in science. Disruptive research diverges from current lines of research by opening up new lines. In the current study, we included the initially proposed indicator of this new type (Funk & Owen-Smith, 2017; Wu, Wang, & Evans, 2019) and several variants with DI1: DI5, DI1n, DI5n, and DEP. Since indicators should measure what they propose to measure, we investigated the convergent validity of the indicators. We used a list of milestone papers, selected and published by editors of Physical Review Letters, and investigated whether this human (experts)-based list is related to values of the several disruption indicators variants and – if so – which variants show the highest correlation with expert judgements. We used bivariate statistics, multiple regression models, and (coarsened) exact matching (CEM) to investigate the convergent validity of the indicators. The results show that the indicators correlate differently with the milestone paper assignments by the editors. It is not the initially proposed disruption index that performed best (DI1), but the variant DI5 which has been introduced by Bornmann, Devarakonda, Tekles, and Chacko (2020a). In the CEM analysis of this study, the DEP variant – introduced by Bu, Waltman, and Huang (in press) – also showed favorable results.
... Another approach to get rid of was suggested by Bu et al. (2021) in a paper that introduced the dependency indicator (DEP). 5 The "DEP is defined as the average number of citation links 5 The original name of DEP is actually MR[cited_pub], but this paper follows the denotation adopted by the subsequent literature, specifically Bornmann et al. (2020a), Bornmann and Tekles (2021), and Bittmann et al. ...
... Based on the assumption that milestone assignments are a proxy for disruption, the three studies investigated whether the DI1 and its variants assign higher values to milestone than to non-milestone publications. We will start with two studies that are very similar: Bornmann and Tekles (2021) and Bittmann et al. (2022). ...
... Since bothBornmann and Tekles (2021) andBittmann et al. (2022) mainly relied on CEM, we forgo a detailed explanation of the different types of matching algorithms and focus on CEM instead. Unlike other matching algorithms, CEM tries to find exact matches and actively discards dissimilar cases from the calculation. ...
The purpose of this paper is to provide a review of the literature on the original disruption index (DI1) and its variants in scientometrics. The DI1 has received much media attention and prompted a public debate about science policy implications, since a study published in Nature found that papers in all disciplines and patents are becoming less disruptive over time. This review explains in the first part the DI1 and its variants in detail by examining their technical and theoretical properties. The remaining parts of the review are devoted to studies that examine the validity and the limitations of the indices. Particular focus is placed on (1) possible biases that affect disruption indices (2) the convergent and predictive validity of disruption scores, and (3) the comparative performance of the DI1 and its variants. The review shows that, while the literature on convergent validity is not entirely conclusive, it is clear that some modified index variants, in particular DI5, show higher degrees of convergent validity than DI1. The literature draws attention to the fact that (some) disruption indices suffer from inconsistency, time-sensitive biases, and several data-induced biases. The limitations of disruption indices are highlighted and best practice guidelines are provided. The review encourages users of the index to inform about the variety of DI1 variants and to apply the most appropriate variant. More research on the validity of disruption scores as well as a more precise understanding of disruption as a theoretical construct is needed before the indices can be used in the research evaluation practice.
... Wu et al. (2019) simplified this indicator to measure the degree of disruption caused by an FP to existing research streams. Bornmann et al. (2020) and Bornmann and Tekles (2021) further refined the original disruption indicator by proposing several alternative indicators and examined the convergent validity of these indicators. It should be noted that domain experts also frequently use citation patterns to identify scientific advances (Mugabushaka et al., 2020). ...
... The independent variables are the disruption scores of papers. We ran the regression model with several other factors being controlled, including the number of years since the paper was published, citation counts, and the number of coauthors, to separate their effects from the explanatory variables (Bornmann & Tekles, 2021). The citation counts were logarithmically transformed by log10 to normalize the skewed citation distribution. ...
... Most scientific breakthroughs ranked relatively lower by mED ent ð Þ (Figure 6b), mCD (Figure 6c), and DI 1 (Figure 6e). This finding may indicate that the scientific breakthroughs recognized by various channels often include not only disruptive papers but also developmental research (Bornmann & Tekles, 2021 Figure 7 illustrates the number of ground-truth scientific breakthroughs among the annual top 0.1% disruptive papers in PubMed. The total number of top 0.1% disruptive papers over years rated by mED rel ð Þ , mED ent ð Þ , mCD, DI 5 , and DI 1 are 6,126, 6,215, 6,124, 6,125, and 6,130, respectively. ...
Compared to previous studies that generally detect scientific breakthroughs based on citation patterns, this article proposes a knowledge entity‐based disruption indicator by quantifying the change of knowledge directly created and inspired by scientific breakthroughs to their evolutionary trajectories. Two groups of analytic units, including MeSH terms and their co‐occurrences, are employed independently by the indicator to measure the change of knowledge. The effectiveness of the proposed indicators was evaluated against the four datasets of scientific breakthroughs derived from four recognition trials. In terms of identifying scientific breakthroughs, the proposed disruption indicator based on MeSH co‐occurrences outperforms that based on MeSH terms and three earlier citation‐based disruption indicators. It is also shown that in our indicator, measuring the change of knowledge inspired by the focal paper in its evolutionary trajectory is a larger contributor than measuring the change created by the focal paper. Our study not only offers empirical insights into conceptual understanding of scientific breakthroughs but also provides practical disruption indicator for scientists and science management agencies searching for valuable research.
... This disruption index has had quite an impact on the scientific literature, although they are not the only ones that have been published Bornmann and Tekles, 2021), variants have been proposed Leydesdorff et al., 2021), versions of the indicator adapted to a specific discipline have even been developed Tekles, 2019a;Bornmann et al., 2019). ...
... On the other hand, the scores of different disruption indicators have also been studied. Bornmann and Tekles (2021) and Bittmann et al. (2021) conclude that one of the disruption indicators that works most consistently is the so-called DI 5 defined by . ...
An indicator to measure disruption has recently been proposed (Funk & Owen-Smith, 2017; Wu, Wang, & Evans, 2019) which has given rise to a large number of variants (Bornmann et al., 2020). In this work we are going to focus on the original indicator DI and the one that seems to have a better performance DI5 (Bornmann and Tekles, 2021; Bittmann et al., 2021) carrying out a large-scale study comparing the scores assigned to each paper with other bibliometric indicators. The result is that the papers to which the bibliometric indicators assign more value do not obtain better scores. Reviews and short surveys have higher scores than articles and conference papers. Excellent papers have worse scores than non-excellent ones. Works with international collaboration obtain worse values than those without it. Works published in Q1 journals have worse scores than those published in journals of other quartiles. And there is also a small negative correlation with the normalized impact and with the technological impact.
... Thelwall and Sud (2021) showed that new research topics are cited more in some disciplines and emerging topics were found to benefit from both within-and outside-field citation links (Kwon et al., 2019). Bornmann and Tekles (2021) argue that new topics are built upon so called disruptive publications, i.e. publications characterized by their ability to overthrow established thinking (Bornmann et al., 2020). Newness (or novelty) together with the fast growth of publications in an area is predicted to lead to prominent impact in the future (Rotolo et al., 2015) and recent research has shown that new topics have a citation advantage (Thelwall & Sud, 2021). ...
... The growing number of publications in a topic may be due to disruptive publications existing in the topic (Funk & Owen-Smith, 2017;Wu et al., 2019). Bornmann and Tekles (2021) argue that indicators of disruption is linked to the theory of Kuhn (1996) in which science is not seen as stable and cumulative, but rather as occasionally shifting focus and being transformed by novel ideas and breakthroughs. Disruptive publications may therefore be the starting point of new topics and may spur a growing number of publications in the topic, and thereby a lot of citations to the early publications within the topic. ...
Citations are used for research evaluation, and it is therefore important to know which factors influence or associate with citation impact of articles. Several citation factors have been studied in the literature. In this study we propose a new factor, topic growth, that no previous study has studied empirically. The growth rate of topics may influence future citation counts because a high growth in a topic means there are more publications citing previous publications in that topic. We construct topics using community detection in a citation network and use a two-part regression model to study the association between topic growth and citation counts in eight broad disciplines. The first part of the model uses quantile regression to estimate the effect of growth ratio on citation counts for publications with more than three citations. The second part of the model uses logistic regression to model the influence of the explanatory variables on the probability of being lowly cited versus being modestly or highly cited. Both models control for three variables that may distort the association between the topic growth and citations: journal impact, number of references, and number of authors. The regression model clearly shows that publications in fast-growing topics have a citation advantage compared to publications in slow-growing or declining topics in all of the eight disciplines. Using citation indicators for research evaluation may give incentives for researchers to publish in fast-growing topics, but they may cause research to be less diversified. The results have also some implications for citation normalization.
... Thelwall and Sud (2021) showed that new research topics are cited more in some disciplines and emerging topics were found to benefit from both within-and outside-field citation links (Kwon et al., 2019). Bornmann and Tekles (2021) argue that new topics are built upon so called disruptive publications. Newness (or novelty) together with the fast growth of publications in an area is predicted to lead to prominent impact in the future (Rotolo et al., 2015) and recent research has shown that new topics have a citation advantage (Thelwall & Sud, 2021). ...
... The growing number of publications in a topic may be due to disruptive publications existing in the topic (Funk & Owen-Smith, 2017;Wu et al., 2019). Bornmann and Tekles (2021) argue that indicators of disruption is linked to the theory of Kuhn (1996) in which science is not seen as stable and cumulative, but rather as occasionally shifting focus and being transformed by novel ideas and breakthroughs. Disruptive publications may therefore be the starting point of new topics and may spur a growing number of publications in the topic, and thereby a lot of citations to the early publications within the topic. ...
Citations are used for research evaluation, and it is therefore important to know which factors influence or associate with citation impact of articles. Several citation factors have been studied in the literature. In this study we propose a new factor, topic growth, that no previous study has taken into consideration. The growth rate of topics may influence future citation counts, because a high growth in a topic means there are more publications citing previous publications in that topic. We construct topics using community detection in a citation network and use a two-part regression model is used to study the association between topic growth and citation counts in eight broad disciplines. The first part of the model uses quantile regression to estimate the effect of growth ratio on citation counts for publications with more than three citations. The second part of the model uses logistic regression to model the influence of the independent variables on the probability of being lowly cited versus being modestly or highly cited. Both models control for three variables that may distort the association between the topic growth and citations: journal impact, number of references, and number of authors. The regression model clearly shows that publications in fast-growing topics have a citation advantage compared to publications in slow-growing or declining topics in all of the eight disciplines. Using citation indicators for research evaluation may give incentives for researchers to publish in fast-growing topics, but they may cause research to be less diversified. The results have also some implications for citation normalization.
... The decision not to report alternative justifiable analyses may be arbitrary or motivated by the desire to present certain results (Simmons et al., 2011;Simonsohn et al., 2020). In the case of the DI1 and its variants, there is concrete evidence in the research literature that disruption scores can vary significantly depending on the variant of the index (Bittmann et al., 2022;Bornmann et al., 2020;Bornmann & Tekles, 2021;Deng & Zeng, 2023;Wang et al., 2023) as well as choices made during data processing such as the selection of the citation window (Bornmann & Tekles, 2019a;Liang et al., 2022) or the treatment of data artefacts (Holst et al., 2024;Liang et al., 2022;Ruan et al., 2021). Thus, there is the risk that bibliometric analyses based on the DI1 turn into a "garden of forking paths" (Gelman & Loken, 2014, p. 463) (Steegen et al., 2016), multi-model analysis (Muñoz & Young, 2018;Young & Holsteen, 2017), specification-curve analysis (Simonsohn et al., 2020), and vibration of effects analysis (Palpacuer et al., 2019;Patel et al., 2015;Tierney et al., 2021). ...
Following Funk and Owen-Smith (2017), Wu et al. (2019) proposed the disruption index (DI1) as a bibliometric indicator that measures disruptive and consolidating research. When we summarized the literature on the disruption index for our recently published review article (Leibel & Bornmann, 2024), we noticed that the calculation of disruption scores comes with numerous (hidden) degrees of freedom. In this Letter to the Editor, we explain why this analytical flexibility endangers the credibility of bibliometric research based on the DI1 (and its variants) and advertise the application of multiverse-style methods to increase the transparency of the research.
... The AVE for both PS (.701) and LE (.885) were above the threshold of .50 (Bornmann & Tekles, 2021). ...
Adolescents' interaction with their parents at home is pivotal for their physical and mental health, which can further influence their academic achievement at school. However, every culture has distinctive parenting elements which are distinguishable from other cultures. This study explored the influence of parenting styles on adolescents' academic achievement through the mediation role of learning engagement in a Chinese cultural context. A questionnaire was distributed among parents and students , and 1,557 valid responses were received and analyzed using SPSS software, PROCESS macro, Microsoft Excel, and JASP. The results indicated a positive and significant effect of parenting styles on students' academic achievement, with stu-dents' learning engagement partially mediating such relationship. Dimension-wise, parents' emotional acceptance positively influenced students' academic achievement, but rejecting and overprotecting them had negative repercussions. A comparison between fathers and mothers showed a higher effect of mothers' parenting styles compared to that of fathers. Generally, stu-dents' behaviors result from different forces which act in and outside them, making this study a wake-up call for cherishing collective efforts from parents, teachers, and students toward improving academic performance. Plain Language Summary This study examined how parenting styles in the Chinese cultural context impact adolescents' academic achievement through their level of engagement in learning. Survey data from 1557 participants were analyzed to uncover key findings. The results revealed that parenting styles significantly influence students' academic success, with their level of engagement in learning partially mediating this relationship. Specifically, parents' emotional acceptance positively affected academic achievement, while rejection and overprotection had negative effects. Mothers' parenting styles were found to have a stronger influence than fathers'. Overall, the study highlights the importance of parental involvement and positive interactions in shaping adolescents' academic outcomes. It emphasizes the need for collaboration between parents, teachers, and students to enhance academic performance effectively.
... The AVE for both PS (.701) and LE (.885) were above the threshold of .50 (Bornmann & Tekles, 2021). ...
Adolescents’ interaction with their parents at home is pivotal for their physical and mental health, which can further influence their academic achievement at school. However, every culture has distinctive parenting elements which are distinguishable from other cultures. This study explored the influence of parenting styles on adolescents’ academic achievement through the mediation role of learning engagement in a Chinese cultural context. A questionnaire was distributed among parents and students, and 1,557 valid responses were received and analyzed using SPSS software, PROCESS macro, Microsoft Excel, and JASP. The results indicated a positive and significant effect of parenting styles on students’ academic achievement, with students’ learning engagement partially mediating such relationship. Dimension-wise, parents’ emotional acceptance positively influenced students’ academic achievement, but rejecting and overprotecting them had negative repercussions. A comparison between fathers and mothers showed a higher effect of mothers’ parenting styles compared to that of fathers. Generally, students’ behaviors result from different forces which act in and outside them, making this study a wake-up call for cherishing collective efforts from parents, teachers, and students toward improving academic performance.
... The original disruption index proposed by Wu et al. (2019), namely DI 1 , and its variant indicator, DI 5 , are used in this study. According to the disruption index and its related research (Wu et al. 2019, Bornmann et al. 2020, Bornmann and Tekles 2021, Bu et al. 2021, Leydesdorff and Bornmann 2021, the definition of the disruption index for a focal article is as follows: ...
Although many studies suggest that interdisciplinary research fosters creativity and breakthroughs, there has been no quantitative study to confirm this belief. In recent years, several indicators have been developed to measure novelty or disruption in research. Compared with the citation impact, this type of indicator can more directly characterize research quality and contribution. Based on the F1000 Prime database and Scopus datasets accessed via ICSR Lab, F1000 novelty tags and two disruption indices (DI 1 and DI 5) were used in this study for the assessment of research quality and contribution, and it was explored whether interdisciplinarity is more likely to produce novel or disruptive research. Interestingly, DI 1 and DI 5 exhibit different relationships with F1000 novelty tags; the reason for this may be that DI 5 Scientometrics, 2024 2 highlights disruptive research within a given discipline and amplifies the disruptive signal within that discipline. Furthermore, it is found that interdisciplinarity (RS and LCDiv) is positively associated with F1000 novelty tags and the disruption indices (DI 1 and DI 5). As a result, it is demonstrated that interdisciplinarity helps to produce novel or disruptive research.
... To address this question, we conducted an experiment to examine the performance and efficiency of the k-step h-indices through convergent validity analysis. Convergent validity refers to the extent to which the metrics accurately measure what they are intended to measure (Bornmann & Tekles, 2021;Rowlands, 2018). By utilizing expert-selected items, including Nobel Prize-winning papers and milestone papers identified by the American Physical Society at the paper level, distinguished Physics awards laureates (such as Nobel Prize, Wolf Prize, and Dirac Medal recipients) at the author level, and top-ranking institutions identified by the Shanghai Ranking at the institution level, we can validate the convergent validity of the k-step h-indices. ...
The evaluation of scientific impact plays a crucial role in assessing research contributions. In this study, we introduce the concept of the k-step h-index and investigate its applicability in citation networks at different levels, including papers, authors, and institutions. By incorporating higher generations of citation information, the k-step h-index provides a comprehensive and nuanced measure of scientific influence. It demonstrates exponential growth in k-step citations, capturing valuable information from the Hirsch core and tail. Through power law distribution analysis, we uncover the presence of highly influential entities coexisting with less influential ones, revealing the heterogeneity of impact within citation networks. To validate the effectiveness of the k-step h-index, we utilize a vast dataset from APS, conducting a thorough examination of its consistency and convergent validity. Our findings demonstrate strong correlations between the k-step h-index and conventional metrics, as well as alignment with measures of innovation. This confirms the reliability of the k-step h-index and its ability to capture innovative contributions. Notably, when compared to benchmarks, the k-step h-index outperforms in accurately ranking expert-selected items, including milestone papers, distinguished authors, and prestigious institutions. Higher values of the k-step h-index consistently exhibit superior performance, showcasing their predictive power in identifying prominent scientific entities. These findings hold significant implications for research evaluation, policy-making, and strategic planning, as they pave the way for a more holistic understanding of scholarly contributions.
... The formula below was used to calculate the validity, and the results indicated the acceptability of the model, for the AVE values in all the constructs ranged from .701 to .885, of which were all above the threshold of .50 (Bornmann & Tekles, 2021). ...
The effect of media illiteracy on adolescents’ and youths’ mental growth is underrated in most of the developing countries. Notwithstanding, media influences almost everything in today’s life among adolescents and youths. Early sexual debut, time balance between academics and dating issues, and unrealistic expectations from partners continue to cause endless cries among dating adolescents and youths. This study explored the effect of media illiteracy among adolescents and youths on three behaviors; learning engagement in higher education, setting dating or couple goals, and sexual attitudes and behavior. From a total of 1037 (66.2% male and 33.8% female) valid questionnaire responses, the results indicated that adolescents and youths are major victims of media misinformation, obstructing them from accessing the factual truth about intimacy and sexuality. Media illiteracy negatively affects decisions about sexual intimacy and related behaviors. Adolescents’ and youths’ sexual permissiveness is prejudiced by lack of skills, knowledge, and understanding of how to decipher credible information from social media platforms. Conclusively, media literacy is essential to access and iden- tify authentic information for different purposes, especially for decisions on sexuality and intimacy. Benefits are immense only if an accurate intellectual framework is cultivated to find, understand, evaluate and use information from various media platforms properly.
... We extend the approach for an entire sample (a set of publications). Our approach is illustrated with a reanalysis of publication data from Bornmann and Tekles (2021) and Bittmann, Tekles, and Bornmann (2022) regarding the two journals Physical Review Letters and Physical Review E. The data are especially interesting, since assessments by experts are available whether the publications in the sets can be denoted as landmark papers (that are disruptive in all likelihood) or not. ...
L. Wu, Wang, and Evans (2019) introduced the disruption index (DI) which has been designed to capture disruptiveness of individual publications based on dynamic citation networks of publications. In this study, we propose a statistical modelling approach to tackle open questions with the DI: (1) how to consider uncertainty in the calculation of DI values, (2) how to aggregate DI values for paper sets, (3) how to predict DI values using covariates, and (4) how to unambiguously classify papers into either disruptive or not disruptive. A Bayesian multilevel logistic approach is suggested that extends an approach of Figueiredo and Andrade (2019). A reanalysis of sample data from Bornmann and Tekles (2021) and Bittmann, Tekles, and Bornmann (2022) shows that the Bayesian approach is helpful in tackling the open questions. For example, the modelling approach is able to predict disruptive papers (milestone papers in physics) in a good way.
... To measure the impact of scientific papers and scientists, numerous citation-based indicators, such as the h-index (Hirsch, 2005;Schubert, 2009) and the g-index (Egghe, 2006), have been proposed to measure the impact of scientific papers and scientists over the years. However, traditional metrics have been criticized for their inconsistency (Brito & Navarro, 2021;Waltman & Van Eck, 2012), especially in measuring milestone research and top scientists' impact (Bornmann & Tekles, 2021;Herrmannova et al., 2018;Wang et al., 2023). These issues have led scientists to resist traditional research evaluation procedures through the San Francisco Declaration (Cagan, 2013) and the Leiden Manifesto (Hicks et al., 2015), and to call for more comprehensive metrics. ...
In this study, we introduce a novel metric called the Apex index, which is a hit-based measure for assessing the impact of scientific publications and scientists. The basic principle of the Apex index involves quantifying the number of hit papers that cite the focal paper. Specifically, we identify the top 1% most highly cited papers in a given field and year as the hit papers. We then calculate the Apex index for all publications in the MAG database, which contains approximately 200 million documents. Our study reveals that Nobel Prize-winning papers display a higher Apex index compared to other papers, and Nobel laureates exhibit a higher Apex index than their peers. Moreover, we demonstrate that the Apex index has a higher convergent validity in evaluating scientists and identifying laureates. Overall, the Apex index presents a valuable and effective tool for assessing the impact of scientific publications and researchers.
... To illustrate, a paper can be seen as consolidating the existing knowledge if it is frequently cited together with its references, while a paper can be seen as novel and disruptive if the forward citations to it do not acknowledge its intellectual forebears. This metric embodies the peer evaluation by differentiating two types of citation patterns and has been validated through self-and peer-reported novelty judgment (Bornmann, Devarakonda, et al., 2019;Bornmann & Tekles, 2021;Shibayama & Wang, 2020). ...
Novel ideas often experience resistance from incumbent forces. While evidence of the bias against novelty has been widely identified in science, there is still a lack of large‐scale quantitative work to study this problem occurring in the prepublication process of manuscripts. This paper examines the association between manuscript novelty and handling time of publication based on 778,345 articles in 1,159 journals indexed by PubMed. Measuring the novelty as the extent to which manuscripts disrupt existing knowledge, we found systematic evidence that higher novelty is associated with longer handling time. Matching and fixed‐effect models were adopted to confirm the statistical significance of this pattern. Moreover, submissions from prestigious authors and institutions have the advantage of shorter handling time, but this advantage is diminishing as manuscript novelty increases. In addition, we found longer handling time is negatively related to the impact of manuscripts, while the relationships between novelty and 3‐ and 5‐year citations are U‐shape. This study expands the existing knowledge of the novelty bias by examining its existence in the prepublication process of manuscripts.
... Bornmann and Tekles (2020) used the list of milestone papers published in journal Physical Review Letters (PRL) (39 papers from over 44,000 articles published between 1980 and 2002) to validate several research disruptiveness indicators. • Wuestman et al. (2020a) used the list of breakthroughs of the year of Science magazine ...
A long-standing research question in bibliometrics is how one identifies publications, which represent major advances in their fields, making high impact in there and other areas. In this context, the term "Breakthrough" is often used and commonly used approaches rely on citation links between publications implicitly positing that peers who use or build upon previously published results collectively inform about their standing in terms of advancing the research frontiers. Here we argue that the "Breakthrough" concept is rooted in the Kuhnian model of scientific revolution which has been both conceptually and empirically challenged. A more fruitful approach is to consider various ways in which authoritative actors in scholarly communication system signal the importance of research results. We bring to discussions different "recognition channels" and pilot the creation of an open dataset of editorial highlights from regular lists of notable research advances. The dataset covers the last ten years and includes: the "discoveries of the year" from Science magazine and La Recherche and weekly editorial highlights from Nature ("research highlights") and Science ("editor's choice"). The final dataset includes 230 entries in the "discoveries of the years" (with over 720 references) and about 9,000 weekly highlights (with over 8,000 references).
The relationship between transparency and credibility has long been a subject of theoretical and analytical exploration within the realm of social sciences, and it has recently attracted increasing attention in the context of scientific research. Retraction serves as a pivotal mechanism in addressing concerns about research integrity. This study aims to empirically examining the relationship between open access level and the effectiveness of current mechanism, specifically academic purification centered on retracted articles. In this study, we used matching and Difference-in-Difference (DiD) methods to examine whether gold open access is helpful for academic purification in biochemistry field. We collected gold open access (Gold OA) and non-open access (non-OA) biochemistry retracted articles as the treatment group, and matched them with corresponding unretracted articles as the control group from 2005 to 2021 based on Web of Science and Retraction Watch database. The results showed that compared to non-OA, Gold OA is advantageous in reducing the retraction time of flawed articles, but does not demonstrate a significant advantage in reducing citations after retraction. This indicates that Gold OA may help expedite the detection and retraction of flawed articles, ultimately promoting the practice of responsible research.
Encompassing an intricately profound propensity for revolutionary, paradigm-shifting ramifications and the potential to wield an irrefutably disruptive sway on forthcoming research endeavors, the notion of the Disruption Index (DI) has surfaced as an object of fervent scientific scrutiny within the realm of scientometrics. Nevertheless, its implementation faces multifaceted constraints. Through a meticulous inquiry, we methodically dissect the limitations of DI, encompassing: (a) susceptibility to variations in reference numbers, (b) vulnerability to intentional author manipulations, (c) heterogeneous manifestations across diverse subject fields, (d) disparities across publication years, (e) misalignment with established scientific impact measures, (f) inadequacy in convergent validity with expert-selected milestones , and (g) a prevalent concentration around zero in its distribution. Unveiling the root causes of these challenges, we propose a viable solution encapsulated in the Rescaled Disruption Index (RDI), achieved through comprehensive rescaling across fields, years, and references. Our empirical investigations unequivocally demonstrate the efficacy of RDI, unveiling the universal nature of disruption distributions in science. This introduces a robust and refined framework for assessing disruptive potential in the scientific landscape while preserving the core principles of the index.
The purpose of this paper is to provide a review of the literature on the original disruption index (DI1) and its variants in scientometrics. The DI1 has received much media attention and prompted a public debate about science policy implications, since a study published in Nature found that papers in all disciplines and patents are becoming less disruptive over time. This review explains in the first part the DI1 and its variants in detail by examining their technical and theoretical properties. The remaining parts of the review are devoted to studies that examine the validity and the limitations of the indices. Particular focus is placed on (1) possible biases that affect disruption indices (2) the convergent and predictive validity of disruption scores, and (3) the comparative performance of the DI1 and its variants. The review shows that, while the literature on convergent validity is not entirely conclusive, it is clear that some modified index variants, in particular DI5 , show higher degrees of convergent validity than DI1. The literature draws attention to the fact that (some) disruption indices suffer from inconsistency, time-sensitive biases, and several data-induced biases. The limitations of disruption indices are highlighted and best practice guidelines are provided. The review encourages users of the index to inform about the variety of DI1 variants and to apply the most appropriate variant. More research on the validity of disruption scores as well as a more precise understanding of disruption as a theoretical construct is needed before the indices can be used in the research evaluation practice.
Scientific breakthroughs have the potential to revolutionize the course of research and shape the trajectory of scientific knowledge. This study investigates the characteristics of Disruptive Citing Papers (DCP) and Consolidating Citing Papers (CCP) associated with Nobel‐winning scientific breakthroughs, aiming to provide insights into the mechanisms of knowledge creation and dissemination. By analyzing a dataset of Nobel‐winning papers and their citation networks, we find that Nobel‐winning papers tend to attract a higher proportion of DCP compared to CCP. However, CCP exhibit a higher impact, as evidenced by their citation counts and likelihood of becoming hit papers. Furthermore, DCP are associated with larger research teams, highlighting the collaborative nature of disruptive research, while CCP employ a higher degree of professional language style characterized by shorter titles and specialized jargon. These findings deepen our understanding of the role played by disruptive and consolidating impact in scientific breakthroughs, shedding light on the dynamics of knowledge creation and dissemination in the scientific community. This research contributes to the broader understanding of scientific progress and provides valuable insights for researchers, policymakers, and stakeholders in the scientific ecosystem.
This study proposes a novel approach for evaluating the impact of scientists by introducing a new set of metrics and a dual measurement framework that combines the concepts of disruption and consolidation. Traditional metrics like total citation and h-index are limited in their ability to capture the full range of a scientist's influence, and therefore the Scientists' Disruptive Citation (SDC), Disruptive h-index (D h-index), and consolidating metrics are introduced to provide a more comprehensive evaluation of scientists' disruptive and consolidating influence. Using a dataset of 463,348 papers, 234,086 disambiguated scientists, and data on three important awards, including Nobel Prize, Wolf Prize, and Dirac Medal, in the field of Physics, this study demonstrates that the SDC and D h-index are superior to all benchmark metrics, including the conventional and normalized disruption-based measures, in terms of convergent validity. Second, this study analyzes the distribution of academic characteristics between award-winning and non-laureates, explores various metrics of scientists with high SDC and Scientists' Consolidating Citation (SCC), and finds that disruptive impact can identify successful scientists from their counterparts and serve as an early signal of successful scientists. Third, this study reveals that the disruptive citation proposed in this study is less susceptible to manipulation, making it a more reliable metric for assessing a scientist's or a single paper's disruptive impact than the CD-index. The results suggest that the SDC and D h-index are reliable metrics for measuring scientists' innovative influence and can aid in the development of future scientific research. Overall, this study provides a scientifically sound and effective new perspective on measuring scientists using a dual measurement of disruptive and consolidating influence.
For many years, the journal evaluation system has been centered on impact indicators, resulting in evaluation results that do not reflect the academic innovation of journals. To solve this issue, this study attempts to construct the Journal Disruption Index (JDI) from the perspective of measuring the disruption of each journal article. In the actual study, we measured the disruption of articles of 22 selected virology journals based on the OpenCitations Index of Crossref open DOI-to-DOI citations (COCI) first. Then we calculated the JDI of 22 virology journals based on the absolute disruption index (\({D}_{Z}\)) of the articles. Finally, we conducted an empirical study on the differences and correlations between the impact indicators and disruption indicators as well as the evaluation effect of the disruption index. The results of the study show: (1) There are large differences in the ranking of journals based on disruption indicators and impact indicators. Among the 22 journals, 12 are ranked higher by JDI than Cumulative Impact Factor for 5 years (CIF5), the Journal Index for PR6 (JIPR6) and average Percentile in Subject Area (aPSA). The ranking difference of 17 journals between the two kinds of indicators is greater than or equal to 5. (2) There is a medium correlation between disruption indicators and impact indicators at the level of journals and papers. JDI is moderately correlated with CIF5, JIPR6 and aPSA, with correlation coefficients of 0.486, 0.471 and − 0.448, respectively. \({D}_{Z}\) was also moderately correlated with Cumulative Citation (CC), Percentile Ranking with 6 Classifications (PR6) and Percentile in Subject Area (PSA) with correlation coefficients of 0.593, 0.575 and − 0.593, respectively. (3) Compared with traditional impact indicators, the results of journal disruption evaluation are more consistent with the evaluation results of experts’ peer review. JDI reflects the innovation level of journals to a certain extent, which is helpful to promote the evaluation of innovation in sci-tech journals.
As teams become prevalent in contemporary science, how to establish collaborations is key to tomorrow’s breakthrough, which has broad implications to individual scientists, institutions, and funding agencies. In this paper, we focus on the association between collaboration networks and scientists producing novel and disruptive research, based on the publication data from the American Physical Society. In particular, we focus on the role of spanning structural holes on producing novel and disruptive research. Our primary finding is that scientists whose collaboration networks span over structural holes in their collaboration networks not only produce more novel and disruptive research, but also have higher chance to produce novel and disruptive research. Although both male and female scientists benefit from structural holes, we find suggestive evidence that female researchers benefit more. This paper provides empirical evidence on the relationship between structural holes and novel/disruptive research in the field of physics, which has policy implications for nurturing scientists and developing science policies.
Research on the evaluation of the quality of academic papers is attracting more attention from scholars in scientometrics. However, most previous researches have assessed paper quality based on external indicators, such as citations, which failed to account for the content of the research. To that end, this paper proposed a new method for measuring a paper's originality. The method was based on knowledge units in semantic networks, focusing on the relationship and semantic similarity of different knowledge units. Connectivity and path similarity between different content elements were used in particular networks as indicators of originality. This study used papers published between 2014 and 2018 in three categories (i.e. Library & Information Science, Educational Psychology, and Carbon Nanotubes) and divided their content into three parts (i.e. research topics, research methods and research results). It was found that the originality in all categories increase each year. Furthermore, a comparison of our new method with previous models of citation network analysis and knowledge combination analysis showed that our new method is better than those previous methods when used in measuring originality.
Recent works aimed to understand how to identify “milestone” scientific papers of great significance from large-scale citation networks. To this end, previous results found that global ranking metrics that take into account the whole network structure (such as Google’s PageRank) outperform local metrics such as the citation count. Here, we show that by leveraging the recursive equation that defines the PageRank algorithm, we can propose a family of local impact metrics. Our results reveal that the obtained PageRank-based local metrics outperform the citation count and other local metrics in identifying the seminal papers. Compared with global metrics, these local metrics can reach similar performance in the identification of seminal papers in shorter computational time, without requiring the whole network topology. Our findings could help to better understand the nature of groundbreaking research from citation network analysis and find practical applications in large-scale data.
Controlling for confounding factors is one of the central aspects of quantitative research. While methods like linear regression models are common, their results can be misleading under certain conditions. We demonstrate how statistical matching can be utilized as an alternative that enables the inspection of post-matching balancing. This contribution serves as an empirical demonstration of matching in bibliometrics and discusses advantages and potential pitfalls. We propose matching as an easy-to-use approach in bibliometrics to estimate effects and remove bias. To exemplify matching, we use data about papers published in Physical Review E and a selection classified as milestone papers. We analyze whether milestone papers score higher in terms of a proposed class of indicators for measuring disruptiveness than non-milestone papers. We consider disruption indicators DI1, DI5, DI1n, DI5n and DEP and test which of the disruption indicators performs best, based on the assumption that milestone papers should have higher disruption indicator values than non-milestone papers. Four matching algorithms (propensity score matching (PSM), coarsened exact matching (CEM), entropy balancing (EB) and inverse probability weighting (IPTW)) are compared. We find that CEM and EB perform best regarding covariate balancing and DI5 and DEP are well-performing to evaluate disruptiveness of published papers.
Peer Review
https://publons.com/publon/10.1162/qss_a_00158
Science and technology develop not only along historical trajectories, but also as next-order regimes that periodically change the landscape. Regimes can incur on trajectories which are then disrupted. Using citations and references for the operationalization, we discuss and quantify both the recently proposed “disruption indicator” and the older indicator for “critical transitions” among reference lists as changes which may necessitate a rewriting of history. We elaborate this with three examples in order to provide a proof of concept. We shall show how the indicators can be calculated using Web-of-Science data. The routine is automated (available at < http://www.leydesdorff.net/software/di/index.htm >) so that it can be upscaled in future research. We suggest that “critical transitions” can be used to indicate disruption at the regime level, whereas disruption is developed at the trajectory level. Both conceptually and empirically, however, continuity is grasped more easily than disruption.
The citation impact of a scientific publication is usually seen as a one-dimensional concept. We introduce a multi-dimensional framework for characterizing the citation impact of a publication. In addition to the level of citation impact, quantified by the number of citations received by a publication, we also conceptualize and operationalize the depth and breadth and the dependence and independence of the citation impact of a publication. The proposed framework distinguishes between publications that have a deep citation impact, typically in a relatively narrow research area, and publications that have a broad citation impact, probably covering a wider area of research. It also makes a distinction between publications that are strongly dependent on earlier work and publications that make a more independent scientific contribution. We use our multi-dimensional citation impact framework to report basic descriptive statistics on the citation impact of highly cited publications in all scientific disciplines. In addition, we present a detailed case study focusing on the field of scientometrics. The proposed citation impact framework provides a more in-depth understanding of the citation impact of a publication than a traditional one-dimensional perspective.
Peer Review
https://publons.com/publon/10.1162/qss_a_00109
In the context of recent developments in scientometrics to measure novelty or creative potential, Wu, Wang, and Evans (2019) propose a new disruption index that measures the extent to which a publication disrupts the field of science. We calculated the disruption index for some example papers. The analyses of the index values (using our Web of Science in-house database) show that they depend on the citation window (the period of time over which citations are collected).
Recently, Wu, Wang, and Evans (2019) proposed a new family of indicators, which measure whether a scientific publication is disruptive to a field or tradition of research. Such disruptive influences are characterized by citations to a focal paper, but not its cited references. In this study, we are interested in the question of convergent validity. We used external criteria of newness to examine convergent validity: in the post-publication peer review system of F1000Prime, experts assess papers whether the reported research fulfills these criteria (e.g., reports new findings). This study is based on 120,179 papers from F1000Prime published between 2000 and 2016. In the first part of the study we discuss the indicators. Based on the insights from the discussion, we propose alternate variants of disruption indicators. In the second part, we investigate the convergent validity of the indicators and the (possibly) improved variants. Although the results of a factor analysis show that the different variants measure similar dimensions, the results of regression analyses reveal that one variant (????????5) performs slightly better than the others.
Wu et al. (Nature 566:378–382, 2019) introduced a new indicator measuring disruption (\({DI}_{1}\)). Bornmann et al. (Do disruption index indicators measure what they propose to measure? The comparison of several indicator variants with assessments by peers, 2019. https://arxiv.org/abs/1911.08775) compared variants of the disruption index and pointed to \({DI}_{5}\) as an interesting variant. The calculation of a field-specific version of \({DI}_{5}\) (focusing on disruptiveness within the same field) for Scientometrics papers in the current study reveals that the variant is possibly able to identify landmark papers in scientometrics. This result is in contrast to the Scientometrics analysis previously published by Bornmann and Tekles (Scientometrics 120(1):331–336, 2019) based on the original disruption index (\({DI}_{1}\)).
Originality has a self-evident importance for science, but objectively measuring originality poses a formidable challenge. We conceptualise originality as the degree to which a scientific discovery provides subsequent studies with unique knowledge that is not available from previous studies. Accordingly, we operationalise a new measure of originality for individual scientific papers building on the network betweenness centrality concept. Specifically, we measure the originality of a paper based on the directed citation network between its references and the subsequent papers citing it. We demonstrate the validity of this measure using survey information. In particular, we find that the proposed measure is positively correlated with the self-assessed theoretical originality but not with the methodological originality. We also find that originality can be reliably measured with only a small number of subsequent citing papers, which lowers computational cost and contributes to practical utility. The measure also predicts future citations, further confirming its validity. We further characterise the measure to guide its future use.
Peer review of candidates' proposals for research position is generally used as the best method available to select the most
promising researchers, but it is very costly and has its limitations. This article analyzes to what extent bibliometric indicators
can predict the results of the peer review exercise using the example of a particular selection process. Two composite indicators
are found to be strongly correlated with peer review-based decisions. We calculated that the probability of the estimated
prediction, as determined by the composite indicators, for a selected applicant to be higher than the estimated prediction
determined for a rejected applicant is about 75%.
Research articles produced through international collaboration are more highly cited than other work, but are they also more novel? Using measures developed by Uzzi et al. (2013), and replicated by Boyack and Klavans (2014), this article tests for novelty and conventionality in international research collaboration. Scholars have found that coauthored articles are more novel and have suggested that diverse groups have a greater chance of producing creative work. As such, we expected to find that international collaboration tends to produce more novel research. Using data from Web of Science and Scopus in 2005, we failed to show that international collaboration tends to produce more novel articles. In fact, international collaboration appears to produce less novel and more conventional knowledge combinations. Transaction costs and communication barriers to international collaboration may suppress novelty. Higher citations to international work may be explained by an audience effect, where more authors from more countries results in greater access to a larger citing community. The findings are consistent with explanations of growth in international collaboration that posit a social dynamic of preferential attachment based upon reputation.
One of the most universal trends in science and technology today is the growth of large teams in all areas, as solitary researchers and small teams diminish in prevalence1–3. Increases in team size have been attributed to the specialization of scientific activities³, improvements in communication technology4,5, or the complexity of modern problems that require interdisciplinary solutions6–8. This shift in team size raises the question of whether and how the character of the science and technology produced by large teams differs from that of small teams. Here we analyse more than 65 million papers, patents and software products that span the period 1954–2014, and demonstrate that across this period smaller teams have tended to disrupt science and technology with new ideas and opportunities, whereas larger teams have tended to develop existing ones. Work from larger teams builds on more-recent and popular developments, and attention to their work comes immediately. By contrast, contributions by smaller teams search more deeply into the past, are viewed as disruptive to science and technology and succeed further into the future—if at all. Observed differences between small and large teams are magnified for higher-impact work, with small teams known for disruptive work and large teams for developing work. Differences in topic and research design account for a small part of the relationship between team size and disruption; most of the effect occurs at the level of the individual, as people move between smaller and larger teams. These results demonstrate that both small and large teams are essential to a flourishing ecology of science and technology, and suggest that, to achieve this, science policies should aim to support a diversity of team sizes.
In the current parlance of evidence-based policy, indicators are increasingly called upon to inform policymakers, including in the research and innovation domain. However, few studies have scrutinized how such indicators come about in practice. We take as an example the development of an indicator by the European Commission, the Research Excellence in Science & Technology indicator. First, we outline tensions related to defining and measuring research excellence for policy using the notion of 'essentially contested concept'. Second, we explore the construction and use of the aforementioned indicator through in-depth interviews with relevant actors and the coproduction of indicators, that is the interplay of their making vis-a-vis academic practices and policy expectations. We find that although many respondents in our study feel uncomfortable with the current usage of notions of excellence as indicator of quality of research practices, few alternatives are suggested. We identify a number of challenges which may contribute to the debate of indicator development, suggesting that the making of current indicators for research policy in the EU may be in need of serious review.
Citations are increasingly used as performance indicators in research policy and within the research system. Usually, citations are assumed to reflect the impact of the research or its quality. What is the justification for these assumptions and how do citations relate to research quality? These and similar issues have been addressed through several decades of scientometric research. This article provides an overview of some of the main issues at stake, including theories of citation and the interpretation and validity of citations as performance measures. Research quality is a multidimensional concept, where plausibility/soundness, originality, scientific value, and societal value commonly are perceived as key characteristics. The
article investigates how citations may relate to these various research quality dimensions. It is argued that citations reflect aspects related to scientific impact and relevance, although with important limitations. On the contrary, there is no evidence that citations reflect other key dimensions of research quality. Hence, an increased use of citation indicators in research evaluation and funding may imply less attention to these other research quality dimensions, such as solidity/plausibility, originality, and societal value.
Several authors have proposed that a large number of unusual combinations of cited references in a paper point to its high creative potential (or novelty). However, it is still not clear whether the number of unusual combinations can really measure the creative potential of papers. The current study addresses this question on the basis of several case studies from the field of scientometrics. We identified some landmark papers in this field. Study subjects were the corresponding authors of these papers. We asked them where the ideas for the papers came from and which role the cited publications played. The results revealed that the creative ideas might not necessarily have been inspired by past publications. The literature seems to be important for the contextualization of the idea in the field of scientometrics. Instead, we found that creative ideas are the result of finding solutions to practical problems, result from discussions with colleagues, and profit from interdisciplinary exchange. The roots of the studied landmark papers are discussed in detail.
The use of bibliometric measures in the evaluation of research has increased considerably based on expertise from the growing research field of evaluative citation analysis (ECA). However, mounting criticism of such metrics suggests that the professionalization of bibliometric expertise remains contested. This paper investigates why impact metrics, such as the journal impact factor and the h-index, proliferate even though their legitimacy as a means of professional research assessment is questioned. Our analysis is informed by two relevant sociological theories: Andrew Abbott’s theory of professions and Richard Whitley’s theory of scientific work. These complementary concepts are connected in order to demonstrate that ECA has failed so far to provide scientific authority for professional research assessment. This argument is based on an empirical investigation of the extent of reputational control in the relevant research area. Using three measures of reputational control that are computed from longitudinal inter-organizational networks in ECA (1972–2016), we show that peripheral and isolated actors contribute the same number of novel bibliometric indicators as central actors. In addition, the share of newcomers to the academic sector has remained high. These findings demonstrate that recent methodological debates in ECA have not been accompanied by the formation of an intellectual field in the sociological sense of a reputational organization. Therefore, we conclude that a growing gap exists between an academic sector with little capacity for collective action and increasing demand for routine performance assessment by research organizations and funding agencies. This gap has been filled by database providers. By selecting and distributing research metrics, these commercial providers have gained a powerful role in defining de-facto standards of research excellence without being challenged by expert authority.
Research produced through international collaboration is often more highly cited than other work, but is it also more novel? Using measures of conventionality and novelty developed by Uzzi et al. (2013) and replicated by Boyack and Klavans (2013), we test for novelty and conventionality in international research collaboration. Many studies have shown that international collaboration is more highly cited than national or sole-authored papers. Others have found that coauthored papers are more novel. Scholars have suggested that diverse groups have a greater chance of producing creative work. As such, we expected to find that international collaboration is also more novel. Using data from Web of Science and Scopus in 2005, we failed to show that international collaboration tends to produce more novel articles. In fact, international collaboration appeared to produce less novel and more conventional research. Transaction costs and the limits of global communication may be suppressing novelty, while an audience effect may be responsible for higher citation rates. Closer examination across the sciences, social sciences, and arts and humanities, as well as examination of six scientific specialties further illuminates the interplay of conventionality and novelty in work produced by international research teams.
The whys and wherefores of SciSci
The science of science (SciSci) is based on a transdisciplinary approach that uses large data sets to study the mechanisms underlying the doing of science—from the choice of a research problem to career trajectories and progress within a field. In a Review, Fortunato et al. explain that the underlying rationale is that with a deeper understanding of the precursors of impactful science, it will be possible to develop systems and policies that improve each scientist's ability to succeed and enhance the prospects of science as a whole.
Science , this issue p. eaao0185
In this large-scale contribution, we deal with the relationship between properties of cited references of Web of Science articles and the field normalized citation rate of these articles. Using nearly 1 million articles, and three classification systems with different levels of granularity, we study the effects of number of cited references, share of references covered by Web of Science, mean age of references and mean citation rate of references on field normalized citation rate. To expose the relationship between the predictor variables and the response variable, we use quantile regression. We found that a higher number of references, a higher share of references to publications within Web of Science and references to more recent publications correlate with citation impact. A correlation was observed even when normalization was done with a finely grained classification system. The predictor variables affected citation impact to a larger extent at higher quantile levels. Regarding the relative importance of the predictor variables, citation impact of the cited references was in general the least important variable. Number of cited references carried most of the importance for both low and medium quantile levels, but this importance was lessened at the highest considered level.
This study provides an overview of the literature dealing with the process of citing publications (focusing on the literature from the recent decade). It presents theories, which have been proposed for explaining the citation process, and studies having empirically analyzed this process. The overview is structured based on three core elements in the citation process: the context of the cited document, processes from selection to citation of documents, and the context of the citing document. The overview can be used to understand the process of citing and delivers basic information for the proper application of citations in research evaluation.
The Independent Review of the Role of Metrics in Research Assessment and Management was set up in April 2014 to investigate the current and potential future roles that quantitative indicators can play in the assessment and management of research. Its report, ‘The Metric Tide’, was published in July 2015 and is available below.
The review was chaired by James Wilsdon, professor of science and democracy at the University of Sussex, supported by an independent and multidisciplinary group of experts in scientometrics, research funding, research policy, publishing, university management and research administration. Through 15 months of consultation and evidence-gathering, the review looked in detail at the potential uses and limitations of research metrics and indicators, exploring the use of metrics within institutions and across disciplines.
The main findings of the review include the following:
There is considerable scepticism among researchers, universities, representative bodies and learned societies about the broader use of metrics in research assessment and management.
Peer review, despite its flaws, continues to command widespread support as the primary basis for evaluating research outputs, proposals and individuals. However, a significant minority are enthusiastic about greater use of metrics, provided appropriate care is taken.
Carefully selected indicators can complement decision-making, but a ‘variable geometry’ of expert judgement, quantitative indicators and qualitative measures that respect research diversity will be required.
There is legitimate concern that some indicators can be misused or ‘gamed’: journal impact factors, university rankings and citation counts being three prominent examples.
The data infrastructure that underpins the use of metrics and information about research remains fragmented, with insufficient interoperability between systems.
Analysis concluded that that no metric can currently provide a like-for-like replacement for REF peer review.
In assessing research outputs in the REF, it is not currently feasible to assess research outputs or impacts in the REF using quantitative indicators alone.
In assessing impact in the REF, it is not currently feasible to use quantitative indicators in place of narrative case studies. However, there is scope to enhance the use of data in assessing research environments.
The review identified 20 recommendations for further work and action by stakeholders across the UK research system. They propose action in the following areas: supporting the effective leadership, governance and management of research cultures; improving the data infrastructure that supports research information management; increasing the usefulness of existing data and information sources; using metrics in the next REF; and coordinating activity and building evidence.
These recommendations are underpinned by the notion of ‘responsible metrics’ as a way of framing appropriate uses of quantitative indicators in the governance, management and assessment of research. Responsible metrics can be understood in terms of the following dimensions:
Robustness: basing metrics on the best possible data in terms of accuracy and scope
Humility: recognising that quantitative evaluation should support – but not supplant – qualitative, expert assessment
Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results
Diversity: accounting for variation by field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system
Reflexivity: recognising and anticipating the systemic and potential effects of indicators, and updating them in response.
text from: http://www.hefce.ac.uk/pubs/rereports/year/2015/metrictide/
Bibliometric indicators such as journal impact factors, h-indices, and total citation counts are algorithmic artifacts that can be used in research evaluation and management. These artifacts have no meaning by themselves, but receive their meaning from attributions in institutional practices. We distinguish four main stakeholders in these practices: (1) producers of bibliometric data and indicators; (2) bibliometricians who develop and test indicators; (3) research managers who apply the indicators; and (4) the scientists being evaluated with potentially competing career interests. These different positions may lead to different and sometimes conflicting perspectives on the meaning and value of the indicators. The indicators can thus be considered as boundary objects which are socially constructed in translations among these perspectives. This paper proposes an analytical clarification by listing an informed set of (sometimes unsolved) problems in bibliometrics which can also shed light on the tension between simple but invalid indicators that are widely used (e.g., the h-index) and more sophisticated indicators that are not used or cannot be used in evaluation practices because they are not transparent for users, cannot be calculated, or are difficult to interpret.
Citations between scientific papers and related bibliometric indices, such as the $h$-index for authors and the impact factor for journals, are being increasingly used -- often in controversial ways -- as quantitative tools for research evaluation. Yet, a fundamental research question remains still open: to which extent do quantitative metrics capture the significance of scientific works? We analyze the network of citations among the $449,935$ papers published by the American Physical Society (APS) journals between $1893$ and $2009$, and focus on the comparison of metrics built on the citation count with network-based metrics. We contrast five article-level metrics with respect to the rankings that they assign to a set of fundamental papers, called Milestone Letters, carefully selected by the APS editors for "making long-lived contributions to physics, either by announcing significant discoveries, or by initiating new areas of research". A new metric, which combines PageRank centrality with the explicit requirement that paper score is not biased by paper age, outperforms the others in identifying the Milestone Letters short after they are published. The lack of time bias in the new metric makes it also possible to use it to compare papers of different age on the same scale. We find that network-based metrics generally identify the Milestone Letters better than metrics based on the citation count, which suggests that the structure of the citation network contains information that can be used to improve the ranking of scientific publications. The methods and results presented here are relevant for all evolving systems where network centrality metrics are applied, for example the World Wide Web and online social networks.
On rare occasions in the history of science, remarkable discoveries transform human society and forever alter mankind’s view of the world. Examples of such discoveries include the heliocentric theory, Newtonian physics, the germ theory of disease, quantum theory, plate tectonics and the discovery that DNA carries genetic information. The science philosopher Thomas Kuhn famously described science as long periods of normality punctuated by times of crisis, when anomalous observations culminate in revolutionary changes that replace one paradigm with another. This essay examines several transformative discoveries in the light of Kuhn’s formulation. We find that each scientific revolution is unique, with disparate origins that may include puzzle solving, serendipity, inspiration, or a convergence of disparate observations. The causes of revolutionary science are varied and lack an obvious common structure. Moreover, it can be difficult to draw a clear distinction between so-called normal and revolutionary science. Revolutionary discoveries often emerge from basic science and are critically dependent on nonrevolutionary research. Revolutionary discoveries may be conceptual or technological in nature, lead to the creation of new fields, and have a lasting impact on many fields in addition to the field from which they emerge. In contrast to political revolutions, scientific revolutions do not necessarily require the destruction of the previous order. For humanity to continue to benefit from revolutionary discoveries, a broad palette of scientific inquiry with a particular emphasis on basic science should be supported.
In their Report “The increasing dominance of teams in production of knowledge” (18 May, p. [1036][1]), S. Wuchty et al. observe that references with multiple authors receive more citations than solo-authored ones. They conclude that research led by teams has more quality than solo-led research,
Peer evaluation is the cornerstone of science evaluation. In this paper, we analyze whether or not a form of peer evaluation, the pre-publication selection of the best papers in Computer Science (CS) conferences, is better than random, when considering future citations received by the papers.
Considering 12 conferences (for several years), we collected the citation counts from Scopus for both the best papers and the non-best papers. For a different set of 17 conferences, we collected the data from Google Scholar. For each data set, we computed the proportion of cases whereby the best paper has more citations. We also compare this proportion for years before 2010 and after to evaluate if there is a propaganda effect. Finally, we count the proportion of best papers that are in the top 10% and 20% most cited for each conference instance.
The probability that a best paper will receive more citations than a non best paper is 0.72 (95% CI = 0.66, 0.77) for the Scopus data, and 0.78 (95% CI = 0.74, 0.81) for the Scholar data. There are no significant changes in the probabilities for different years. Also, 51% of the best papers are among the top 10% most cited papers in each conference/year, and 64% of them are among the top 20% most cited.
There is strong evidence that the selection of best papers in Computer Science conferences is better than a random selection, and that a significant number of the best papers are among the top cited papers in the conference.
We study the correlation between citation-based and expert-based assessments of journals and series, which we collectively refer to as sources. The source normalized impact per paper (SNIP), the Scimago Journal Rank 2 (SJR2) and the raw impact per paper (RIP) indicators are used to assess sources based on their citations, while the Norwegian model is used to obtain expert-based source assessments. We first analyze – within different subject area categories and across such categories – the degree to which RIP, SNIP and SJR2 values correlate with the quality levels in the Norwegian model. We find that sources at higher quality levels on average have substantially higher RIP, SNIP, and SJR2 values. Regarding subject area categories, SNIP seems to perform substantially better than SJR2 from the field normalization point of view. We then compare the ability of RIP, SNIP and SJR2 to predict whether a source is classified at the highest quality level in the Norwegian model or not. SNIP and SJR2 turn out to give more accurate predictions than RIP, which provides evidence that normalizing for differences in citation practices between scientific fields indeed improves the accuracy of citation indicators.
The percentages of shares of world publications of the European Union and its
member states, China, and the United States have been represented differently
as a result of using different databases. An analytical variant of the
Web-of-Science (of Thomson Reuters) enables us to study the dynamics in the
world publication system in terms of the field-normalized top-1% and top-10%
most-frequently-cited publications. Comparing the EU28, USA, and China at the
global level shows a top-level dynamics that is different from the analysis in
terms of shares of publications: the United States remains far more productive
in the top-1% of all papers; China drops out of the competition for elite
status; and the EU28 increased its share among the top-cited papers from
2000-2010. Some of the EU28 member states overtook the U.S. during this decade,
but a clear divide remains between EU15 (Western Europe) and the Accession
Countries. Network analysis shows that internationally co-authored top-1%
publications perform far above expectation and also above top-10% ones. In
2005, China was embedded in this top-layer of internationally co-authored
publications. These publications often involve more than a single European
nation.
Thomas Kuhn's The Structure of Scientific Revolutions offers an insightful and engaging theory of science that speaks to scholars across many disciplines. Though initially widely misunderstood, it had a profound impact on the way intellectuals and educated laypeople thought about science. K. Brad Wray traces the influences on Kuhn as he wrote Structure, including his 'Aristotle epiphany', his interactions, and his studies of the history of chemistry. Wray then considers the impact of Structure on the social sciences, on the history of science, and on the philosophy of science, where the problem of theory change has set the terms of contemporary realism/anti-realism debates. He examines Kuhn's frustrations with the Strong Programme sociologists' appropriations of his views, and debunks several popular claims about what influenced Kuhn as he wrote Structure. His book is a rich and comprehensive assessment of one of the most influential works in the modern sciences.
What are the landmark papers in scientific disciplines? Which papers are indispensable for scientific progress? These are typical questions which are of interest not only for researchers (who frequently know the answers – or guess to know them) but also for the interested general public. Citation counts can be used to identify very useful papers since they reflect the wisdom of the crowd – in this case, the scientists using published results for their research. In this study, we identified with recently developed methods for the program CRExplorer landmark publications in nearly all Web of Science subject categories (WoS-SCs). These are publications which belong more frequently than other publications during the citing years to the top-1‰ in their subject area. As examples, we show the results of five subject categories: ‘Information Science & Library Science’, ‘Computer Science, Information Systems’, ‘Computer Science, Software Engineering’, ‘Psychology, Social’ and, ‘Chemistry, Physical’. The results of the other WoS-SCs can be found online at http://crexplorer.net . An analyst of the results should keep in mind that the identification of landmark papers depends on the used methods and data. Small differences in methods and/or data may lead to other results.
Lee et al. (2015) – based on Uzzi et al. (2013) – and Wang et al. (2017) proposed scores based on cited references (cited journals) data which can be used to measure the novelty of papers (named as novelty scores U and W in this study). Although previous research has used novelty scores in various empirical analyses, no study has been published up to now – to the best of our knowledge – which quantitatively tested the convergent validity of novelty scores: do these scores measure what they propose to measure? Using novelty assessments by faculty members (FMs) at F1000Prime for comparison, we tested the convergent validity of the two novelty scores (U and W). FMs’ assessments do not only refer to the quality of biomedical papers, but also to their characteristics (by assigning certain tags to the papers): for example, are the presented findings or formulated hypotheses novel (tags “new findings” and “hypothesis”)? We used these and other tags to investigate the convergent validity of both novelty scores. Our study reveals different results for the novelty scores: the results for novelty score U are mostly in agreement with previously formulated expectations. We found, for instance, that for a standard deviation (one unit) increase in novelty score U, the expected number of assignments of the “new finding” tag increase by 7.47%. The results for novelty score W, however, do not reflect convergent validity with the FMs’ assessments: only the results for some tags are in agreement with the expectations. Thus, we propose – based on our results – the use of novelty score U for measuring novelty quantitatively, but question the use of novelty score W.
The application of a new citation metric prompts a reassessment of the relationship between the size of scientific teams and research impact, and calls into question the trend to emphasize ‘big team’ science.
The central challenge in bibliometrics is finding the best ways to represent complex constructs like 'quality,' 'impact' or 'excellence' using quantitative methods. The marketplace for bibliometric data and services has evolved rapidly and users now face quite unprecedented choice when it comes to the range of data now available: from traditional citation-based indicators to reader ratings and Wikipedia mentions. Choice and ease of access have democratised bibliometrics and this is a tool now available to everyone. The era of 'desktop bibliometrics' should be welcomed: it promises greater transparency and the opportunity for experimentation in a field that has frankly become a little jaded. The downside is that we are in danger of chasing numbers for numbers' sake, with little understanding of what they mean. There is a looming crisis in construct validity, fuelled by supply side choice and user-side impatience, and this has significant implications for all stakeholders in the research evaluation space.
In many countries the scientific funding system is shifting from an internal block funding model toward a competitive project funding model. However, there is growing concern that the competitive project funding system favors relatively safe, conventional projects at the expense of risky, novel research. It is important to assess different funding models in order to design better funding systems for science. This paper empirically tests for differences in the novelty of funded outputs between internal block funding and competitive project funding, in the setting of Japan, where both funding models play a significant role. Combining survey data from a large sample of research projects in Japan and bibliometric information about the publications produced from these projects, we find that projects funded by competitive funds on average have higher novelty compared to those funded by internal block funds. However, such positive effects only hold for researchers with high status, such as senior and male researchers. In contrast, compared to internal block funding, competitive project funding has a negative relation to novelty for low status scientists (especially junior and female researchers). The findings suggest that the competitive project selection procedure is less receptive to novel ideas from researchers with low academic status and therefore discourages their novel research. These findings can serve as a warning about potential biases in competitive funding allocation procedures and suggest the importance of secure stable funding for allowing researchers with low status to pursue their novel ideas.
Eugene Garfield, always insisted that citation analysis “can be used wisely or abused” and that it is “up to the scientific community to prevent abuse of the SCI by devoting the necessary attention to its proper and judicious exploitation” (Garfield in Nat 227:669–671, 1970). Dedicated to his memory, this paper aim to assess the significance of a parameter that is seldom taken into account in evaluation studies: the existence of a USA comparative citation (visibility) advantage built in the database and thus affecting countries that collaborate more with the USA than with other countries. We analyze how this USA citation advantage affects the measure of the scientific impact (usually measured through citations received) of major countries. The main conclusion coming out of this study is that, given the strong presence of the USA in the WoS database, the comparative rankings tend, by construction, to give a citation advantage to countries having the closest relation to that country.
The objective of the study behind this report was to compare the consistency of indicators of ‘interdisciplinarity’ and to identify a preferred methodology. The outcomes reveal that choice of data, methodology and indicators can produce seriously inconsistent results despite a common set of disciplines and countries.
This raises questions about how interdisciplinarity is identified and assessed. It reveals a disconnect between the research metadata analysts typically use and the research activity they assume they have analysed. The results highlight issues around the responsible use of ‘metrics’ and the importance of analysts clarifying the link between any quantitative proxy indicator and the assumed policy target.
Citation Delay (D) introduced by Wang et al. (2015) is a measure of citation durability of articles reflecting information on the entire citation life-time. The characteristics of the measure and relationships of it to other article characteristics are examined in the six different fields using the citation data over 15 years of the articles published in 2000 in these fields. D distributes normally with good approximation and is not so much dependent on the subject field as the citation count. Although articles with higher D (more lately cited) tend to gain more citations in their life-time, this relationship is not linear but the mean of citations reaches a maximum at a certain value of D. Multiple regression analysis explaining D showed that articles with a higher Price index (i.e. citing more recent references) will receive most of the citations relatively earlier and that there is a weak tendency that articles containing more figures are cited earlier and those containing more tables are cited later. A seemingly contradictory result is found that more highly cited articles tend to have higher citation durability in individual journals while high-impact journals tend to include more articles with lower citation durability in higher proportions.
We discuss a method for improving causal inferences called ‘‘Coarsened Exact Matching’’ (CEM), and the new ‘‘Monotonic Imbalance Bounding’’ (MIB) class of matching methods from which CEM is derived. We summarize what is known about CEM and MIB, derive and illustrate several new desirable statistical properties of CEM, and then propose a variety of useful extensions. We show that CEM possesses a wide range of statistical properties not available in most other matching methods but is at the same time exceptionally easy to comprehend and use. We focus on the connection between theoretical properties and practical applications. We also make available easy-to-use open source software for R, Stata, and SPSS that implement all our suggestions.
This article outlines a network approach to the study of technological change. We propose that new inventions reshape networks of interlinked technologies by shifting inventors’ attention to or away from the knowledge on which those inventions build. Using this approach, we develop novel indexes of the extent to which a new invention consolidates or destabilizes existing technology streams. We apply these indexes in analyses of university research commercialization and find that, although federal research funding pushes campuses to create inventions that are more destabilizing, deeper commercial ties lead them to produce technologies that consolidate the status quo. By quantifying the effects that new technologies have on their predecessors, the indexes we propose allow patent-based studies of innovation to capture conceptually important phenomena that are not detectable with established measures. The measurement approach presented here offers empirical insights that support theoretical development in studies of innovation, entrepreneurship, technology strategy, science policy, and social network theory.
This paper was accepted by Lee Fleming, entrepreneurship and innovation.
Abstract
The majority of academic papers are scarcely cited while a few others are highly cited. A large number of studies indicate that there are many factors influencing the number of citations. An actual review is missing that provides a comprehensive review of the factors predicting the frequency of citations. In this review, we performed a search in WoS, Scopus, PubMed and Medline to retrieve relevant papers. In overall, 2087 papers were retrieved among which 198 relevant papers were included in the study. Three general categories with twenty eight factors were identified to be related to the number of citations: Category one: “paper related factors”: quality of paper; novelty and interest of subject; characteristics of fields and study topics; methodology; document type; study design; characteristics of results and discussion; use of figures and appendix in papers; characteristics of the titles and abstracts; characteristics of references; length of paper; age of paper; early citation and speed of citation; accessibility and visibility of papers. Category two: “journal related factors”: journal impact factor; language of journal; scope of journal; form of publication. Category three: “author(s) related factors”: number of authors; author’s reputation; author’s academic rank; self-citations; international and national collaboration of authors; authors’ country; gender, age and race of authors; author’s productivity; organizational features; and funding. Probably some factors such as the quality of the paper, journal impact factor, number of authors, visibility and international cooperation are stronger predictors for citations, than authors’ gender, age and race; characteristics of results and discussion and so on.
For testing that an underlying population is normally distributed the skewness and kurtosis statistics, √b1and b2, and the D’Agostino-Pearson K2 statistic that combines these two statistics have been shown to be powerful and informative tests. Their use, however, has not been as prevalent as their usefulness. We review these tests and show how readily available and popular statistical software can be used to implement them. Their relationship to deviations from linearity in normal probability plotting is also presented.
The academic and research policy communities have seen a long debate concerning the merits of peer review and quantitative citation-based metrics in evaluation of research. Some have called for replacing peer review with use of metrics for some evaluation purposes, while others have called for the use peer review informed by metrics. Whatever one's position, a key question is the extent to which peer review and quantitative metrics agree. In this paper we study the relation between the three journal metrics source normalized impact per paper (SNIP), raw impact per paper (RIP) and Journal Impact Factor (JIF) and human expert judgement. Using the journal rating system produced by the Excellence in Research for Australia (ERA) exercise, we examine the relationship over a set of more than 10,000 journals categorized into 27 subject areas. We analyze the relationship from the dimensions of correlation, distribution of the metrics over the rating tiers, and ROC analysis. Our results show that SNIP consistently has stronger agreement with the ERA rating, followed by RIP and then JIF along every dimension measured. The fact that SNIP has a stronger agreement than RIP demonstrates clearly that the increase in agreement is due to SNIP's database citation potential normalization factor. Our results suggest that SNIP may be a better choice than RIP or JIF in evaluation of journal quality in situations where agreement with expert judgment is an important consideration.
It has been proven that several features of scientific papers are relevant to citation impact. The purpose of this paper is to evaluate the role of these features and unravel which features have greater influence on citation impact. A feature space is established to describe four types of scientific papers' features: features of a paper itself features of authors, features of published journal, and features of citations. For a group of 676 articles published in 12 journals in the subject category of Information Science & Library Science (IS&LS) in 2007, we analyze quantitatively the difference among high-, medium-, and low-cited papers, and capture their influence on citation impact. The results make it clear that among these four feature types, the quality of a paper and the reputation of authors are the most and the least significant factor affecting the citation impact respectively, and a paper itself has greater influence than the published journal. The findings lay the foundation for the citation impact prediction by using the features of scientific papers.
The article presents three advanced citation-based methods used to detect
potential breakthrough papers among very highly cited papers. We approach the
detection of such papers from three different perspectives in order to provide
different typologies of breakthrough papers. In all three cases we use the
classification of scientific publications developed at CWTS based on direct
citation relationships. This classification establishes clusters of papers at
three levels of aggregation. Papers are clustered based on their similar
citation orientations and it is assumed that they are focused on similar
research interests. We use the clustering as the context for detecting
potential breakthrough papers. We utilize the Characteristics Scores and Scales
(CSS) approach to partition citation distributions and implement a specific
filtering algorithm to sort out potential highly-cited followers, papers not
considered breakthroughs in themselves. After invoking thresholds and
filtering, three methods are explored: A very exclusive one where only the
highest cited paper in a micro-cluster is considered as a potential
breakthrough paper (M1); as well as two conceptually different methods, one
that detects potential breakthrough papers among the two percent highest cited
papers according to CSS (M2a), and finally a more restrictive version where, in
addition to the CSS two percent filter, knowledge diffusion is also taken in as
an extra parameter (M2b). The advance citation-based methods are explored and
evaluated using specifically validated publication sets linked to different
Danish funding instruments including centres of excellence.
We consider statistical inference for regression when data are grouped into clusters, with regression model errors independent across clusters but correlated within clusters. Examples include data on individuals with clustering on village or region or other category such as industry, and state- year differences- in- differences studies with clustering on state. In such settings, default standard errors can greatly overstate estimator precision. Instead, if the number of clusters is large, statistical inference after OLS should be based on cluster- robust standard errors. We outline the basic method as well as many complications that can arise in practice. These include cluster- specifi c fi xed effects, few clusters, multiway clustering, and estimators other than OLS. © 2015 by the Board of Regents of the University of Wisconsin System.
During the Italian research assessment exercise, the national agency ANVUR performed an experiment to assess agreement between grades attributed to journal articles by informed peer review (IR) and by bibliometrics. A sample of articles was evaluated by using both methods and agreement was analyzed by weighted Cohen’s kappas. ANVUR presented results as indicating an overall “good” or “more than adequate” agreement. This paper re-examines the experiment results according to the available statistical guidelines for interpreting kappa values, by showing that the degree of agreement (always in the range 0.09–0.42) has to be interpreted, for all research fields, as unacceptable, poor or, in a few cases, as, at most, fair. The only notable exception, confirmed also by a statistical meta-analysis, was a moderate agreement for economics and statistics (Area 13) and its sub-fields. We show that the experiment protocol adopted in Area 13 was substantially modified with respect to all the other research fields, to the point that results for economics and statistics have to be considered as fatally flawed. The evidence of a poor agreement supports the conclusion that IR and bibliometrics do not produce similar results, and that the adoption of both methods in the Italian research assessment possibly introduced systematic and unknown biases in its final results. The conclusion reached by ANVUR must be reversed: the available evidence does not justify at all the joint use of IR and bibliometrics within the same research assessment exercise.
Scholarly impact is studied frequently and used to make consequential decisions (e.g., hiring, tenure, promotion, research support, professional honors), and therefore it is important to measure it accurately. Developments in information technology and statistical methods provide promising new metrics to complement traditional information sources (e.g., peer reviews). The introduction of Hirsch's (200528.
Hirsch , J. E. 2005. An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences, 102: 16569–16572. doi:10.1073/pnas.0507655102 [CrossRef], [PubMed], [Web of Science ®]View all references) h index—the largest number h such that at least h articles are cited h times each, or the length of the largest square in a citations × articles array—sparked an explosion in research on the measurement of scholarly impact. We evaluate 22 metrics, including conventional measures, the h index, and many variations on the h theme. Our criteria encompass conceptual, empirical, and practical issues: ease of understanding, accuracy of calculation, effects on incentives, influence of extreme scores, and validity. Although the number of publications fares well on several criteria, the most attractive measures include h, several variations that credit citations outside the h square, and two variations that control for career stage. Additional data suggest that adjustments for self-citations or shared authorship probably would not improve these measures much, if at all. We close by considering which measures are most suitable for research and practical applications.
The increasing dominance of team science highlights the importance of understanding the effects of team composition on the creativity of research results. In this paper, we analyze the effect of team size, and field and task variety on creativity. Furthermore, we unpack two facets of creativity in science: novelty and impact. We find that increasing team size has an inverted-U shaped relation with novelty. We also find that the size–novelty relationship is largely due to the relation between size and team field or task variety, consistent with the information processing perspective. On the other hand, team size has a continually increasing relation with the likelihood of a high-impact paper. Furthermore, variety does not have a direct effect on impact, net of novelty. This study develops our understanding of team science and highlights the need for a governance approach to scientific work. We also advance the creativity literature by providing an ex ante objective bibliometric measure that distinguishes novelty from impact, and illustrate the distinct team-level drivers of each. We conclude with a discussion of the policy implications of our findings.
IN SOME APPLICATIONS of the general linear model, the usual assumptions of homoscedastic disturbances and fixed coefficients may be questioned. When these requirements are not met, the loss in efficiency in using ordinary least squares (OLS) may be substantial and, more importantly, the biases in estimated standard errors may lead to invalid inferences. This has caused a number of writers to propose models which relax these conditions and to devise estimators for their more general specifications, e.g., Goldfeld and Quandt (8) for heteroscedasticity and Hildreth and Houck (11) for random coefficients. However, because the effect of introducing random coefficient variation is to give the dependent variable a different variance at each observation, models with this feature can be considered as particular heteroscedastic formulations for the purpose of detecting departure from the standard linear model. A test for heteroscedasticity with the same asymptotic properties as the likelihood ratio test in standard situations, but which can be computed by two least squares regressions, thereby avoiding the iterative calculations necessary to obtain maximum likelihood estimates of the parameters in the full model, is considered in this paper. The approach is based on the Lagrangian multiplier
This study examines whether there are some general trends across subject fields regarding the factors affecting the number of citations of articles, focusing especially on those factors that are not directly related to the quality or content of articles (extrinsic factors). For this purpose, from 6 selected subject fields (condensed matter physics, inorganic and nuclear chemistry, electric and electronic engineering, biochemistry and molecular biology, physiology, and gastroenterology), original articles published in the same year were sampled (n = 230–240 for each field). Then, the citation counts received by the articles in relatively long citation windows (6 and 11 years after publication) were predicted by negative binomial multiple regression (NBMR) analysis for each field. Various article features about author collaboration, cited references, visibility, authors' achievements (measured by past publications and citedness), and publishing journals were considered as the explanatory variables of NBMR. Some generality across the fields was found with regard to the selected predicting factors and the degree of significance of these predictors. The Price index was the strongest predictor of citations, and number of references was the next. The effects of number of authors and authors' achievement measures were rather weak.