Figure 1 - uploaded by Robert Schulz
Content may be subject to copyright.
Reporting prevalence for rigour and sample criteria. This plot displays the percentage of trials that addressed each criteria. For information on the actual randomisation or blinding status, please refer to the text. The different coloured data points are for better visual differentiation of each subcategory. Created by the authors.
Source publication
Objectives
Transparent reporting of clinical trials is essential to assess the risk of bias and translate research findings into clinical practice. While existing studies have shown that deficiencies are common, detailed empirical and field-specific data are scarce. Therefore, this study aimed to examine current clinical trial reporting and transpa...
Contexts in source publication
Context 1
... hundred and seventy-five articles were screened, and 168 articles were reviewed from 27 sports medicine and orthopaedics journals (online supplemental figure S1, online supplemental table S1). Eleven articles were excluded because they did not meet the ICMJE clinical trial criteria. ...
Context 2
... reporting prevalence of sample size calculations and related results can be found in figure 1. In trials not reporting a priori sample size calculation (figure 1), 2% (95% CI 0% to 5%; n=4) reported that no sample size calculation was performed because the study was an exploratory pilot study. ...
Context 3
... reporting prevalence of randomisation, allocation concealment and related results can be found in figure 1. In trials not addressing randomisation (figure 1), two trials (1%; 95% CI 0% to 3%) were not randomised, and five trials did not mention randomisation (3%; 95% CI 0% to 6%). ...
Context 4
... reporting prevalence of statements on blinding of different stakeholders can be found in figure 1. The actual blinding status of included trials is visualised in figure 2. Two-thirds of the trials addressed blinding (figure 2). ...
Context 5
... reporting prevalence of criteria related to the study sample can be found in figure 1. Approximately threequarters of the trials reported the inclusion and exclusion criteria and provided complete information on the number of participants at enrolment, after enrolment and included in data analysis ( figure 1). ...
Context 6
... reporting prevalence of criteria related to the study sample can be found in figure 1. Approximately threequarters of the trials reported the inclusion and exclusion criteria and provided complete information on the number of participants at enrolment, after enrolment and included in data analysis ( figure 1). Fewer trials used a flow chart to illustrate the number of included and excluded participants at each stage. ...
Citations
... This supports the idea for the need for transparent justification of new studies and the need for strong research methods in orthopaedics, sports medicine and rehabilitation [2,4]. Applying checklists and reporting guidelines during the production and publication process and ensuring the implementation of research findings are relevant steps to improve research practice, which in turn makes the research outcome more valuable for daily practice [5,6]. ...
... There is increasing interest in the rigour and transparency, or lack thereof, of Sports Medicine, Exercise Science and Orthopaedic research [16]. Reporting in sports medicine and exercise science papers is often inadequate, and there are concerns about the reproducibility and veracity of many findings [16][17][18][19][20]. Improving reporting practices in sports medicine should be a priority for researchers and publishers [16,21]. ...
... There is increasing interest in the rigour and transparency, or lack thereof, of Sports Medicine, Exercise Science and Orthopaedic research [16]. Reporting in sports medicine and exercise science papers is often inadequate, and there are concerns about the reproducibility and veracity of many findings [16][17][18][19][20]. Improving reporting practices in sports medicine should be a priority for researchers and publishers [16,21]. ...
... There is increasing interest in the rigour and transparency, or lack thereof, of Sports Medicine, Exercise Science and Orthopaedic research [16]. Reporting in sports medicine and exercise science papers is often inadequate, and there are concerns about the reproducibility and veracity of many findings [16][17][18][19][20]. Improving reporting practices in sports medicine should be a priority for researchers and publishers [16,21]. This exploratory research aimed to answer the following research question: How accurately can an AI-LLM measure reporting guideline compliance in a sample of sports medicine clinical trial reports? ...
Background: Adherence to established reporting guidelines can improve clinical trial reporting standards, but attempts to improve adherence have produced mixed results. This exploratory study aimed to determine how accurately a Large Language Model generative AI system (AI-LLM) could measure reporting guideline compliance in a sample of sports medicine clinical trial reports.
Methods: The OpenAI GPT-3.5 AI-LLM was evaluated for its ability to determine reporting guideline adherence in a sample of 113 published clinical trial reports from the Sports Medicine field. For each paper, the model was prompted to answer a series of reporting guideline questions paired with the paper's relevant text. The dataset was randomly split (80/20) into a TRAIN and TEST dataset, stratified by paper section. Hyperparameter and model fine-tuning were performed using the TRAIN dataset. Model performance (F1-score, classification accuracy) was assessed using the TEST dataset.
Results: across all questions, the AI-LLM demonstrated acceptable performance (F1-score = 86%). However, there was significant variation in performance between different reporting guideline questions (accuracy between 70-100%). The model was most accurate when asked to identify a defined primary objective or endpoint and least accurate when asked to identify an effect size and related confidence interval.
Discussion: Limitations notwithstanding, the AI-LLM showed promise as a tool for assessing reporting guideline compliance, potentially reducing the workload for peer reviewers and editors. Next steps should include developing a cost-effective, open-source AI-LLM and exploring methods to improve model accuracy.
... The high rate of significant results reported in sport-science and -medicine journals is concerning, as many studies may be reporting false positive results suggesting the effectiveness of a treatment when it is actually not effective. 1 Given the high proportion of significant findings reported in sport-science journals (eg, 82% of studies published in sport-science journals report significant results 2 ), it is unlikely that all of these findings represent true effects. Indeed, results from high-quality designed studies shall be adequately powered to detect true effects. ...
... Z 1−β = 0.84 (critical value at 1 − β, assuming a power of 0.8). Using the ratio of sample size as 1, we can set the sample size of the control group (nC) as 1. nE = [(Z α/2 + Z 1−β ) 2 × (σ 1 5. Since you specified a ratio of 1, the sample size of the control group (nC) will also be 38.5.Therefore, the recommended sample size for each group would be approximately 39 participants ( Figure S4 in the Supplementary Material [available online]). ...
Purpose:
To investigate the accuracy of ChatGPT (Chat generative pretrained transformer), a large language model, in calculating sample size for sport-sciences and sports-medicine research studies.
Methods:
We conducted an analysis on 4 published papers (ie, examples 1-4) encompassing various study designs and approaches for calculating sample size in 3 sport-science and -medicine journals, including 3 randomized controlled trials and 1 survey paper. We provided ChatGPT with all necessary data such as mean, percentage SD, normal deviates (Zα/2 and Z1-β), and study design. Prompting from 1 example has subsequently been reused to gain insights into the reproducibility of the ChatGPT response.
Results:
ChatGPT correctly calculated the sample size for 1 randomized controlled trial but failed in the remaining 3 examples, including the incorrect identification of the formula in one example of a survey paper. After interaction with ChatGPT, the correct sample size was obtained for the survey paper. Intriguingly, when the prompt from Example 3 was reused, ChatGPT provided a completely different sample size than its initial response.
Conclusions:
While the use of artificial-intelligence tools holds great promise, it should be noted that it might lead to errors and inconsistencies in sample-size calculations even when the tool is fed with the necessary correct information. As artificial-intelligence technology continues to advance and learn from human feedback, there is hope for improvement in sample-size calculation and other research tasks. However, it is important for scientists to exercise caution in utilizing these tools. Future studies should assess more advanced/powerful versions of this tool (ie, ChatGPT4).
... Scientific medical publications are a key tool in supporting this process. Therefore, raising publication standards in orthopedics and sports medicine will be a crucial task in the coming years [38,41]. With methodological courses and a series of articles on research methods published in KSSTA, the European Society of Sports Traumatology, Knee Surgery, and Arthroscopy (ESSKA) is continuously contributing to the process of educating physicians and other professionals in the field. ...
This Article provides recommendations on which Checklists to use when reporting new medical research and appropriate Risk of Bias Tools when assessing research quality.
... A phenomenon more subtle and deep-rooted in institutional norms than misconduct, and which may do more long-term harm to scientific enquiry, is questionable research practice (QRP; John et al., 2012;Schulz et al., 2022). QRPs are introduced deliberately or inadvertently into study design (e.g., biased and poorly controlled experiments), data collection (e.g., insufficient blinding), data analysis (e.g., incorrect or inappropriate statistical procedures), and data reporting (e.g., post hoc hypothesizing), leading to nonreplicable results and conclusions (Büttner et al., 2020). ...
Increasing transparency and openness in science is an ongoing endeavor, one that has stimulated self-reflection and reform in many fields. However, kinesiology and its related disciplines are among those exhibiting an "ostrich effect" and a reluctance to acknowledge their methodological shortcomings. Notwithstanding several high-profile cases of scientific misconduct, scholars in the field are frequently engaged in questionable research practices (QRPs) such as biased experimental designs, inappropriate statistics, and dishonest/inexplicit reporting. To advance their careers, researchers are also "gaming the system" by manipulating citation metrics and publishing in predatory and/or pay-to-publish journals that lack robust peer review. The consequences of QRPs in the discipline may be profound: from increasing the false positivity rate to eroding public trust in the very institutions tasked with informing public health policy. But what are the incentives underpinning misconduct and QRPs? And what are the solutions? This narrative review is a consciousness raiser that explores (a) the manifestations of QRPs in kinesiology; (b) the excessive publication pressures, funding pressures, and performance incentives that are likely responsible; and (c) possible solutions for reform.
... A phenomenon more subtle and deep-rooted in institutional norms than misconduct, and which may do more long-term harm to scientific enquiry, is questionable research practice (QRP) (John et al., 2012;R. Schulz et al., 2022). Questionable research practices are introduced deliberately or inadvertently into study design (e.g., biased and poorly controlled experiments), data collection (e.g., insufficient blinding), data analysis (e.g., incorrect or inappropriate statistical procedures), and data reporting (e.g., post hoc hypothesizing), leading to non-replic ...
... Despite the general eagerness of authors to improve the quality of their studies, a recent paper highlighted the diversity in reporting clinical trials and the transparency in orthopaedics and sports medicine research practices [8]. The definition of the primary outcome and a clear presentation of the hypothesis for a study are essential at the end of its introduction, and the clinical importance of its findings should be emphasized. ...
... An important limitation for further generalization is the sample size of this study due to the nature of its pilot character. For reaching high method standards, as nowadays recommended for orthopedic trials [18,19], funding is usually needed and the value of the study must be proven in advance. Whenever possible, new studies should be justified through high standard Systematic Reviews showing the need for further studies on the topic [20][21][22]. ...
This pilot study aimed to determine the reliability of a newly developed ultrasound-based protocol for the assessment of patella diameter and sulcus angle. The diameter of the patella expressed in mm and the sulcus angle, expressed in degrees were measured in the right knee in 12 healthy participants (eight women and four men) in two separate sessions by two examiners (experienced rater and inexperienced rater) using ultrasonography according to a developed standardized protocol. The reliability was determined on the calculated intraclass correlation coefficient, ICC, expressed as a 95% confidence interval (lower bound, upper bound). For the patella diameter measurement, intra-rater and inter-rater reliability were good to excellent, with the ICC exceeding 0.836-0.998 and 0.859-0.997, respectively. The intra-rater and inter-rater reliability of the sulcus measurement was moderate to excellent, as the ICC amounted to 0.559-0.993 and 0.559-0.990, respectively. The reliability of both measures increased with the experience of the examiner. Therefore, it was determined that the newly developed protocol for an ultrasound-based assessment of patella diameter and sulcus angle is reliable. Further studies validating their clinical use should be carried out.
... A single-blind controlled crossover design was used and the study was conducted in accordance with CONSORT checklist of information to include when reporting randomized crossover trials [7]. Current recommendations for reporting sports medicine and orthopedic clinical trials have been respected [8]. The study was approved by the appropriate ethical committee related to the institution in which it was performed. ...
Objective:
This single-blind randomized controlled crossover pilot trial investigated whether hard or soft knee orthotics affect the back in action (BIA) test battery performance.
Methods:
Twenty-four healthy participants (13 males, 11 females) were randomly assigned into three equal groups differentiated through the order of device use. The data were collected in a laboratory setting. BIA test battery (balance tests, vertical jumps, and parkour hop tests) was run with a rigid orthotic device, a soft brace, or no aid in a crossover order. Analysis of Variance repeated measures and Friedman Test were used to calculate depended-group differences.
Results:
No significant or clinically relevant effect or differences was observed between running the BIA with a soft brace, rigid orthosis, or no aid (p = 0.53-0.97) for all included tests. No adverse events have been observed.
Conclusion:
Soft and rigid knee braces do not affect performance in healthy participants. Missing experience with the devices might explain a few influences on feedback mechanisms. There is no disadvantage to be expected regarding healthy participants running back to sports.
Objectives
To synthesise research investigating data and code sharing in medicine and health to establish an accurate representation of the prevalence of sharing, how this frequency has changed over time, and what factors influence availability.
Design
Systematic review with meta-analysis of individual participant data.
Data sources
Ovid Medline, Ovid Embase, and the preprint servers medRxiv, bioRxiv, and MetaArXiv were searched from inception to 1 July 2021. Forward citation searches were also performed on 30 August 2022.
Review methods
Meta-research studies that investigated data or code sharing across a sample of scientific articles presenting original medical and health research were identified. Two authors screened records, assessed the risk of bias, and extracted summary data from study reports when individual participant data could not be retrieved. Key outcomes of interest were the prevalence of statements that declared that data or code were publicly or privately available (declared availability) and the success rates of retrieving these products (actual availability). The associations between data and code availability and several factors (eg, journal policy, type of data, trial design, and human participants) were also examined. A two stage approach to meta-analysis of individual participant data was performed, with proportions and risk ratios pooled with the Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis.
Results
The review included 105 meta-research studies examining 2 121 580 articles across 31 specialties. Eligible studies examined a median of 195 primary articles (interquartile range 113-475), with a median publication year of 2015 (interquartile range 2012-2018). Only eight studies (8%) were classified as having a low risk of bias. Meta-analyses showed a prevalence of declared and actual public data availability of 8% (95% confidence interval 5% to 11%) and 2% (1% to 3%), respectively, between 2016 and 2021. For public code sharing, both the prevalence of declared and actual availability were estimated to be <0.5% since 2016. Meta-regressions indicated that only declared public data sharing prevalence estimates have increased over time. Compliance with mandatory data sharing policies ranged from 0% to 100% across journals and varied by type of data. In contrast, success in privately obtaining data and code from authors historically ranged between 0% and 37% and 0% and 23%, respectively.
Conclusions
The review found that public code sharing was persistently low across medical research. Declarations of data sharing were also low, increasing over time, but did not always correspond to actual sharing of data. The effectiveness of mandatory data sharing policies varied substantially by journal and type of data, a finding that might be informative for policy makers when designing policies and allocating resources to audit compliance.
Systematic review registration
Open Science Framework doi: 10.17605/OSF.IO/7SX8U .
