Figure 4 - uploaded by Martin Hensher
Content may be subject to copyright.
Source publication
Objective
The study sought to review the different assessment items that have been used within existing health app evaluation frameworks aimed at individual, clinician, or organizational users, and to analyze the scoring and evaluation methods used in these frameworks.
Materials and Methods
We searched multiple bibliographic databases and conducte...
Context in source publication
Context 1
... Other studies did not elaborate on their scaling method (n ¼ 23). 2,[4][5][6][9][10][11][12][13]17,[92][93][94][95][96][97][98][99][100][101][102][103][104] For the scoring modalities, the most popular approach taken was the calculation of mean or average scores (n ¼ 22 studies, 23% of the total number of studies) (Figure 4; Supplementary Appendix S4). 20,21,27,30,32,33,35,[37][38][39]41,43,45,46,50,54,56,70,71,81,98 Thirteen (13%) studies presented their scores as a sum or total, and 11 (11%) studies used a mixed of mean, median, interquartile range, percentage, or total scoring. ...
Citations
... These findings suggest that user ratings and downloads are not a good proxy for overall app quality. However, most frameworks are not all-encompassing [20][21][22][23], for example, the MARS doesn't include privacy questions. Hence, from the previous findings, it is unclear whether user ratings and download rates may be associated with compliance with quality components, such as user experience (UX), professional/clinical assurance (PCA) and data privacy (DP). ...
Objective
To analyse the relationship between health app quality with user ratings and the number of downloads of corresponding health apps.
Materials and methods
Utilising a dataset of 881 Android-based health apps, assessed via the 300-point objective Organisation for the Review of Care and Health Applications (ORCHA) assessment tool, we explored whether subjective user-level indicators of quality (user ratings and downloads) correlate with objective quality scores in the domains of user experience, data privacy and professional/clinical assurance. For this purpose, we applied spearman correlation and multiple linear regression models.
Results
For user experience, professional/clinical assurance and data privacy scores, all models had very low adjusted R squared values (< .02). Suggesting that there is no meaningful link between subjective user ratings or the number of health app downloads and objective quality measures. Spearman correlations suggested that prior downloads only had a very weak positive correlation with user experience scores (Spearman = .084, p = .012) and data privacy scores (Spearman = .088, p = .009). There was a very weak negative correlation between downloads and professional/clinical assurance score (Spearman = -.081, p = .016). Additionally, user ratings demonstrated a very weak correlation with no statistically significant correlations observed between user ratings and the scores (all p > 0.05). For ORCHA scores multiple linear regression had adjusted R-squared = -.002.
Conclusion
This study highlights that widely available proxies which users may perceive to signify the quality of health apps, namely user ratings and downloads, are inaccurate predictors for estimating quality. This indicates the need for wider use of quality assurance methodologies which can accurately determine the quality, safety, and compliance of health apps. Findings suggest more should be done to enable users to recognise high-quality health apps, including digital health literacy training and the provision of nationally endorsed “libraries”.
... For example, mechanisms, such as the One-Time-Pin (OTP), biometric trait verification, and liveness assessment to authenticate customers digitally are increasingly becoming standard practice across various industries. Unfortunately, individual software development teams continuously reinvent the wheel as no standard approach exists when defining a new sophisticated security mechanism [4] [5]. In addition to the challenges software developers face in keeping up with evolving security threats, the 2021 Global DevSecOps Survey [6] indicates constant friction between security professionals and software developers. ...
To stay competitive in a fast-evolving landscape, lifestyle, e-commerce, finance, and health sectors use mobile apps to enhance their traditional capabilities with new innovative features to satisfy customer demands. Developing sophisticated and secure mobile apps requires a skilled understanding of software development techniques, protection mechanisms, and mobile security practices to safeguard customers against cyberattacks. Software development teams often apply frameworks and best practices according to their unique experience and knowledge, resulting in vulnerable mobile apps. Numerous software development challenges, a lack of guidelines, and a standardised agile approach for creating secure mobile apps need to be addressed. The primary objective of this research is to initiate the process of delineating mobile application security drivers. This initial step lays the foundation for the subsequent development of a comprehensive, secure software development framework for mobile applications, with an overarching emphasis on security across all stages of development. This framework is designed to be adaptable and customizable to meet the specific security needs of diverse industries. The security drivers can form the foundation for a novel framework to guide the creation of secure mobile apps.
... Along with the growth of the mHealth market, numerous measurement tools have been developed to evaluate the user experience of mHealth apps. However, no universal evaluation framework covers the user experience factors identified as important in mHealth app evaluation [56]. In addition, since most evaluation measurement tools were developed with a focus on usability, they lacked detailed considerations of health literacy and aesthetics. ...
The rapid growth of the mHealth market has led to the development of several tools to evaluate user experience. However, there is a lack of universal tools specifically designed for this emerging technology. This study was conducted with the aim of developing and verifying a user experience evaluation scale for mHealth apps based on factors proposed in previous research. The initial draft of the tool was created following a comprehensive review of existing questionnaires related to mHealth app evaluation. The validity of this scale was then tested through exploratory and confirmatory factor analysis. The results of the factor analysis led to the derivation of 16 items, which were conceptually mapped to five factors: ease of use and satisfaction, information architecture, usefulness, ease of information, and aesthetics. A case study was also conducted to improve mHealth apps concerning personal health records using this scale. In conclusion, the developed user experience evaluation scale for mHealth apps can provide comprehensive user feedback and contribute to the improvement of these apps.
... Apps can be affected by the areas of, for example, data protection, data security, consumer protection, medicine, and social law. App-specific laws, on the contrary, are rare [18,19]. In Germany, apps as medical devices have been officially made available for prescription since October 2020. ...
... Apps are not entirely unregulated, but app-specific law is still rare [18,19]. For example, data protection law requires a legal basis for the data processing of an app [38], and the MDR sets high standards of quality and safety for software entering the market as a medical device [39]. ...
Background: The terms health app and medical app are often used interchangeably but do not necessarily mean the same thing. To better understand these terms and better regulate such technologies, we need distinct definitions of health and medical apps.
Objective: This study aimed to provide an overview of the definitions of health and medical apps from an interdisciplinary perspective. We summarized the core elements of the identified definitions for their holistic understanding in the context of digital public health.
Methods: The legal frameworks for medical device regulation in the United States, the European Union, and Germany formed the basis of this study. We then searched 6 databases for articles defining health or medical apps from an interdisciplinary
perspective. The narrative literature review was supported by a forward and backward snowball search for more original definitions of health and medical apps. A qualitative analysis was conducted on the identified relevant aspects and core elements of each definition. On the basis of these findings, we developed a holistic definition of health and medical apps and created a decision flowchart to highlight the differences between the 2 types.
Results: The legal framework showed that medical apps could be regulated as mobile medical devices, whereas there is no legal term for health apps. Our narrative literature review identified 204 peer-reviewed publications that offered a definition of health and medical apps. After screening for original definitions and applying the snowball method, 11.8% (24/204) of the publications were included in the qualitative analysis. Of these 24 publications, 22 (88%) provided an original definition of health apps and 11 (44%) described medical apps. The literature suggests that medical apps are a part of health apps. To describe health or medical apps, most definitions used the user group, a description of health, the device, the legal regulation, collected data, or technological functions. However, the regulation should not be a distinction criterion as it requires legal knowledge, which is neither suitable nor practical. An app’s intended medical or health use enables a clear differentiation between health and medical apps. Ultimately, the health aim of an app and its main target group are the only distinction criteria.
Conclusions: Health apps are software programs on mobile devices that process health-related data on or for their users. They can be used by every health-conscious person to maintain, improve, or manage the health of an individual or the community. As an umbrella term, health apps include medical apps. Medical apps share the same technological functions and devices. Health professionals, patients, and family caregivers are the main user groups. Medical apps are intended for clinical and medical purposes and can be legally regulated as mobile medical devices.
... Issues of best practices, evidence-base, privacy, and security are particularly important for app design and development. Moreover, the subjective nature of user experience factors renders them difficult to assess reliably in evaluation frameworks, with measurement dependent on the rater (Hensher et al., 2021;Lagan et al., 2021a). Human-centered design approaches that involve intended users in the design and development of the app may be useful (Ng et al., 2019). ...
American Indian/Alaska Native (AI/AN) communities experience high rates of several mental health (MH) difficulties (e.g., trauma, substance use, suicide) and encounter significant barriers to accessing MH care. Several barriers relate to rural residence (e.g., geographical distance to healthcare facilities, provider shortages) and cultural factors (e.g., racial discrimination, cultural misunderstandings, mistrust). Remotely delivered care via telehealth has demonstrated promise in increasing AI/AN communities’ access to MH care, but provider shortages limit scalability. Mobile MH applications (“apps”) may represent a scalable option for increasing AI/AN communities’ access to MH interventions but need to be culturally tailored to address cultural barriers to MH care. Although additional research is needed, several MH apps have demonstrated promise in addressing MH concerns (e.g., depression, anxiety). Few MH apps have been developed or evaluated for AI/AN communities, and greater attention to the development of MH apps for them is needed. Several models exist for evaluating MH apps, but they have not been applied to apps for AI/AN communities. This article provides guidance on factors to consider when developing apps for AI/AN communities, based on a review of the literature and existing apps for AI/AN communities. Some recommendations include using a bidirectional approach rooted in humility and cultural safety principles to promote mutual learning and patient-centered care. Partnering with AI/AN communities in developing MH apps is vital. These recommendations also may be valuable for those serving additional populations experiencing healthcare inequities and cultural barriers to MH care.
... To make digital health applications and platforms more accessible, several European countries are in the process of adopting a statutory reimbursement obligation by amending the traditional assessment frameworks, to fit the particularities of digital health solutions. Looking at reimbursement process of EU countries, different regulations can be identified for DHTs, while there is clearly an increased interest in implementing reimbursement options coupled with assessment frameworks (4,5). The reimbursement options are mixed: in some countries DHTs are partly be paid voluntarily by patients and individual health insurers, in others, a DHT with certain proof of patient benefit are paid obligatorily by all health insurers (e.g., Germany) (6). ...
Background
Evaluating and integrating digital health technologies is a critical component of a national healthcare ecosystem in the 2020s and is expected to even increase in significance.
Design
The paper gives an overview of international practices on public financing and health technology assessment of digital health technologies (DHTs) in five European Union (EU) countries and outlines recommendations for country-level action that relevant stakeholders can consider in order to support uptake of digital health solutions in Hungary. A scoping review was carried out to identify and gather country-specific classifications and international practices on the financing DHTs in five pioneering EU countries: Germany, France, Belgium, the United Kingdom and Finland.
Results
Several frameworks have been developed for DHTs, however there is no single, unified framework or method for classification, evaluation, and financing of digital health technologies in European context. European countries apply different taxonomy, use different assessment domains and regulations for the reimbursement of DHTs. The Working Group of the Hungarian Health Economic Society recommends eight specific points for stakeholders, importantly taking active role in shaping common clinical evidence standards and technical quality criteria across in order for common standards to be developed in the European Union single market.
Conclusion
Specificities of national healthcare contexts must be taken into account in decisions to allocate public funds to certain therapies rather than others.
... Although there is guidance such as International Organization for Standardization (ISO) [9], currently there appears to be no gold standard for digital health apps' UX assessment [10][11][12][13]. The current literature [10][11][12][13] suggests that improvements could be made in the development and evaluation of digital health apps in terms of clinical assurance, data privacy and UX. ...
... Although there is guidance such as International Organization for Standardization (ISO) [9], currently there appears to be no gold standard for digital health apps' UX assessment [10][11][12][13]. The current literature [10][11][12][13] suggests that improvements could be made in the development and evaluation of digital health apps in terms of clinical assurance, data privacy and UX. ...
... Although a 2021 scoping review [10] examined the domain/criteria needed to evaluate digital health apps found that UX together with information validity has been the most evaluated criteria. This scoping review included frameworks such as system usability scale (SUS) and system usability measurement inventory (SUMI). ...
There are more than 350,000 digital health apps in the app stores today. There is an average of around 250 digital health apps being added daily to the app stores. This indicates a public appetite to consume digital health apps, and a potential to decrease the pressure on healthcare providers by promoting digital technology. However, for this to be a viable option, digital health apps must be effective and safe to use. The quality, defined as “compliance with best practice standards”, of such apps must be of the highest standard for these to be recommended by healthcare professionals. One crucially important aspect of the quality of an app is user experience (UX). The UX of digital health apps is important to assure that the technology is being used safely and as intended. The objective of this study was to describe common practices related to the UX for digital health app design. This study analysed a sample of 2,053 digital health apps with a focus on the UX practices in the digital health app design. The data included in this study was collected using the Organisation for the Review of Care and Health Applications (ORCHA) assessment tool. ORCHA is a United Kingdom (UK) based digital health compliance company that specialises in assessing the quality of digital health apps, which includes UX. The ORCHA UX assessment consists of 15 polar questions (Yes/No) and 3 multiple selection questions. A score of 65 is considered a ‘threshold score’ and a starting point, based on answers this score will increase, decrease, or stay the same.KeywordsDigital healthmHealthUser experience
... CURRENT EVIDENCE ASSESSMENT FRAMEWORKS Dozens of frameworks have been proposed to assess evidentiary support for DHIs 13,[24][25][26][27][28] . A comprehensive review is outside the scope of this paper, but a preliminary catalogue of frameworks All of the following: ...
Dozens of frameworks have been proposed to assess evidence for digital health interventions (DHIs), but existing frameworks may not facilitate DHI evidence reviews that meet the needs of stakeholder organizations including payers, health systems, trade organizations, and others. These organizations may benefit from a DHI assessment framework that is both rigorous and rapid. Here we propose a framework to assess Evidence in D igital health for EF fectiveness of IN terventions with E valuative D epth (Evidence DEFINED). Designed for real-world use, the Evidence DEFINED Quick Start Guide may help streamline DHI assessment. A checklist is provided summarizing high-priority evidence considerations in digital health. Evidence-to-recommendation guidelines are proposed, specifying degrees of adoption that may be appropriate for a range of evidence quality levels. Evidence DEFINED differs from prior frameworks in its inclusion of unique elements designed for rigor and speed. Rigor is increased by addressing three gaps in prior frameworks. First, prior frameworks are not adapted adequately to address evidence considerations that are unique to digital health. Second, prior frameworks do not specify evidence quality criteria requiring increased vigilance for DHIs in the current regulatory context. Third, extant frameworks rarely leverage established, robust methodologies that were developed for non-digital interventions. Speed is achieved in the Evidence DEFINED Framework through screening optimization and deprioritization of steps that may have limited value. The primary goals of Evidence DEFINED are to a) facilitate standardized, rapid, rigorous DHI evidence assessment in organizations and b) guide digital health solutions providers who wish to generate evidence that drives DHI adoption.
... The solution with the resilient, informed and supported peers is target-oriented for the peer chat (O'Leary, Schueller, Wobbrock, & Pratt, 2018), but has to be evaluated again within the project with the target group (e.g., subjective security), as well as the usability of the prototype. The development effort for this will be large, because usability testing is too rarely done for health apps (Grist, Rebecca, Porter, & Stallard, 2017), often has limitations (Dawson, et al., 2020), and is rarely based on validated frameworks (Chan, Torous, Hinton, & Yellowlees, 2015;Hensher, et al., 2021). In addition, many evaluation tools are for experts (e.g., mHAT), rather than end users (Azad-Khaneghah, Neubauer, Cruz, & Liu, 2021;O'Rourke, Pryss, Schlee, & Probst, 2020) and there are only questionnaires on subjective safety when using websites (Corritore, Kracher, & Wiedenbeck, 2003) in general and chatbots (Jian, Bisantz, & Drury, 2000), but not for from AI (Note 1). ...
Objectives: Post-traumatic stress disorder (PTSD) is a common psychological burden for (adolescent) refugees. Therefore, low-threshold and target group-oriented support services are needed. Methods: an artificial-intelligence-based PTSD self-screening tool was developed and integrated into a support app, which was also developed within the project itself. To make the app attractive, a target group analysis was implemented with adolescent refugees in Germany (N = 53; age 20.2 years, SD = 3.2; average length of stay in Germany 4.6 years, SD = 1.4) regarding desired psychoeducational offers and content. Results: The adolescents showed a strong interest in visually presented information, self-directed posting, and peer chat. Responses were discussed in an interdisciplinary workshop for the ethical legal and social impact (ELSI-workshop) (N = 12) to generate implementation options as follows. To enable self-determined posting and protect privacy, it was agreed to allow anonymous user uploads after an administrator check to ensure quality and maintain anonymity. For the peer chat, an implementation with trained young adults (age 18-25), with migration experience but without traumatization, was considered suitable, if they are integrated into support and supervision structures. Furthermore, it was decided to prepare existing information materials (e. g., about traumatization and different forms of therapy) as video and infographics. Conclusion and Implications: This article shows how the development of technology for vulnerable target groups can be implemented in a user-oriented manner while taking ELSI aspects into account.
... In line with studies such as Azad-Khaneghah et al. (2021), it is striking to note the large number of papers that refer to the MARS scale as the first tool to measure the quality of health apps. Although it has been adapted into different languages and versions, and has gained obvious recognition in the scientific community Böhme et al., 2019;Domnich et al., 2016;Martín-Payo et al., 2019;Messner et al., 2020;Saliasi et al., 2021;Stoyanov et al., 2016), there are some shortcomings, as no dimensions or indicators related to privacy and security issues are included; conclusions that were also reached by Hensher et al. (2021) and Jeminiwa et al. (2019). ...
En los últimos años ha aumentado significativamente la descarga de aplicaciones móviles (apps) orientadas a desarrollar hábitos saludables en la población en general. No obstante, la mayoría no están sujetas a procesos de regulación de calidad, y los instrumentos de evaluación disponibles pueden estar obsoletos. Esta revisión se ha realizado para conocer el estado del arte sobre procesos de diseño, construcción, validación y principales resultados obtenidos en estudios empíricos acerca de los instrumentos de evaluación de la calidad de dichas apps, todo ello a fin de identificar las principales dimensiones e indicadores comunes. Se han llevado a cabo búsquedas en cuatro bases de datos de alto impacto. Se concluye que es un tema de interés reciente y actual en la literatura científica. De 11 documentos incluidos, se han hallado 97 indicadores organizados en 17 dimensiones. Sólo en dos de ellos se incluyeron aspectos relacionados con la privacidad y la seguridad. En definitiva, se ha configurado una lista significativa y actualizada que podrá ser la base para construir un marco de evaluación que permita hacer un estudio integral de estas apps, pues muchos de los instrumentos de evaluación no han sido diseñados pensando en la población en general.
