Content uploaded by Jasmin Bhawra
Author content
All content in this area was uploaded by Jasmin Bhawra on Jan 25, 2024
Content may be subject to copyright.
JMIR Preprints Buchan et al
Evaluation of digital health platform development:
Application of an innovative methodology to build
infrastructure for digital transformation of health
systems
M. Claire Buchan, Tarun Reddy Katapally, Jasmin Bhawra
Submitted to: JMIR Formative Research
on: January 24, 2024
Disclaimer: © The authors. All rights reserved. This is a privileged document currently under peer-review/community
review. Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for
review purposes only. While the final peer-reviewed paper may be licensed under a CC BY license on publication, at this
stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Table of Contents
Original Manuscript ....................................................................................................................................................................... 5
Supplementary Files ..................................................................................................................................................................... 29
Figures ......................................................................................................................................................................................... 30
Figure 1
...................................................................................................................................................................................... 31
Figure 2
...................................................................................................................................................................................... 32
Figure 3
...................................................................................................................................................................................... 33
Figure 4
...................................................................................................................................................................................... 34
Multimedia Appendixes ................................................................................................................................................................. 35
Multimedia Appendix 1
.................................................................................................................................................................. 36
Multimedia Appendix 2
.................................................................................................................................................................. 36
Multimedia Appendix 3
.................................................................................................................................................................. 36
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Evaluation of digital health platform development: Application of an
innovative methodology to build infrastructure for digital transformation of
health systems
M. Claire Buchan1 BSc, MSc; Tarun Reddy Katapally2, 3, 4 MBBS, PhD; Jasmin Bhawra5 MSc, PhD
1School of Public Health Sciences University of Waterloo Waterloo CA
2DEPtH Lab, Faculty of Health Sciences Western University London CA
3Department of Epidemiology and Biostatistics Schulich School of Medicine and Dentistry Western University London CA
4Lawson Health Research Institute London CA
5CHANGE Research Lab, School of Occupational and Public Health Toronto Metropolitan University London CA
Corresponding Author:
Jasmin Bhawra MSc, PhD
CHANGE Research Lab, School of Occupational and Public Health
Toronto Metropolitan University
350 Victoria Street
London
CA
Abstract
Background: The current public health crises we face, ranging from communicable disease pandemics such as Coronavirus
disease (COVID-19) to endemic chronic diseases, require cohesive, collective, and deliberate societal efforts to address inherent
decision-making gaps in our health systems. Digital health platforms that leverage big data ethically from citizens can transform
health systems by enabling real-time data collection, communication, as well as precision prediction and health system rapid
responses. However, the lack of standardized and evidence-based methods to develop and implement digital health platforms
currently limits their application.
Objective: This study aimed to evaluate the development of a novel rapid response COVID-19 digital health platform by
engaging with the development team which includes computer programmers and data scientists, as well as the research team
consisting of interdisciplinary researchers (i.e., key stakeholders).
Methods: Using a developmental evaluation approach, this evaluation included two key components: 1) A qualitative survey
assessing digital health platform objectives, modifications, and challenges administered to five key members of the software
development team; and 2) A role-play pilot with key stakeholders to simulate real-world conditions, followed by a self-report
survey, to evaluate the utility of the digital health platform for each of its objectives. Survey data were analyzed using an
inductive thematic analysis approach. Post-pilot test survey data were aggregated and synthesized by participant role.
Results: The final digital health platform met original objectives, and was expanded to accommodate evolving needs of potential
users and COVID-19 regulations. Key challenges noted by the development team included navigating changing government
policies and restrictions, and supporting the data sovereignty of platform users. Strong team cohesion, communication, and
problem solving were all quintessential in the overall success of program development. Pilot test participants reported positive
experiences interacting with the platform and found its features relatively easy to use. Users in the community member role felt
that the platform accurately reflected their risk of contracting COVID-19, but reported challenges interacting with the interface,
particularly when submitting citizen reports and food status photos. Those in the decision-maker role found the data
visualizations intuitive in helping them to understand the information. Both participant groups highlighted the utility of a tutorial
for future users as there were some questions regarding some of the features.
Conclusions: Evaluation of the digital health platform development process informed our decisions to integrate the research
team more cohesively with the development team, resulting in a data scientist being part of both teams going forward. Another
key development process decision was to integrate more interdisciplinarity into the research process by providing health system
training to computer programmers – a key factor in human-centered artificial intelligence development. The developmental
evaluation changed development sprint processes, which paved the way to shorter sprints with quick internal evaluation of
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
ongoing progress.
(JMIR Preprints 24/01/2024:53339)
DOI: https://doi.org/10.2196/preprints.53339
Preprint Settings
1) Would you like to publish your submitted manuscript as preprint?
Please make my preprint PDF available to anyone at any time (recommended).
Please make my preprint PDF available only to logged-in users; I understand that my title and abstract will remain visible to all users.
Only make the preprint title and abstract visible.
No, I do not wish to publish my submitted manuscript as a preprint.
2) If accepted for publication in a JMIR journal, would you like the PDF to be visible to the public?
Yes, please make my accepted manuscript PDF available to anyone at any time (Recommended).
Yes, but please make my accepted manuscript PDF available only to logged-in users; I understand that the title and abstract will remain visible to all users (see Important note, above). I also understand that if I later pay to participate in <a href="https://jmir.zendesk.com/hc/en-us/articles/360008899632-What-is-the-PubMed-Now-ahead-of-print-option-when-I-pay-the-APF-" target="_blank">JMIR’s PubMed Now! service</a> service, my accepted manuscript PDF will automatically be made openly available.
Yes, but only make the title and abstract visible (see Important note, above). I understand that if I later pay to participate in <a href="https://jmir.zendesk.com/hc/en-us/articles/360008899632-What-is-the-PubMed-Now-ahead-of-print-option-when-I-pay-the-APF-" target="_blank">JMIR’s PubMed Now! service</a> service, my accepted manuscript PDF will automatically be made openly available.
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Original Manuscript
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Title: Evaluation of digital health platform development: Application of an innovative methodology
to build infrastructure for digital transformation of health systems
Authors: M. Claire Buchan1, Tarun Reddy Katapally2,3,4, Jasmin Bhawra5*
Affiliations:
1School of Public Health Sciences, University of Waterloo, Waterloo, ON, Canada
2DEPtH Lab, Faculty of Health Sciences, Western University, London, ON, Canada
3Department of Epidemiology and Biostatistics, Schulich School of Medicine and Dentistry, Western
University, London, ON, Canada
4Lawson Health Research Institute, London, ON, Canada
5CHANGE Research Lab, School of Occupational and Public Health, Toronto Metropolitan
University, Toronto, ON, Canada
*Corresponding author: Jasmin Bhawra
School of Occupational and Public Health
Toronto Metropolitan University
350 Victoria St., Toronto, Ontario
Canada, M5B 2K3
Email: jasmin.bhawra@torontomu.ca
ABSTRACT
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Background: The current public health crises we face, ranging from communicable disease
pandemics such as Coronavirus disease (COVID-19) to endemic chronic diseases, require cohesive,
collective, and deliberate societal efforts to address inherent decision-making gaps in our health
systems. Digital health platforms that leverage big data ethically from citizens can transform health
systems by enabling real-time data collection, communication, as well as precision prediction and
health system rapid responses. However, the lack of standardized and evidence-based methods to
develop and implement digital health platforms currently limits their application.
Objective: This study aimed to evaluate the development of a novel rapid response COVID-19
digital health platform by engaging with the development team which includes computer
programmers and data scientists, as well as the research team consisting of interdisciplinary
researchers (i.e., key stakeholders).
Methods: Using a developmental evaluation approach, this evaluation included two key
components: 1) A qualitative survey assessing digital health platform objectives, modifications, and
challenges administered to five key members of the software development team; and 2) A role-play
pilot with key stakeholders to simulate real-world conditions, followed by a self-report survey, to
evaluate the utility of the digital health platform for each of its objectives. Survey data were analyzed
using an inductive thematic analysis approach. Post-pilot test survey data were aggregated and
synthesized by participant role.
Results: The final digital health platform met original objectives, and was expanded to accommodate
evolving needs of potential users and COVID-19 regulations. Key challenges noted by the
development team included navigating changing government policies and restrictions, and
supporting the data sovereignty of platform users. Strong team cohesion, communication, and
problem solving were all quintessential in the overall success of program development. Pilot test
participants reported positive experiences interacting with the platform and found its features
relatively easy to use. Users in the community member role felt that the platform accurately reflected
their risk of contracting COVID-19, but reported challenges interacting with the interface,
particularly when submitting citizen reports and food status photos. Those in the decision-maker role
found the data visualizations intuitive in helping them to understand the information. Both
participant groups highlighted the utility of a tutorial for future users as there were some questions
regarding some of the features.
Conclusions: Evaluation of the digital health platform development process informed our decisions
to integrate the research team more cohesively with the development team, resulting in a data
scientist being part of both teams going forward. Another key development process decision was to
integrate more interdisciplinarity into the research process by providing health system training to
computer programmers – a key factor in human-centered artificial intelligence development. The
developmental evaluation changed development sprint processes, which paved the way to shorter
sprints with quick internal evaluation of ongoing progress.
INTRODUCTION
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
The current public health crises we face are global in nature, ranging from communicable disease
pandemics such as Coronavirus disease (COVID-19), to endemic non-communicable disease and
long-term care burden.1,2 The presence of multiple overlapping health issues, i.e., syndemics, is
further complicated in the age of polycrisis, as crisis response increasingly requires coordination
from multiple sectors, including but not limited to healthcare, environment, and social services.3,4 In
order to address the gaps in our current systems of care, deliberate and cohesive societal efforts are
required to understand and respond to existing inefficiencies in health systems.
Digital health platforms, which range from mobile health applications (apps) and virtual care
products to digital health dashboards,5,6 have immense potential to transform our health systems by
increasing citizen/patient access to care, predicting symptoms and outcomes, and enabling rapid
responses to health crises.4,6–9 For instance, digital health platforms can enable patients to connect
with their healthcare providers remotely10,11 and in turn, healthcare providers are able to predict risks
by ethically leveraging big data from patients12,13 and provide support by engaging with citizens and
patients remotely.10,14 More importantly, the application of digital health platforms is not limited to
patients, i.e., they can be used by apparently healthy individuals to self-monitor and track their health
behaviours and outcomes,15,16 as well as share their data ethically with healthcare providers and
scientists.13,17
Given the ubiquity of digital devices,18–20 their adaptability, and reach across geographic regions and
sociodemographic groups, digital health platforms are capable of bridging existing gaps in health
information and care access.6,21,22 More importantly, such platforms, particularly with the
incorporation of artificial intelligence (AI) and machine learning, can enable precision prediction of
health outcomes,23,24 as well as rapid responses to help monitor, mitigate, and manage existing and
emerging health crises.6,25 While the development of digital health platforms has increased
significantly in the past decade, and in particular during the COVID-19 pandemic,26 there are no
standardized processes for development and evaluation to ensure evidence-based approaches are
followed by utilizing interdisciplinary expertise.
As the role of digital health platforms in managing public health and promoting healthcare access is
predicted to grow exponentially,27 it is critical that the development of these platforms is evaluated
using rigorous methods to ensure effectiveness and efficacy. The World Health Organization recently
released a guide for evaluating digital health interventions,28 and many other groups have adapted
methods to assess digital platform technology.29–31 However, these guidelines do not provide
direction to evaluate digital platform development processes, including prototype development
sprints and troubleshooting28 – key steps that come well ahead of actual implementation of digital
interventions.
Another key challenge is in ensuring data privacy and data sovereignty of digital health platforms –
particularly when serving communities that have been historically disenfranchised or discriminated
against. Data sovereignty refers to meaningful control or ownership of one’s own data and is a
critical aspect of self-governance and self-determination, particularly among Indigenous and other
colonized communities.32,33 In creating digital health platforms which serve these communities, the
development process thus requires integration of rigid privacy and data protocol protocols, in
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
addition to consideration of culturally-appropriate features tailored to communities’ specific
needs.4,7,32,34 Moreover, as technology continues to evolve, it is crucial to evaluate the development
and use of AI, especially to ensure that AI is designed in a way that is truly human-centered.35,36
Evaluating development processes helps to identify and mitigate the risks associated with AI, which
could include neglecting user needs, overlooking the social impact, and failing to address biases and
accountability issues.37 By systematically evaluating the development processes, we can foster AI
systems that are better aligned with human values and needs (i.e., health services).36
Overall, evaluation of digital health platform development has enormous consequences for eventual
citizen and patient health and wellbeing, as well as data safety, security, and data sovereignty.38 The
potential for causing potential harm to populations, the sensitivity of personally-identifiable big data
that are collected via digital health platforms, as well as the need to develop human-centered AI,
requires a rigorous evaluation process including internal pilot testing before public testing. Most
importantly, a significant gap exists in terms of lack of peer-reviewed literature that describes the
approaches to development of evidence-based digital health platforms, particularly in academic
settings, where there is a critical need for co-design39 not only from a citizen/patient perspective, but
also by cohesive engagement between research and development teams. To address this critical gap
in digital health platform development, this study aimed to evaluate the development of a novel
digital health platform that was exclusively developed by a research and development team working
remotely during the pandemic to manage, monitor, and mitigate household risk of COVID-19.
METHODS
CO-Away digital health platform
In an effort to address the imminent public health crisis of COVID-19, the CO-Away digital health
platform was developed to track, manage, and mitigate household risk. During the pandemic, rural,
remote, and northern communities in Canada were disproportionately impacted and experienced
challenges with health information and care access.40 In the Canadian context, Indigenous
communities reported gaps in access to health information and care,34,41 thus after conducting a
comprehensive needs assessment,34 the CO-Away platform was developed to enable near real-time
monitoring, as well as rapid response. CO-Away comprises a progressive web application (PWA) for
users to manage household COVID-19 risk, as well as a backend digital decision-making dashboard
which visualizes aggregated and anonymized big data relayed in real-time from the PWA.6 PWAs are
a type of web app that leverage web technologies to provide a more app-like user experience,
combining the best features of web and mobile apps.42 Figure 1 shows screenshots of the CO-Away
home page and dashboard. The CO-Away platform provides local jurisdictional decision-makers
access to aggregate-level data to track and respond to emerging risk patterns and trends in near real-
time.
Figure 1: The CO-Away platform
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Given the wide-ranging impacts of COVID-19 on other aspects of health and wellness, CO-Away
takes a holistic approach which extends beyond COVID-19 symptom assessment to include other
key digital platform features: 1) food security request, which helps monitor and manage food
shortages within a jurisdiction; and 2) citizen reporter, which reports any public service or access
issues experienced by community members.6 CO-Away serves as a link to connect citizens with
decision-makers and offers value to both households and decision-makers. This digital health
platform, driven by citizen-generated big data collected from ubiquitous tools, not only supports
citizens by providing them with real-time support and valuable insights to enhance their decision-
making abilities, but also aggregates and anonymizes citizen data. These aggregated, anonymized
data are then delivered to decision-makers, enabling real-time exchanges of information and alerts
through direct bi-directional engagement between citizens and decision-makers. This innovative
approach represents a paradigm shift in the way community health is approached, placing a priority
on addressing the immediate needs of citizens. This study evaluates the development processes of
both the app that citizens/community members use as well as the digital health dashboard that
decision-makers use.
Evaluation Approach
To examine the evolution, challenges, successes, and utility of the CO-Away platform, a
developmental evaluation was performed after completion of the first prototype to inform subsequent
iterations of the platform. Developmental evaluations generate learnings to inform the development
of an initiative, thus are often used in complex and unpredictable scenarios such as the COVID-19
pandemic.43,44 A developmental evaluation is utilization-focused and should be designed and
implemented in ways that maximize utility for the primary intended users.45 This evaluation approach
generates data and findings in near real-time, thereby facilitating developmental decision-making and
course corrections throughout the development process.43,46
Evaluation Questions
This evaluation was guided by three overarching evaluation questions:
1. What factors influenced CO-Away digital health platform development?
2. How has the CO-Away digital platform development evolved over the course of the project?
3. a) To what extent does the CO-Away digital health platform achieve its goals and objectives
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
from the perspective of 1) community members/ users, and 2) decision makers?
b) How do these findings translate to the development process?
4. What direction will the CO-Away project take going forward?
Evaluation Design
The evaluation included two key components: 1) A qualitative survey assessing program objectives,
modifications, and challenges administered to five key members of the software development team;
and 2) A role-playing pilot test conducted with key stakeholders who were part of the research team,
followed by a self-report survey, to evaluate the utility of the program for each of its
goals/objectives. Ethics approval was not required for this project as it was conducted internally
within the research and development teams, and did not ask team members to provide any personal
data.
Data Collection
Data collection for the CO-Away developmental evaluation included two primary components: 1)
Development team survey (see Appendix A for survey), 2) Research team pilot test.
Part A: Development team survey
Five members of the CO-Away software development team were asked to complete an online self-
report survey. The survey was designed to assess the evolution of program objectives, target
audience, and key modifications made throughout the development process. These key team
members were identified by the principal investigator and the evaluation team, and included software
developers and data scientists Participants were all involved in the development of CO-Away and
project organization/management. Due to the potential for bias that could arise from having key
members of the development team involved in the data collection process, participants were not
involved in the preparation of the evaluation survey, and were blinded to the responses of their
fellow interviewees.
Participants were asked to complete the survey within a one week period from July 18th to 25th,
2022. The survey was designed to be completed in under 60 minutes and was administered through
Qualtrics.47 Participants were asked to describe their role in the digital platform design, to describe
any major deviations from the initial project proposal, any changes in the target audience, and the
digital platform’s potential for impact. Participants were also asked to reflect on the barriers and
facilitators to developing the CO-Away platform within the context of the COVID-19 pandemic.
Part B: Pilot test
To evaluate the utility of the CO-Away platform for each of its objectives, a role-playing pilot test
was conducted with key stakeholders who were part of the interdisciplinary research team, who had
expertise in digital health, epidemiology, and public health. Participants were given character
descriptions that they used to create avatars within the CO-Away app. These characters consisted of
six community members, two decision-makers, and one participant holding two concomitant roles.
The evaluation team developed the characters to simulate a range of potential user backgrounds, and
provided each participant with a brief overview of their character detailing their age, household
composition, occupation, remote work ability, vaccination status, and food security status (Figure 2).
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Participants were advised to use this profile as a starting point and to interact with the platform as
they saw fit to simulate real-world conditions. The pilot test was conducted over a two-week period
from July 8th to July 21st, 2022. Over the course of the activity, participants were sent daily
reminders via email to input their data into the app using the integrated features.
Figure 2: Pilot test character overview
Following the pilot test, participants were asked to complete a brief self-report survey detailing their
experience using the app. The survey assessed general usability of the app and specific app features,
including COVID-19 risk, food security, and citizen reporter features. Two surveys were developed
by the evaluation team, one for community member participants (Appendix B) and one for the
decision-maker participants (Appendix C). The survey administered was dependent on the
participant’s assumed role in the study (i.e., community members or decision-makers), while some
general questions were asked of both groups, the majority of questions differed between groups. The
one participant that assumed both a community member and a decision-maker role was asked to
complete both surveys. The survey was designed to be completed in 30 minutes and was distributed
electronically using Qualtrics.47 A copy of the anonymized surveys can be found in Appendix B
(community member) and Appendix C (decision-maker).
By accessing the cloud-based database,6 Daily user engagement reports were generated over the
course of the pilot test to examine user engagement across the features of the platform. Reports
included data on user engagement (e.g., number of new sign-ups, total log-ins), platform usability
(e.g., number of households reporting their behaviours, number of food requests, number of food
requests processed and delivered), and reported issues (e.g., number of reported issues using the
platform).
Analysis
Data from the development team survey were downloaded from Qualtrics and synthesized in
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Microsoft Word. Data were analyzed using an inductive thematic analysis approach as outlined in
Braun and Clarke (2006).48 To generate an initial coding manual, one evaluation team member
(MCB) independently reviewed the first survey and organized the data into themes and meaningful
codes. A second evaluation team member (JB) subsequently reviewed the coded transcript to ensure
consistency and agreement. Once all transcripts were coded by one evaluation team member (MCB),
the evaluation team (MCB, JB) reconvened to discuss relationships between codes that could be
grouped into themes.
Data from the post-pilot test survey were downloaded from Qualtrics and aggregated in Microsoft
Excel. Data were analyzed and summarized for each question and results were grouped by
community members and decision-makers. Results tables were sorted according to question topic
areas (e.g., COVID-19, food security, citizen reporter). Quotations in results tables indicate
participant responses to open text fields on the survey. For questions where participants could select
more than one option, responses were totalled across categories, and as such percentages may add to
greater than 100.
RESULTS
Part A: Development team survey
All five members of the software development team completed the survey. The roles of the
participating team members included the principal investigator, three software developers – one of
whom was involved in project organization and management – and one project coordinator. The
following five subsections describe the key themes which arose through analysis of the development
survey.
Theme 1: Alignment with original objectives
Over the course of the development development process, several modifications to the app structure
were made; many of which were made to accommodate the evolving nature of the COVID-19
pandemic. These changes included modifying existing features of the app (i.e., the COVID-19
feature), as well as the addition of new features (e.g., food security feature, citizen reporter). Despite
these modifications, the overall objective of CO-Away remained consistent throughout development.
One participant noted that:
“The prototype satisfies the originally planned platform's mandate. The changes
that took place were more logistical, which were beyond the scope of the
development team. The critical takeaway is that the concept and the ultimate
impact remain the same.” (DT5)
Additionally, the team noted improvements in the visual appearance of the app. Overall, the
development team indicated that the final app not only met original objectives, but the plan expanded
to accommodate evolving needs of the community and COVID-19 regulations.
Theme 2: Project challenges
There were a series of challenges identified by the development team, including tailoring the
platform to a diverse audience (i.e., both young and older community members), learning new
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
software development tools (e.g., programming languages), and the strict regulations surrounding
COVID-related platforms on mobile app stores. The CO-Away app required integrating various
disciplines to address community health needs, which was described as a unique challenge:
“As this project’s goal is to evaluate a COVID-19 mitigating digital health
platform for Indigenous self-governance, determination, and data sovereignty in a
remote [Indigenous] community […], the initiative from its inception has been
complex not only due to the integration of multiple disciplines (epidemiology, data
science, Indigenous health, public health, among others), but also due to the
ultimate purpose of the digital platform: rapidly responding to community needs.”
(DT5)
A digital platform developed for a remote Indigenous community app would have a substantially
greater chance for success if it took a holistic approach, reiterating Traditional Indigenous
Knowledge of holistic health. This approach required continuous collaboration with the Citizen
Scientist Advisory Council,6,34 and part of this process required developing strategies and features
within the app that manage and mitigate other population health crises that were worsened by
COVID-19 (e.g., food security, mental health and substance misuse, and negative interactions of
youth with law enforcement).
One of the key challenges that was pertinent for COVID-19 app development, in particular, was
constantly evolving government policies. Participants reported that several previously existing
features required repeated modifications, and features not initially outlined in the project plan
required development on the go. For example, one participant commented that in order to calculate
risk of COVID-19, this required incorporating changing guidance on vaccination dosage:
“...changes with increase in the number of doses of vaccination.” (DT4)
The nature of COVID-19 not only required the software development team to adapt to changing
government policies, but the evolution of the virus variants themselves. Another participant
highlighted the addition of the vaccine passport feature to the app that was not within the original
project scope.
While the ability for team members to work remotely was noted as a facilitator to project execution,
participants underscored the delays that resulted from the constant need to update program features
to align with government policies. Of high importance were the challenges associated with
supporting data sovereignty of the platform users. One participant commented:
“We have had to go above and beyond existing data safety regulations in the
world to build a digital health platform that provides Indigenous citizens control
and ownership over their data.” (DT5)
Despite the plethora of challenges posed by COVID-19, the software development team was able to
adapt their approach and practices to build a successful app.
Theme 3: Factors influencing app development
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
There were a range of factors which influenced app development, particularly for an app focused on
addressing the rapidly changing COVID-19 virus during the COVID-19 pandemic. In particular,
participants noted that working during the COVID-19 pandemic posed specific barriers to
recruitment of developers. Moreover, navigating the development process dominated by, and
dependent on big technology companies (BigTech), as well as a shift to a new academic institution
for the principal investigator, were key external factors which influenced overall app development.
Recruitment challenges and a change in academic institution created logistical challenges that added
to the complexity of this large-scale project:
“Due to the restriction of movement during the pandemic, we faced significant
difficulties to recruit high-quality personnel. This issue is especially challenging
in the disciplines of computer science and data science, where most of the
personnel who fit our criteria are recent international graduates, who face IRCC
[Immigration, Refugees and Citizenship Canada] backlogs for work permits.”
(DT5)
One challenge noted by several participants was launching a platform in Big Tech app stores, which
ultimately resulted in the creation of a PWA. For example, one participant stated:
“To truly ensure self-determination and data sovereignty, we had to abandon
launching our digital health platform on Apple Store and Google PlayStore to
eliminate the control of these stores over the development process. This decision
meant the development of a progressive web-based platform that does not need
these stores to be launched and the platform would only be provided to the
community members via a password protected process.” (DT5)
The two primary app hosting/launching BigTech companies, Apple Store and Google Play Store
implemented strict regulations around launching COVID-related platforms on their mobile app
stores, which hindered the development process. While this transition created delays and roadblocks
that the development team needed to overcome, it was essential in honouring the team’s commitment
to data sovereignty and self-determination.
Overall, the development team indicated that strong communication, workload distribution and
delegation, and continuous evaluation were key internal factors that positively contributed to the
platform development.
“The communication was really strong between the team despite working from
home and not working physically in the same place” (DT2)
Participants also noted that having an agile development mindset as a team was helpful, particularly
when collaborating with a Citizen Scientist Advisory Council consisting of project stakeholders.
Digital literacy was also identified as an important component of app usability by the development
team. It was therefore important for the development team to increase the accessibility of the app by
modifying traditional surveys (i.e., socio-demographic questions) to include more visuals and
graphics while minimizing text where possible (see Figure 1).
Theme 4: Project successes and areas of improvement
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Participants noted three primary moments of success: 1) the development of the household risk
management feature, 2) the launch of the PWA, and 3) the clearance of the final software tests. One
participant noted in particular that:
“The constant variation of policies and virus strains resulted in the creation of
household risk management, rather than individual risk management, which I
think is going to change the way we develop risk management tools for infectious
diseases in the future.” (DT5)
When asked which factors stood out in making the development process successful, participants
underscored the importance of the team dynamic. In particular, participants noted the strong
leadership, cohesion, ability to problem solve, and their commitment and dedication.
Participants indicated that improving the efficiency of the team overall could be beneficial moving
forward:
“Work on developing a more efficient workload distribution model” (DT4)
particularly through simplifying platform features, or through implementing agile49 methodology
(i.e., managing a project by breaking it up into several phases). As described in theme 3, the
development team experienced challenges launching health-focused apps, particularly restrictions for
COVID-19 apps, thus the team had to shift to a PWA. One participant noted that if they were to re-
start the project today, they would start with a PWA as it would have streamlined project
development quite substantially.
Theme 5: App launch requirements
The participants identified a range of factors that would enable a successful launch. The majority of
these factors focused on successful uptake hence the responses described the importance of
awareness and education. When asked to describe some of factors that would contribute to a
successful app launch, participants noted the platform’s adaptability (i.e., “rapid adaptation to
changing scenarios”), consistent team engagement, and project planning. Strong community
leadership and participation was highlighted as an important contributor to the overall success of the
platform. One participant noted that
“the most important factor that I think will contribute to a successful app launch
is education and awareness [of the community in which the app is launched].”
(DT3)
Taken together, these results suggest that despite the many challenges faced by the development
team, their strong team cohesion, communication, and problem solving were all quintessential in the
overall success of program development.
Part B: Pilot Test
Six stakeholders of the interdisciplinary research team participated in the pilot test playing the role of
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
citizens/community members and decision-maker to simulate real-world conditions.. Participant
character profiles included a diverse range of ages, occupations, household compositions, COVID-19
risk factors, and food security statuses (Figure 2).
Over the course of the two-week pilot test, there were an average of 7 log-ins per day, with the
largest proportion (40%) of these log-ins occurring in the evening (Figure 3). Five participants
reported COVID-19 diagnoses, with a total of 9 symptoms. Four food security requests were placed
over the two week period, all of which were processed by decision-makers using the digital health
dashboard. One participant submitted a citizen report, which was processed by a decision-maker. A
total of 5 community alerts were issued, all of which were categorized as high urgency.
Figure 3: Pilot test platform engagement
Community Members
Usability:
All six participants felt that their avatars accurately reflected their COVID-19 risk, found the consent
process clear, and appreciated the anonymity it provided (Figure 1B). All participants reported that
the app was easy to navigate and that they could find each feature when needed. Participants found it
straightforward to set up their avatars, taking five minutes or less. Two participants created avatars
for multiple household members, while the other four created avatars for themselves only. According
to participants, the ideal frequency of notifications from the app was once per day. No participants
reported any issues in using the app.
Feature-specific feedback:
The majority (4/6) of participants reported that they were comfortable interacting with the platform’s
COVID-19 feature, with the two participants reporting that they were neither comfortable nor
uncomfortable. All participants felt that the recommendations for their COVID-19 risk were clear
and easy to understand, though some provided feedback to increase convenience and subsequent
usability (e.g., readily accessible links or saved login information). Nearly all participants (5/6) were
comfortable sharing their vaccination status. Issues like random logouts and confusion with report
creation were raised, though the feature was still found to be easy and clear to use.
The majority (5/6) of participants used the food security feature, all of which found it ‘easy’ to use.
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Three participants expressed comfort with using this feature with the two remaining participants
reporting that they were ‘neither comfortable nor uncomfortable.’ Three participants reported that the
response of the decision-makers were ‘neither appropriate nor inappropriate,’ while the remaining
participants reported ‘appropriate’ responses. Similarly, half of participants were ‘comfortable’
revealing their identity when support was needed, while the remainder felt ‘neither comfortable nor
uncomfortable.’ When asked about room for improvement, one participant suggested the inclusion of
more detailed options for food security requests, while another desired additional instructions for
interacting with the feature.
Out of the six participants, three used the citizen reporter feature and felt comfortable doing so. Two
found it easy to use and one experienced difficulty during the report submission process, expressing
uncertainty regarding the completeness of their submission due to the lack of response. One
participant reported an appropriate response from the decision-maker, while the remaining two
participants reported neither appropriate nor inappropriate responses. Two of the three participants
felt comfortable revealing their identity when seeking support, while the third participant felt neither
comfortable nor uncomfortable.
Overall feedback
Participants generally found the app useful, providing clear information and serving as a good
resource. The straightforwardness, accessibility, and organization of the app was highlighted, as well
as the easy navigation and relevant tips. However, there were mentions of difficulty finding where to
input information initially and occasional login issues requiring password resets. Some participants
also mentioned challenges when using the app on a phone, particularly related to limited screen
visibility.
Decision-makers
Usability:
Decision-makers reported on the usability of the digital dashboard, including ease of finding
information, navigation, and overall use. Both decision-makers found the information used to create
their avatars accurate in reflecting their COVID-19 risk. The two decision-makers were satisfied with
the clarity of the consent process and believed the app was clear and easy to navigate, ensuring
anonymity and accessibility to all features. No issues or improvement suggestions were reported
regarding navigability, and participants felt that they knew who to contact with questions regarding
the app, their data, or their rights. Both decision-makers had no difficulty interacting with
notifications, finding them organized and user-friendly.
Feature-specific feedback:
Figure 4 shows examples of how community member data were aggregated and visualized for
decision-makers to review. Participants had a positive experience with the dashboard data
visualizations of the dashboard, finding them easy to understand and appreciating the level of control
they had over them. Decision-makers felt confident in using the presented data to make informed
decisions regarding the community's response to COVID-19. Response rates to individual incidents
were relatively short, typically taking less than five minutes to complete. No suggestions for
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
improvement or increasing participants' confidence were provided.
Figure 4: Co-Away decision-maker dashboard
Similar to the COVID-19 feature, both decision-makers found the data visualizations easy to
understand and appreciated the appearance and level of control over them. Decision-makers
expressed confidence in using the presented data for informed decision-making regarding COVID-19
response. Response rates for individual incidents were deemed prompt, with completion times under
five minutes. No suggestions for improving confidence or the visualizations were given.
Only one of the two decision-makers interacted with the citizen reporter feature and reported that the
data visualizations were easy to understand, appreciated their appearance, and level of control. The
decision-maker expressed confidence in using the presented data for informed decision-making and
the participant considered the response times for individual incidents, completed in under five
minutes, to be appropriate. No suggestions for improving the visualizations or participant confidence
were given.
Overall feedback
Participants had a positive overall experience with the platform, with one suggesting the addition of
reference material, such as a brief presentation or a short video, for future use. The food security
section was found to be easy to navigate and straightforward, providing users with simplified access.
However, there were uncertainties regarding the ability to delete requests and whether citizens could
contact local decision-makers and/or food services for questions or concerns, as one participant did
not receive a notification when attempting to do so.
DISCUSSION
Evaluation of the CO-Away digital health platform enabled systematic capture of the development
team’s perspective on the overall development process, as well as testing of platform usability and
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
functionality from multiple stakeholders simulating real-world scenarios before finalizing platform
features. The developmental evaluation findings will not only influence decisions of both research
and development teams going forward, but the process described for conducting an assessment of the
CO-Away development phase also importantly advances a replicable methodology that can inform
empirical development, implementation and evaluation of digital health platforms.6 In essence, the
evaluation documented the needs and challenges of both the research and development teams, which
are integral to the development of digital health platforms.
Overall, evaluation of the digital health platform development informed our decisions to integrate the
research team more cohesively with the development team, which resulted in a data scientist being
part of both teams going forward. Including a common member in both teams improved
communication and coordination during the development process. By bridging these traditionally
distinct teams, we have been able to maximize cross-disciplinary collaboration, with research and
practice informing development decisions. This integration also proved instrumental in aligning the
platform's features with users' needs and developmental feasibility, i.e., integrating interdisciplinarity
into the development process.
In recognizing the potential of concurrent health domain expertise and software development skills,
we initiated a health systems training for all computer programmers.50,51 This training contributed to
increasing interdisciplinary understanding and communication, and was deemed necessary for the
creation of user-centric CO-Away feature development which considered distinct risks that could be
introduced with AI systems.36,37 This training not only supports the programmers' comprehension of
the health context, but also provides them with the ability to envision the platform's functionalities
from the user's perspective – a key step in both co-designing digital infrastructure as well as
designing human-centered AI35,52 – which is a gap in the current literature.53–56 In general, this
approach resulted in digital health platform features that were user-friendly, i.e., an improved user
experience (i.e., UX), which is critical to the success of platform usage and implementation.54,57
The developmental evaluation also changed development sprint processes,49,58 which paved the way
to shorter sprints with quick internal evaluation of ongoing progress. Software development sprints,
which are part of the agile coding process, are implemented as part of the agile methodology, an
iterative project management framework that breaks projects down into several dynamic phases,
commonly known as “sprints.”49 When the team started the CO-Away development process, sprints
were approximately 4-5 months, but the developmental evaluation findings informed our decision to
shorten the sprints to 3-4 months to enable faster internal testing and modification of the digital
platform. This decision has allowed the software development team to pivot quickly when necessary,
responding to stakeholder feedback and incorporating changes to the digital platform iteratively.
The evaluation provided noteworthy insights into the dynamics of our team, particularly the role that
strong communication played throughout the development process. The pandemic presented a unique
set of challenges that required adaptive strategies.59–61 Despite the physical distancing mandates, our
team managed to function cohesively, leveraging digital tools, regular virtual meetings, and
transparent channels of communication, i.e., open discussion of weekly updates and key decisions
accessible to all team members on a virtual portal. The development process not only involved
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
remote hiring of our team,62 but also remote functioning,63 and retention of our team even after the
pandemic restrictions ended – lessons which we implemented into the continuous functioning of our
development processes. For instance, we currently work in a hybrid setting,64 which leverages the
flexibility of virtual work as well as in-person brainstorming, a system that has further improved
communication and collaboration with research and development teams.
Perhaps the defining element that shaped the development process of the CO-Away platform were
the complexities that arose with its COVID-19-specific focus. The evolving nature of the COVID-19
virus,65–67 coupled with the rapidly changing landscape of pandemic management policies,68–70
presented a constant need for flexibility and adaptability. The virus's mutations necessitated
continuous updates to the platform's algorithms to ensure that users would have access to the most
up-to-date information. Similarly, the ever-changing policies surrounding lockdowns,71,72 vaccine
requirements,73,74 and mask mandates75,76 required the platform's features to be responsive to shifting
guidelines.6 Navigating these complexities highlighted the importance of further adapting the agile
methodology49,58 throughout the platform design and development process to accommodate the
changing COVID landscape.
The role-play pilot test highlighted the effectiveness of adhering to user-centered design principles
throughout the development of the CO-Away digital health platform. The stakeholders who
participated in the pilot reported overall positive reactions to the platform avatars, transparent
consent procedures, and intuitive navigation - aspects that reiterated the importance of tailoring the
platform design to users' needs and ensuring a user-friendly experience.54,57 While participants
generally found the app user-friendly, their feedback on minor obstacles such as entering initial
information and sporadic login difficulties highlight the importance of iterative improvement during
the development process. This constructive feedback provided an opportunity for the development
team to adapt CO-Away to meet the needs of users.
The mixed feedback regarding participants’ comfort levels when interacting with the platform
features reveals the importance of balancing interactivity with user comfort. For instance, while
some participants were comfortable engaging with various features and sharing personal information
for support, others expressed hesitancy. This finding highlights the significance of providing users
with control over their data, a critical aspect of data sovereignty32 that has been built into platform
development after this vital input.
While we did not observe marked differences in the way participants engaged with the platform
based on the assigned participant roles, it is worth noting that there were differences in platform
utilization between decision-makers and community members. These variations align with
expectations, stemming from the differing ways these distinct user groups engage with the platform's
features. For instance, decision-makers are less focused on their own privacy as they are not sharing
data, while the community members are sharing data. We anticipate that various community
members will interact with and utilize the app according to their specific circumstances and needs.
As such, the potential differences in app utilization based on community member roles will be
further explored in the future during the external community pilot phase.
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Finally, the digital health dashboard feature of CO-Away offered decision-makers valuable tools for
informed decision-making, especially within the context of COVID-19 response and community
management. The pilot decision-makers appreciated the usability of the platform and expressed
confidence in using the data visualizations for informed decision-making. However, it's important to
recognize that although some real-world scenarios were simulated in the pilot, a larger external
community pilot will require more complex decision making due to the increase in scale of
implementation. However, the key goal of the pilot was to test the functionality of a cloud-based
digital health dashboard to engage with community members and send real-time alerts - features that
were successfully tested and functionality that was confirmed in the evaluation of the pilot. Real-
world policy changes using real-time big data would require stakeholder buy-in and addressing
nuanced matters such as food security and citizen reporter reports, demanding sensitivity and
community-wide consideration. Further exploration of these dynamics will be carried out during our
upcoming external community pilot.
Key guidelines for digital health platform development
Based on the findings of this developmental evaluation of a digital health platform development, we
suggest the following guidelines:
1. Integration of research and development teams is key to the success of the digital health
platforms, where the software developers are provided opportunity to understand scientific
goals, and the research stakeholders are made privy to cloud computing challenges.
2. Conducting a developmental evaluation after completion of a prototype facilitates
incorporation of research and development team perspectives empirically into the iterative
digital development process before a product is tested widely in the community.
3. Conducting an internal role-play pilot simulating real-world conditions is critical to test not
only the functionality of digital health platforms, but also the nuanced perceptions of
potential users, which will be essential for successful community implementation.
Strengths and Limitations
This developmental evaluation has several notable strengths. First, we deployed a mixed methods
approach to data collection and analysis; open-ended survey questions to capture participant
perceptions of the platform and its usability, and CO-Away usage metrics data provided an objective
assessment of platform engagement. The integration of both quantitative and qualitative data
collection methods enriches the findings generated by the evaluation. Second, the inclusion of
development team surveys allowed for an in-depth understanding not only of CO-Away's
developmental process, but also offered valuable insights that have and will continue to inform
improvements in the development processes. Finally, the use of “character roles” were used to
simulate a community pilot test. Although not a substitute for a real-world pilot, this approach served
as a valuable first-step for exploring real-world simulations using varied user interactions and
experiences - an approach that will shape a future external community pilot. There were also several
study limitations. The participation of the stakeholders in the pilot test may have affected the nature
of the issues reported, i.e., more focus on app function than readability. In order to minimize
potential bias, the evaluation questions were blinded, and the digital platform development team did
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
not participate in the pilot test. Moreover, the stakeholders who participated in the pilot did not
analyze the evaluation data. The use of “character roles” – while insightful – does not capture all user
scenarios that would arise in a real-world setting. As such, it is possible that certain user interactions
and challenges were not captured in this evaluation, and will be explored during our external
community pilot. Notably, the sample size for each population was small, so caution should be taken
when interpreting or comparing percentages. Another limitation pertains to the digital literacy of
participants. While the pilot participants did not have prior experience using the platform, it is
possible that few issues were reported due to inherent high digital literacy of the interdisciplinary
stakeholders. In future digital evaluations, it would be imperative to not only capture the digital
literacy of platform users, but also to enable the increase of digital literacy through innovative digital
literacy programs.77
CONCLUSIONS
The evaluation of digital health platform development is critical for the success of not only platform
functionality, but eventual implementation and scale-up. The innovative approach applied in this
evaluation combines both perspectives of the development team as well as a real-world simulation of
platform users to extend key guidelines for digital health platform development. The evaluation
informed several key decisions of the digital health platform development, including integrating the
research team more cohesively with the development team, which resulted in a data scientist being
part of both teams going forward. Another key development process decision was to integrate more
interdisciplinarity into the development process by providing health system training to computer
programmers. Perhaps most importantly, the developmental evaluation changed our development
sprint processes, which paved the way to shorter sprints with quick internal evaluation of the
progress.
ACKNOWLEDGEMENTS
The authors would like to acknowledge Nadine Elsahli for her support in collating this manuscript,
including creation of figures to visually depict the digital health platform design. Thank you to all
members who participated in the developmental evaluation process and provided valuable feedback
throughout.
CONFLICTS OF INTEREST
The authors have no conflicts of interest to declare.
REFERENCES
1. Alokaily F. COVID-19 Global health crisis. Saudi Med J. 2021;42(1):3-4.
doi:10.15537/smj.2021.1.25663
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
2. Noncommunicable diseases. World Health Organization. Published September 16, 2023.
Accessed October 31, 2023.
https://www.who.int/news-room/fact-sheets/detail/noncommunicable-diseases
3. Lawrence M, Homer-Dixon T, Janzwood S, Rockstrom J, Renn O, Donges JF. Global polycrisis:
The causal mechanisms of crisis entanglement. SSRN Journal. Published online 2023.
doi:10.2139/ssrn.4483556
4. Katapally TR. A Global Digital Citizen Science Policy to Tackle Pandemics Like COVID-19. J
Med Internet Res. 2020;22(5):e19357. doi:10.2196/19357
5. Sheikh A, Anderson M, Albala S, et al. Health information technology and digital innovation for
national learning health and care systems. The Lancet Digital Health. 2021;3(6):e383-e396.
doi:10.1016/S2589-7500(21)00005-4
6. Katapally TR, Ibrahim ST. Digital Health Dashboards for Decision-Making to Enable Rapid
Responses During Public Health Crises: Replicable and Scalable Methodology. JMIR Res
Protoc. 2023;12:e46810. doi:10.2196/46810
7. Katapally TR. The SMART Framework: Integration of Citizen Science, Community-Based
Participatory Research, and Systems Science for Population Health Science in the Digital Age.
JMIR mHealth and uHealth. 2019;7(8):e14056. doi:10.2196/14056
8. Stoumpos AI, Kitsios F, Talias MA. Digital Transformation in Healthcare: Technology
Acceptance and Its Applications. Int J Environ Res Public Health. 2023;20(4):3407. doi:10.3390/
ijerph20043407
9. Budd J, Miller BS, Manning EM, et al. Digital technologies in the public-health response to
COVID-19. Nat Med. 2020;26(8):1183-1192. doi:10.1038/s41591-020-1011-4
10. Rosenlund M, Kinnunen UM, Saranto K. The Use of Digital Health Services Among Patients
and Citizens Living at Home: Scoping Review. J Med Internet Res. 2023;25:e44711.
doi:10.2196/44711
11. Bokolo Anthony Jnr. Use of Telemedicine and Virtual Care for Remote Treatment in Response to
COVID-19 Pandemic. J Med Syst. 2020;44(7):132. doi:10.1007/s10916-020-01596-5
12. Batko K, Ślęzak A. The use of Big Data Analytics in healthcare. J Big Data. 2022;9(1):3.
doi:10.1186/s40537-021-00553-4
13. Dash S, Shakyawar SK, Sharma M, Kaushik S. Big data in healthcare: management, analysis and
future prospects. Journal of Big Data. 2019;6(1):54. doi:10.1186/s40537-019-0217-0
14. Burton L, Rush KL, Smith MA, et al. Empowering Patients Through Virtual Care Delivery:
Qualitative Study With Micropractice Clinic Patients and Health Care Providers. JMIR Form
Res. 2022;6(4):e32528. doi:10.2196/32528
15. Kuwabara A, Su S, Krauss J. Utilizing Digital Health Technologies for Patient Education in
Lifestyle Medicine. Am J Lifestyle Med. 2019;14(2):137-142. doi:10.1177/1559827619892547
16. Mair JL, Salamanca-Sanabria A, Augsburger M, et al. Effective Behavior Change Techniques in
Digital Health Interventions for the Prevention or Management of Noncommunicable Diseases:
An Umbrella Review. Annals of Behavioral Medicine. 2023;57(10):817-835.
doi:10.1093/abm/kaad041
17. Riso B, Tupasela A, Vears DF, et al. Ethical sharing of health data in online platforms – which
values should be considered? Life Sci Soc Policy. 2017;13:12. doi:10.1186/s40504-017-0060-z
18. Mosa ASM, Yoo I, Sheets L. A Systematic Review of Healthcare Applications for Smartphones.
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
BMC Medical Informatics and Decision Making. 2012;12(1):67. doi:10.1186/1472-6947-12-67
19. Langford AT, Solid CA, Scott E, et al. Mobile Phone Ownership, Health Apps, and Tablet Use in
US Adults With a Self-Reported History of Hypertension: Cross-Sectional Study. JMIR Mhealth
Uhealth. 2019;7(1):e12228. doi:10.2196/12228
20. Kaplan WA. Can the ubiquitous power of mobile phones be used to improve health outcomes in
developing countries? Global Health. 2006;2:9. doi:10.1186/1744-8603-2-9
21. Willis VC, Craig KJT, Jabbarpour Y, et al. Digital Health Interventions to Enhance Prevention in
Primary Care: Scoping Review. JMIR Medical Informatics. 2022;10(1):e33518.
doi:10.2196/33518
22. Haleem A, Javaid M, Singh RP, Suman R. Telemedicine for healthcare: Capabilities, features,
barriers, and applications. Sens Int. 2021;2:100117. doi:10.1016/j.sintl.2021.100117
23. Shahid N, Rappon T, Berta W. Applications of artificial neural networks in health care
organizational decision-making: A scoping review. PLOS ONE. 2019;14(2):e0212356.
doi:10.1371/journal.pone.0212356
24. Vázquez-Ingelmo A, Garcia-Peñalvo FJ, Therón R. Information Dashboards and Tailoring
Capabilities - A Systematic Literature Review. IEEE Access. 2019;7:109673-109688.
doi:10.1109/ACCESS.2019.2933472
25. Kostkova P. Disease surveillance data sharing for public health: the next ethical frontiers. Life
Sci Soc Policy. 2018;14(1):16. doi:10.1186/s40504-018-0078-x
26. Bouabida K, Lebouché B, Pomey MP. Telehealth and COVID-19 Pandemic: An Overview of the
Telehealth Use, Advantages, Challenges, and Opportunities during COVID-19 Pandemic.
Healthcare (Basel). 2022;10(11):2293. doi:10.3390/healthcare10112293
27. Abernethy A, Adams L, Barrett M, et al. The Promise of Digital Health: Then, Now, and the
Future. NAM Perspect. 2022:10.31478/202206e. doi:10.31478/202206e
28. World Health Organization. Monitoring and Evaluating Digital Health Interventions: A
Practical Guide to Conducting Research and Assessment. World Health Organization; 2016.
Accessed October 31, 2023. https://iris.who.int/handle/10665/252183
29. Ippoliti N, Sekamana M, Baringer L, Hope R. Using Human-Centered Design to Develop,
Launch, and Evaluate a National Digital Health Platform to Improve Reproductive Health for
Rwandan Youth. Global Health: Science and Practice. 2021;9(Supplement 2):S244-S260.
doi:10.9745/GHSP-D-21-00220
30. Beynon F, Guérin F, Lampariello R, et al. Digitalizing Clinical Guidelines: Experiences in the
Development of Clinical Decision Support Algorithms for Management of Childhood Illness in
Resource-Constrained Settings. Glob Health Sci Pract. 2023;11(4):e2200439.
doi:10.9745/GHSP-D-22-00439
31. Tamrat T, Chandir S, Alland K, et al. Digitalization of routine health information systems:
Bangladesh, Indonesia, Pakistan. Bull World Health Organ. 2022;100(10):590-600. doi:10.2471/
BLT.22.287816
32. Bhawra J. Decolonizing Digital Citizen Science: Applying the Bridge Framework for Climate
Change Preparedness and Adaptation. Societies. 2022;12(2):71. doi:10.3390/soc12020071
33. Hummel P, Braun M, Tretter M, Dabrock P. Data sovereignty: A review. Big Data & Society.
2021;8(1). doi:10.1177/2053951720982012
34. Bhawra J, Buchan MC, Green B, Skinner K, Katapally TR. A guiding framework for needs
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
assessment evaluations to embed digital platforms in partnership with Indigenous communities.
PLOS ONE. 2022;17(12):e0279282. doi:10.1371/journal.pone.0279282
35. Bingley WJ, Curtis C, Lockey S, et al. Where is the human in human-centered AI? Insights from
developer priorities and user experiences. Computers in Human Behavior. 2023;141:107617.
doi:10.1016/j.chb.2022.107617
36. Ozmen Garibay O, Winslow B, Andolina S, et al. Six Human-Centered Artificial Intelligence
Grand Challenges. International Journal of Human–Computer Interaction. 2023;39(3):391-437.
doi:10.1080/10447318.2022.2153320
37. Chen Y, Clayton EW, Novak LL, Anders S, Malin B. Human-Centered Design to Address Biases
in Artificial Intelligence. J Med Internet Res. 2023;25:e43251. doi:10.2196/43251
38. Murray E, Hekler EB, Andersson G, et al. Evaluating digital health interventions: key questions
and approaches. Am J Prev Med. 2016;51(5):843-851. doi:10.1016/j.amepre.2016.06.008
39. Sanz MF, Acha BV, García MF. Co-Design for People-Centred Care Digital Solutions: A
Literature Review. Int J Integr Care. 21(2):16. doi:10.5334/ijic.5573
40. Mental Health Commission of Canada. The Impact of COVID-19 on Rural and Remote Mental
Health and Substance Use.; 2021. Accessed October 31, 2023.
https://mentalhealthcommission.ca/wp-content/uploads/2021/10/The-Impact-of-COVID-19-on-
Rural-and-Remote-Mental-Health-and-Substance-Use.pdf.pdf
41. Huyser KR, Yellow Horse AJ, Collins KA, et al. Understanding the Associations among Social
Vulnerabilities, Indigenous Peoples, and COVID-19 Cases within Canadian Health Regions. Int
J Environ Res Public Health. 2022;19(19):12409. doi:10.3390/ijerph191912409
42. Rêgo F, Portela F, Santos MF. Towards PWA in Healthcare. Procedia Computer Science.
2019;160:678-683. doi:10.1016/j.procs.2019.11.028
43. Dozois E, Langlois M, Blanchet-Cohen N. DE 201: A Practitioner’s Guide to Developmental
Evaluation. International Institute for Child Rights and Development; 2010.
44. Patton MQ. Emergent Developmental Evaluation Developments. Journal of MultiDisciplinary
Evaluation. 2021;17(41). doi:10.56645/jmde.v17i41.699
45. Patton MQ. Utilization-Focused Evaluation. Vol 31. Sage Publications; 2008.
46. Patton MQ. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation
and Use. Guilford Press; 2010.
47. Qualtrics XM - Experience Management Software. Qualtrics. Published 2022. Accessed October
31, 2023. https://www.qualtrics.com/
48. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology.
2006;3(2):77-101. doi:10.1191/1478088706qp063oa
49. Nazir S, Price B, Surendra NC, Kopp K. Adapting agile development practices for hyper-agile
environments: lessons learned from a COVID-19 emergency response research project. Inf
Technol Manag. 2022;23(3):193-211. doi:10.1007/s10799-022-00370-y
50. Moreb M, Mohammed TA, Bayat O. A Novel Software Engineering Approach Toward Using
Machine Learning for Improving the Efficiency of Health Systems. IEEE Access. 2020;8:23169-
23178. doi:10.1109/ACCESS.2020.2970178
51. Ibeneme S, Karamagi H, Muneene D, Goswami K, Chisaka N, Okeibunor J. Strengthening
Health Systems Using Innovative Digital Health Technologies in Africa. Front Digit Health.
2022;4:854339. doi:10.3389/fdgth.2022.854339
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
52. Mhlanga D. Human-Centered Artificial Intelligence: The Superlative Approach to Achieve
Sustainable Development Goals in the Fourth Industrial Revolution. Sustainability.
2022;14(13):7804. doi:10.3390/su14137804
53. Grindell C, Coates E, Croot L, O’Cathain A. The use of co-production, co-design and co-creation
to mobilise knowledge in the management of health conditions: a systematic review. BMC
Health Services Research. 2022;22(1):877. doi:10.1186/s12913-022-08079-y
54. Nusir M, Rekik M. Systematic review of co-design in digital health for COVID-19 research.
Univers Access Inf Soc. Published online December 31, 2022:1-15. doi:10.1007/s10209-022-
00964-x
55. Slattery P, Saeri AK, Bragge P. Research co-design in health: a rapid overview of reviews.
Health Research Policy and Systems. 2020;18(1):17. doi:10.1186/s12961-020-0528-9
56. Tremblay M, Hamel C, Viau-Guay A, Giroux D. User Experience of the Co-design Research
Approach in eHealth: Activity Analysis With the Course-of-Action Framework. JMIR Hum
Factors. 2022;9(3):e35577. doi:10.2196/35577
57. Madanian S, Nakarada-Kordic I, Reay S, Chetty T. Patients’ perspectives on digital health tools.
PEC Innovation. 2023;2:100171. doi:10.1016/j.pecinn.2023.100171
58. Kortum F, Klünder J, Brunotte W, Schneider K. Sprint Performance Forecasts in Agile Software
Development - The Effect of Futurespectives on Team-Driven Dynamics. In: ; 2019:94-101.
doi:10.18293/SEKE2019-224
59. Graziani AR, Botindari L, Menegatti M, Moscatelli S. Adaptive Coping Strategies at the Time of
COVID-19: The Role of Social and General Trust. Int J Environ Res Public Health.
2023;20(15):6512. doi:10.3390/ijerph20156512
60. Maison D, Jaworska D, Adamczyk D, Affeltowicz D. The challenges arising from the COVID-19
pandemic and the way people deal with them. A qualitative longitudinal study. PLoS One.
2021;16(10):e0258133. doi:10.1371/journal.pone.0258133
61. Fegert JM, Vitiello B, Plener PL, Clemens V. Challenges and burden of the Coronavirus 2019
(COVID-19) pandemic for child and adolescent mental health: a narrative review to highlight
clinical and research needs in the acute phase and the long return to normality. Child and
Adolescent Psychiatry and Mental Health. 2020;14(1):20. doi:10.1186/s13034-020-00329-3
62. Kowalski G, Ślebarska K. Remote Working and Work Effectiveness: A Leader Perspective. Int J
Environ Res Public Health. 2022;19(22):15326. doi:10.3390/ijerph192215326
63. Henke JB, Jones SK, O’Neill TA. Skills and abilities to thrive in remote work: What have we
learned. Front Psychol. 2022;13:893895. doi:10.3389/fpsyg.2022.893895
64. Krajčík M, Schmidt DA, Baráth M. Hybrid Work Model: An Approach to Work–Life Flexibility
in a Changing Environment. Administrative Sciences. 2023;13(6):150.
doi:10.3390/admsci13060150
65. Sharma A, Ahmad Farouk I, Lal SK. COVID-19: A Review on the Novel Coronavirus Disease
Evolution, Transmission, Detection, Control and Prevention. Viruses. 2021;13(2):202.
doi:10.3390/v13020202
66. Sun J, He WT, Wang L, et al. COVID-19: Epidemiology, Evolution, and Cross-Disciplinary
Perspectives. Trends in Molecular Medicine. 2020;26(5):483. doi:10.1016/j.molmed.2020.02.008
67. Talic S, Shah S, Wild H, et al. Effectiveness of public health measures in reducing the incidence
of covid-19, SARS-CoV-2 transmission, and covid-19 mortality: systematic review and meta-
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
analysis. BMJ. 2021;375:e068302. doi:10.1136/bmj-2021-068302
68. Cavalcante de Oliveira AP, Galante ML, Maia LS, et al. Implementation of policy and
management interventions to improve health and care workforce capacity to address the COVID-
19 pandemic response: a systematic review. Hum Resour Health. 2023;21:80.
doi:10.1186/s12960-023-00856-y
69. OECD. First Lessons from Government Evaluations of COVID-19 Responses: A Synthesis.;
2022. Accessed November 3, 2023. https://www.oecd.org/coronavirus/policy-responses/first-
lessons-from-government-evaluations-of-covid-19-responses-a-synthesis-483507d6/
70. Sigahi TFAC, Kawasaki BC, Bolis I, Morioka SN. A systematic review on the impacts of Covid‐
19 on work: Contributions and a path forward from the perspectives of ergonomics and
psychodynamics of work. Hum Factors Ergon Manuf. 2021;31(4):375-388.
doi:10.1002/hfm.20889
71. Yanovskiy M, Socol Y. Are Lockdowns Effective in Managing Pandemics? Int J Environ Res
Public Health. 2022;19(15):9295. doi:10.3390/ijerph19159295
72. Panchal U, Salazar de Pablo G, Franco M, et al. The impact of COVID-19 lockdown on child
and adolescent mental health: systematic review. Eur Child Adolesc Psychiatry.
2023;32(7):1151-1177. doi:10.1007/s00787-021-01856-w
73. Bardosh K, Figueiredo A de, Gur-Arie R, et al. The unintended consequences of COVID-19
vaccine policy: why mandates, passports and restrictions may cause more harm than good. BMJ
Global Health. 2022;7(5):e008684. doi:10.1136/bmjgh-2022-008684
74. Maquiling A, Jeevakanthan A, Ho Mi Fane B. The effect of vaccine mandate announcements on
vaccine uptake in Canada: An interrupted time series analysis. Vaccine. 2023;41(18):2932-2940.
doi:10.1016/j.vaccine.2023.03.040
75. Zhang YSD, Young Leslie H, Sharafaddin-zadeh Y, Noels K, Lou NM. Public Health Messages
About Face Masks Early in the COVID-19 Pandemic: Perceptions of and Impacts on Canadians.
J Community Health. 2021;46(5):903-912. doi:10.1007/s10900-021-00971-8
76. He L, He C, Reynolds TL, et al. Why do people oppose mask wearing? A comprehensive
analysis of U.S. tweets during the COVID-19 pandemic. J Am Med Inform Assoc.
2021;28(7):1564-1573. doi:10.1093/jamia/ocab047
77. Buchan M, Bhawra J, Katapally TR. Improving Digital Literacy in a Digital World:
Development of an Evidence-Based Digital Literacy Program and Assessment Tool to Evaluate
Youth Digital Literacy.; 2023. doi:10.31219/osf.io/aqvcu
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Supplementary Files
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Figures
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
The CO-Away platform.
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Pilot test character overview.
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Pilot test platform engagement.
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Co-Away decision-maker dashboard.
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Multimedia Appendixes
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
JMIR Preprints Buchan et al
Development survey.
URL: http://asset.jmir.pub/assets/5365e2125a9baa0510228505ce7a5668.docx
Post-pilot survey – community member.
URL: http://asset.jmir.pub/assets/223dabd9183be3eec3b50cbbb9662d3c.docx
Post-pilot survey – decision-maker.
URL: http://asset.jmir.pub/assets/d600cd476b365ae4b28a6591f109b7a1.docx
Powered by TCPDF (www.tcpdf.org)
https://preprints.jmir.org/preprint/53339 [unpublished, non-peer-reviewed preprint]
