BookPDF Available

Methods and applications in psychology for clinical settings

July 2023

July 2023

Publisher: Frontiers Media SA
ISBN: 978-2-8325-2850-1

Authors:

Carmen María Galvez Sánchez

University of Murcia

Casandra I Montoro

Universidad de Jaén

Tindara Caprì

Link Campus University

Content uploaded by Casandra I Montoro

Content may be subject to copyright.

Edited by

Carmen María Galvez-Sánchez, Casandra Isabel Montoro

and Tindara Caprì

Published in

Frontiers in Psychology

Frontiers in Public Health

Frontiers in Psychiatry

Methods and applications

in psychology for clinical

settings

July 2023

Frontiers in Psychology frontiersin.org1

About Frontiers

Frontiers is more than just an open access publisher of scholarly articles: it is

a pioneering approach to the world of academia, radically improving the way

scholarly research is managed. The grand vision of Frontiers is a world where

all people have an equal opportunity to seek, share and generate knowledge.

Frontiers provides immediate and permanent online open access to all its

publications, but this alone is not enough to realize our grand goals.

Frontiers journal series

The Frontiers journal series is a multi-tier and interdisciplinary set of open-

access, online journals, promising a paradigm shift from the current review,

selection and dissemination processes in academic publishing. All Frontiers

journals are driven by researchers for researchers; therefore, they constitute

a service to the scholarly community. At the same time, the Frontiers journal

series operates on a revolutionary invention, the tiered publishing system,

initially addressing specific communities of scholars, and gradually climbing

up to broader public understanding, thus serving the interests of the lay

society, too.

Dedication to quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely

collaborative interactions between authors and review editors, who include

some of the world’s best academicians. Research must be certified by peers

before entering a stream of knowledge that may eventually reach the public

- and shape society; therefore, Frontiers only applies the most rigorous

and unbiased reviews. Frontiers revolutionizes research publishing by freely

delivering the most outstanding research, evaluated with no bias from both

the academic and social point of view. By applying the most advanced

information technologies, Frontiers is catapulting scholarly publishing into

a new generation.

What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers

journals series: they are collections of at least ten articles, all centered

on a particular subject. With their unique mix of varied contributions from

Original Research to Review Articles, Frontiers Research Topics unify the

most influential researchers, the latest key findings and historical advances

in a hot research area.

Find out more on how to host your own Frontiers Research Topic or

contribute to one as an author by contacting the Frontiers editorial office:

frontiersin.org/about/contact

FRONTIERS EBOOK COPYRIGHT STATEMENT

The copyright in the text of individual

articles in this ebook is the property

of their respective authors or their

respective institutions or funders.

The copyright in graphics and images

within each article may be subject

to copyright of other parties. In both

cases this is subject to a license

granted to Frontiers.

The compilation of articles constituting

this ebook is the property of Frontiers.

Each article within this ebook, and the

ebook itself, are published under the

most recent version of the Creative

Commons CC-BY licence. The version

current at the date of publication of

this ebook is CC-BY 4.0. If the CC-BY

licence is updated, the licence granted

by Frontiers is automatically updated

to the new version.

When exercising any right under

the CC-BY licence, Frontiers must be

attributed as the original publisher

of the article or ebook, as applicable.

Authors have the responsibility of

ensuring that any graphics or other

materials which are the property of

others may be included in the CC-BY

licence, but this should be checked

before relying on the CC-BY licence

to reproduce those materials. Any

materials must be complied with.

acknowledgement notices may not

be removed and must be displayed

in any copy, derivative work or partial

copy which includes the elements

in question.

All copyright, and all rights therein,

are protected by national and

international copyright laws. The

above represents a summary only.

For further information please read

Frontiers’ Conditions for Website Use

and Copyright Statement, and the

applicable CC-BY licence.

ISSN 1664-8714

ISBN 978-2-8325-2850-1

DOI 10.3389/978-2-8325-2850-1

July 2023

Frontiers in Psychology 2frontiersin.org

Methods and applications in

psychology for clinical settings

Topic editors

Carmen María Galvez-Sánchez — University of Jaén, Spain

Casandra Isabel Montoro — University of Jaén, Spain

Tindara Caprì — Università Link Campus, Italy

Citation

Galvez-Sánchez, C. M., Montoro, C. I., Caprì, T., eds. (2023). Methods and

applications in psychology for clinical settings. Lausanne: Frontiers Media SA.

doi: 10.3389/978-2-8325-2850-1

July 2023

Frontiers in Psychology frontiersin.org3

05 Editorial: Methods and applications in Psychology for Clinical

Settings

Tindara Caprì, Casandra Isabel Montoro and

Carmen María Galvez-Sánchez

07 Psychometric Properties of the HADS Measure of Anxiety and

Depression Among Multiple Sclerosis Patients in Croatia

Ana Jerković, Ana Proroković, Meri Matijaca, Jelena Vuko,

Ana Poljičanin, Angela Mastelić, Ana Ćurković Katić, Vana Košta,

Lea Kustura, Krešimir Dolić, Zoran Ðogaš and Maja Rogić Vidaković

17 Life Design Counseling: Theory, Methodology, Challenges,

and Future Trends

Ya Wen, Kai Li, Huaruo Chen and Fei Liu

24 Open Science Within Pediatric Medical and Mental Health

Systems: Practical Considerations for Behavioral Health

Researchers

Robyn E. Metcalfe

28 Impact of a Remotely Supervised Motor Rehabilitation

Program on Maternal Well-Being During the COVID-19 Italian

Lockdown

Moti Zwilling, Alberto Romano, Martina Favetta, Elena Ippolito and

Meir Lotan

39 Clinical Guidelines of the Egyptian Psychiatric Association for

the Management of Treatment-Resistant Unipolar Depression

in Egypt

Momtaz Abdel-Wahab, Tarek Okasha, Mostafa Shaheen,

Mohamed Nasr, Tarek Molokheya, Abd ElNasser Omar,

Menan A. Rabie, Victor Samy, Hany Hamed and Mohamed Ali

51 Examining Motor Anticipation in Handwriting as an Indicator

of Motor Dysfunction in Schizophrenia

Yasmina Crespo Cobo, Sonia Kandel, María Felipa Soriano and

Sergio Iglesias-Parro

62 The Effectiveness of the Moving to Emptiness Technique on

Clients Who Need Help During the COVID-19 Pandemic: A

Real-World Study

Yanqiang Tao, Yi Chen, Wen Zhou, Lihui Lai and Tianjun Liu

71 Hospital Culture and Healthcare Workers’ Provision of

Patient-Centered Care: A Moderated Mediation Analysis

Xianhong Huang, Yuan Gao, Hanlin Chen, Hao Zhang and

Xiaoting Zhang

85 The Integration of Assistive Technology and Virtual Reality for

Assessment and Recovery of Post-coma Patients With

Disorders of Consciousness: A New Hypothesis

Fabrizio Stasolla, Leonarda Anna Vinci and Maria Cusano

Table of

contents

July 2023

Frontiers in Psychology 4frontiersin.org

91 Application of the unified protocol for the transdiagnostic

treatment of comorbid emotional disorders in patients with

ultra-high risk of developing psychosis: A randomized trial

study protocol

Trinidad Peláez, Raquel López-Carrillero, Marta Ferrer-Quintero,

Susana Ochoa and Jorge Osma

99 Effects of virtual reality natural experiences on factory

workers’ psychological and physiological stress

Mu-Hsing Ho, Meng-Shin Wu and Hsin-Yen Yen

TYPE Editorial

PUBLISHED 14 June 2023

DOI 10.3389/fpsyg.2023.1192774

OPEN ACCESS

EDITED AND REVIEWED BY

Gianluca Castelnuovo,

Catholic University of the Sacred Heart, Italy

*CORRESPONDENCE

Carmen María Galvez-Sánchez

cgalvez@ujaen.es

RECEIVED 23 March 2023

ACCEPTED 30 May 2023

PUBLISHED 14 June 2023

CITATION

Caprì T, Montoro CI and Galvez-Sánchez CM

(2023) Editorial: Methods and applications in

Psychology for Clinical Settings.

Front. Psychol. 14:1192774.

doi: 10.3389/fpsyg.2023.1192774

This is an open-access article distributed under

the terms of the Creative Commons Attribution

License (CC BY). The use, distribution or

reproduction in other forums is permitted,

provided the original author(s) and the

original publication in this journal is cited, in

accordance with accepted academic practice.

No use, distribution or reproduction is

permitted which does not comply with these

terms.

Editorial: Methods and

applications in Psychology for

Clinical Settings

Tindara Caprì1, Casandra Isabel Montoro2and

Carmen María Galvez-Sánchez2*

1Department of Life and Health Sciences, and Health Professions, Link Campus University, Rome, Italy,

2Department of Psychology, University of Jaén, Jaén, Spain

KEYWORDS

Psychology for Clinical Settings, experimental techniques, technology, methods, general

psychology

Editorial on the Research Topic

Methods and applications in Psychology for Clinical Settings

The main aim of this Research Topic was to highlight the latest experimental techniques

and methods used to investigate relevant questions in Psychology for Clinical Settings. This

Research Topic includes 11 papers, six original research articles, two opinion articles, one

mini review, and one brief research report, which include recent techniques and up-to-date

methods that contribute to advancing the science of Psychology for Clinical Settings.

The ﬁrst study (Jerkovi´

c et al.) examined the factor structure, internal consistency, and

correlates of the Croatian version of the Hospital Anxiety and Depression Scale (HADS)

for patients with multiple sclerosis (MS). A total of 179 patients with MS and 999 healthy

subjects completed the HADS. The results of this study demonstrated that the HADS

is a reliable and valid self-assessment scale and suggested that it be used in the clinical

monitoring of the psychiatric and psychological status of patients with MS.

The second study (Cobo et al.) investigated whether motor anticipation in handwriting

can be an indicator of motor dysfunction in schizophrenia. A total of 24 subjects with a

diagnosis of schizophrenia and 24 healthy subjects performed an easy and brief handwriting

task. The authors used three measures to evaluate motor anticipation: the time per stroke

(duration), the path of the pen for each stroke (trajectory), and the number of velocity

peaks (disﬂuency). The results indicated that patients with schizophrenia did not exhibit

any signs of motor anticipation. This study supported the idea of using handwriting analysis

as a quantitative, objective, and reliable tool to detect motor alterations in schizophrenia.

Two studies included in this Research Topic described the eﬀects of two diﬀerent

therapeutic techniques used during the COVID-19 pandemic. The ﬁrst study (Zwilling et al.)

examined the wellbeing level of mothers of girls and women with Rett Syndrome (RTT)

who were involved in a home-based, remotely supervised motor rehabilitation program

before and during the Italian COVID-19 lockdown. In total, 40 subjects with RTT were

randomly assigned to two groups: Group 1 received the intervention immediately before the

lockdown, and Group 2 received it during the lockdown. The motor rehabilitation program

consisted of an individualized daily physical activity program carried out for 12 weeks by the

participants’ parents and supervised every two weeks through Skype calls to plan, monitor,

and accommodate individual activities to the participants’ home lives. The results showed

that the participants’ mothers’ wellbeing was similar in the two groups, indicating that the

Frontiers in Psychology 01 frontiersin.org

Caprì et al. 10.3389/fpsyg.2023.1192774

lockdown inﬂuenced the eﬀect of the rehabilitation program. Thus,

this study proposed that the motor intervention helped the mothers

of patients with RTT to manage the new daily routine at home.

The second study, by Tao et al., described a new therapeutic

technique to reduce psychological symptoms during the COVID-

19 pandemic in China. This technique was the “Moving to

Emptiness Technique” (MET), which combined traditional Chinese

culture with relaxation and the operational process of Cognitive

Behavioral Therapy (CBT). A total of 17 therapists treated 107

subjects using the MET. The participants were subdivided into

two groups: a high-frequency symptom group and a low-frequency

symptom group. The results showed that symptoms decreased

signiﬁcantly in both groups after the intervention, indicating

that MET is a good therapeutic technique. However, the authors

suggested caution in the interpretation of their results due to the

small sample size.

Another study by Huang et al. examined the eﬀects of hospital

culture, self-eﬃcacy, and achievement motivation on healthcare

workers’ perceived delivery of patient-centered care (PCC). In

total, 1,612 healthcare workers from diﬀerent levels of public

hospitals completed a survey interview. The results suggested that

self-eﬃcacy, achievement motivation, and hospital culture were

necessary to promote the use of PCC among healthcare workers.

The last research article (Ho et al.) included in this Research

Topic investigated the eﬀects of virtual reality nature experiences

on the psychological and physiological stress of furniture factory

employees. A total of 35 factory workers were assigned to two

groups: an experimental group and a comparison group. The

experimental group received virtual reality experiences consisting

of 30-min nature-based 360◦videos played in a headset. The

intervention was conducted once a week for 12 weeks. The

comparison group received no intervention; participants in this

group freely performed their activities during their afternoon break.

The results indicated that the experimental group showed an

improvement in distress, depression, and anxiety compared to the

comparison group.

With reference to two opinion articles included in the

present Research Topic, the ﬁrst by Metcalfe discussed practical

considerations for behavioral health researchers when using open

science. The author encouraged researchers to consider open

science practices for their projects in pediatric medicine because

they can be a viable tool for data reporting and connecting

families and peers within pediatric-focused institutions. The second

opinion article (Stasolla et al.) discussed the use of technology in the

evaluation and recovery of post-coma and altered consciousness

patients. In particular, the authors proposed the integration of

assistive technology-based devices and virtual reality setups for

both assessment and recovery purposes.

A mini-review (Wen et al.) examined the state of the art

in Life Design Counseling (LDC), showing the lack of attention

to clients from diverse backgrounds and professional counselors,

the lack of diﬀerent methods in the intervention process, and

the lack of research on LDC. Abdel-Wahab et al. wrote a brief

report to introduce clinical guidelines for the treatment of resistant

depression in Egyptian patients. The guidelines were consistent

with international recommendations and were the ﬁrst in Egypt.

However, the proposed guidelines had a limitation related to the

fact that the panel consisted of only eight doctors.

The present Research Topic oﬀers an overview of the

latest experimental techniques and methods used to investigate

fundamental issues in Psychology for Clinical Settings. The

studies and opinion articles included in this Research Topic were

conducted in diﬀerent clinical populations and settings, which

may represent a strength for the audience, researchers, and health

professionals. Given the collected ﬁndings and the worldwide

high prevalence of mental illness (World Health Organization,

2022a), continuous eﬀorts to explore new clinical intervention and

evaluation methodologies must be strongly encouraged. The latter

is of special relevance after the impact of the pandemic (World

Health Organization, 2022b).

Author contributions

TC drafted the ﬁrst version of this editorial. All authors

contributed to and approved the ﬁnal version.

Acknowledgments

We thank the authors of the articles published in this Research

Topic for their signiﬁcant contributions and the reviewers for their

careful work. We also thank the editorial board of the Psychology

for Clinical Settings Section for their support.

Conﬂict of interest

The authors declare that the research was conducted in the

absence of any commercial or ﬁnancial relationships that could be

construed as a potential conﬂict of interest.

Publisher’s note

All claims expressed in this article are solely those of the

authors and do not necessarily represent those of their aﬃliated

organizations, or those of the publisher, the editors and the

reviewers. Any product that may be evaluated in this article, or

claim that may be made by its manufacturer, is not guaranteed or

endorsed by the publisher.

References

World Health Organization (2022a). Mental Disorders. Available online at:

https://www.who.int/news-room/fact- sheets/detail/mental-disorders (accessed March

23, 2023).

World Health Organization (2022b). Mental Health

and COVID-19: Early Evidence of the Pandemic’s Impact.

Geneva.

Frontiers in Psychology 02 frontiersin.org

fpsyg-12-794353 November 26, 2021 Time: 12:51 # 1

ORIGINAL RESEARCH

published: 30 November 2021

doi: 10.3389/fpsyg.2021.794353

Edited by:

Carmen María Galvez-Sánchez,

University of Jaén, Spain

Reviewed by:

Rafał Watrowski,

University of Freiburg, Germany

Klaas Wardenaar,

University Medical Center Groningen,

Netherlands

Ljubica Konstantinovic,

University of Belgrade, Serbia

*Correspondence:

Maja Rogi ´

c Vidakovi´

maja.rogic@mefst.hr

Specialty section:

This article was submitted to

Psychology for Clinical Settings,

a section of the journal

Frontiers in Psychology

Received: 13 October 2021

Accepted: 12 November 2021

Published: 30 November 2021

Citation:

Jerkovi ´

c A, Prorokovi ´

c A,

Matijaca M, Vuko J, Polji ˇ

canin A,

Masteli ´

c A, ´

Curkovi ´

c Kati ´

c A, Košta V,

Kustura L, Doli ´

cK,Ðogaš Z and

Rogi ´

c Vidakovi´

c M (2021)

Psychometric Properties of the HADS

Measure of Anxiety and Depression

Among Multiple Sclerosis Patients in

Croatia. Front. Psychol. 12:794353.

doi: 10.3389/fpsyg.2021.794353

Psychometric Properties of the

HADS Measure of Anxiety and

Depression Among Multiple

Sclerosis Patients in Croatia

Ana Jerkovi ´

c1, Ana Prorokovi ´

c2, Meri Matijaca3, Jelena Vuko2, Ana Polji ˇ

canin4,5 ,

Angela Masteli ´

c6, Ana ´

Curkovi ´

c Kati ´

c3, Vana Košta3, Lea Kustura7, Krešimir Doli ´

c8,

Zoran Ðogaš1,9 and Maja Rogi ´

c Vidakovi ´

c1*

1Laboratory for Human and Experimental Neurophysiology, Department of Neuroscience, School of Medicine, University

of Split, Split, Croatia, 2Department of Psychology, University of Zadar, Zadar, Croatia, 3Department of Neurology, University

Hospital of Split, Split, Croatia, 4Institute of Physical Medicine and Rehabilitation with Rheumatology, University Hospital

of Split, Split, Croatia, 5Department for Health Studies, University of Split, Split, Croatia, 6Department of Medical Chemistry

and Biochemistry, University of Split School of Medicine, Split, Croatia, 7Department Psychiatry, University Hospital of Split,

Split, Croatia, 8Department of Radiology, University Hospital of Split, Split, Croatia, 9Sleep Medical Center, University

of Split, Split, Croatia

Depression and anxiety are common complaints in patients with multiple sclerosis (MS).

The study objective was to investigate the factor structure, internal consistency, and

correlates of the Croatian version of the Hospital Anxiety and Depression Scale (HADS) in

patients with MS. A total of 179 patients with MS and 999 controls were included in the

online survey. All subjects completed the HADS and self-administered questionnaires

capturing information of demographic, education level, disease-related variables, and

the Multiple Sclerosis Impact Scale-29 (MSIS-29). Psychometric properties were

examined by estimating the validity, reliability, and factor structure of the HADS in

patients with MS. The two HADS subscales (anxiety and depression) had excellent

internal consistencies (Cronbach’s αvalue 0.82–0.83), and factor analysis conﬁrmed

a two-factor structure. The convergent validity of the HADS subscales appeared to

be good due to the signiﬁcant correlations between HADS and MSIS-29. Receiver

operating characteristic (ROC) analysis indicates that the HADS subscales have a

signiﬁcant diagnostic validity for group differentiation. Hierarchical regression analysis

using MSIS-29 subscales as criterion variables showed consistent evidence for the

incremental validity of the HADS. The HADS is a reliable and valid self-assessment scale

in patients with MS and is suggested to be used in clinical monitoring of the psychiatric

and psychological status of patients with MS.

Keywords: Hospital Anxiety and Depression Scale (HADS), psychometrics, depression, multiple sclerosis, anxiety

INTRODUCTION

Multiple sclerosis (MS) has a high prevalence of depression, anxiety, and stress comorbidities

(Marrie et al., 2018;Karimi et al., 2020). Comorbid depression and anxiety disorders aﬀect more

than 20% of the MS population (Beiske et al., 2008;Fiest et al., 2015;Marrie et al., 2015, 2018;Karimi

et al., 2020). Various screening instruments have been used to evaluate depression, anxiety, and

Frontiers in Psychology | www.frontiersin.org 1November 2021 | Volume 12 | Article 794353

fpsyg-12-794353 November 26, 2021 Time: 12:51 # 2

Jerkovi ´

c et al. HADS Psychometrics

stress in a clinical population of people with MS (pwMS)

and non-clinical populations, including the Beck Depression

Inventory-II (BDI-II) (Beck and Steer, 1990;Watson et al., 2014),

Hospital Anxiety and Depression Scale (HADS) (Zigmond and

Snaith, 1983;Honarmand and Feinstein, 2009), and Depression,

Anxiety, and Stress Scale-21 (DASS-21) (Lovibond P.F. and

Lovibond S.H., 1995;Lovibond S.H. and Lovibond P.F., 1995;

Rogi´

c Vidakovi´

c et al., 2021). The HADS is one of the most

commonly used scales for assessing anxiety and depression

among patients in a general hospital setting (Zigmond and

Snaith, 1983;Mitchell et al., 2010). Watson et al. (2014) validated

anxiety and depression measures in pwMS, conﬁrming HADS

as an appropriate questionnaire to assess depression and anxiety

in pwMS. Recently Rogi´

c Vidakovi´

c et al. (2021) reported

psychometric properties of the DASS-21 scale in pwMS. The

normative data for the HADS in pwMS were provided in pwMS

in diﬀerent languages (Honarmand and Feinstein, 2009;Atkins

et al., 2012;Watson et al., 2014;Marrie et al., 2018;Pais-Ribeiro

et al., 2018). A systematic review of the structure of the HADS

(Cosco et al., 2012) found inconsistencies in the latent structure of

the scale, which were mainly related to the diﬀerent latent variable

analysis methods [exploratory factor analysis and conﬁrmatory

factor analysis (CFA)] used for HADS. Regarding factor structure

of HADS in pwMS MS, Pais-Ribeiro et al. (2018) conducted

CFA and exploratory factor analysis providing support for a two-

factor HADS structure in pwMS. There have also been speciﬁc

problems in the translated versions and cross-cultural use of

the HADS (i.e., authors from the same country do not apply

the identical versions of HADS translations) (Wichowicz and

Wieczorek, 2011;Maters et al., 2013;Watrowski and Rohde,

2014). The HADS has been validated in a diverse group of

subjects, including those in primary care patients (el-Rufaie

and Absood, 1995), geriatric patients (Flint and Rifat, 1996),

or cancer patients (Mitchell et al., 2010). In addition, speciﬁc

HADS cut-oﬀ points have been established for patients with

cancer (Ibbotson et al., 1994), gynecological disorders (Abiodun,

1994), stroke (Johnson et al., 1995), and for pwMS (Honarmand

and Feinstein, 2009). In previous studies conducted in Croatia,

the HADS has been used in medical conditions other than MS

(Filipovic-Grcic et al., 2010;Vuleti´

c et al., 2011;Ostoji´

c et al.,

2014;Pokrajac-Bulian et al., 2015;Miljanovi´

c et al., 2017), but

no study determined psychometric properties for the Croatian

version of the HADS in pwMS. Two studies conducted in Croatia

with HADS stated the origin of the Croatian version of HADS

(Miljanovi´

c et al., 2017;Gali´

c et al., 2020), and so far, only

Miljanovi´

c et al. (2017) investigated metric properties of HADS in

terminal cancer patients but having relatively smaller convenient

sample size without a control group.

OBJECTIVE

The purpose of this online survey, was to evaluate the

metric properties of the Croatian version of the HADS in

terms of validity, reliability, and factor structure in pwMS.

The study compared HADS subscales with a non-clinical

population (control healthy subjects) and published data in

pwMS (Watson et al., 2014;Pais-Ribeiro et al., 2018). The study

also investigated the incremental validity of HADS using the

Multiple Sclerosis Impact Scale-29 (MSIS-29) (Hobart et al.,

2001) and relevant demographic and disease-related variables as

the criterion variables.

MATERIALS AND METHODS

Study Population and Procedure

The subjects with MS were recruited by advertising through the

Association of Multiple Sclerosis Societies of Croatia (AMSSC).

A total of 179 pwMS and 999 control subjects were included

in the online survey. The demographic factors, education level,

and disease-related factors for pwMS and control subjects are

presented in Table 1. In the group of pwMS, 84% were women

with a mean age of 41.3 ±11.5 years, and 16% were men

with a mean age of 42.7 ±9.9 years. Most pwMS were right-

handed (92.7%) and 35–49 years old (49%). Most pwMS had high

school degrees (49.1%) and graduate university degrees (23.5%).

Most of the pwMS were diagnosed with MS disease between

0 and 5 years (41.4%), 26.7% were diagnosed between 6 and

11 years, and 31.8% reported having MS over 11 years. The mean

duration of the disease for pwMS was 8.7 ±7.2. A majority

of the people declared to have relapsing-remitting MS (RRMS)

(70.4%), while others reported having secondary progressive MS

(SPMS) (7.8%) and primary progressive MS (PPMS) (10.6%).

Some pwMS (11.2%) did not provide information on the type

of MS. The median Expanded Disability Status Scale (EDSS)

score for all pwMS was 3.5 ±3.5. Of the 179 pwMS, 51.8%

had comorbidities, of which the most common were endocrine,

nutritional, and metabolic diseases (9.9%) and diseases of the

circulatory system (7.8%).

In the group of control subjects, 81% of participants were

women with a mean age of 39.8 ±10.3 years, and 19 percent

(19%) were men with a mean age of 40.3 ±10.1 years. Most

of the controls were right-handed (93.4%) and between 35 and

49 years old (51%), and most of them had graduate university

degrees (43.7%) and high school degrees (25.6%). Of a total,

27.6% of people had comorbidities, of which the most common

were endocrine, nutritional, and metabolic diseases (8.2%) and

diseases of the circulatory system (5.2%).

The data were collected via a Google Forms survey from

December 16, 2020, until January 13, 2021.

Measurements and Data Collection

Demographic Information and Disease-Related

Variables

The participants were characterized by demographic information

(age, sex, and handedness), educational, and disease-related

factors, including duration of the disease, MS type (Lublin et al.,

2014), and the score on the EDSS (McDonald et al., 2001).

Hospital Anxiety and Depression Scale (HADS)

The HADS (Zigmond and Snaith, 1983) is a self-report scale

consisting of two subscales, one measuring anxiety with seven

items (HADS-A) and one measuring depression with seven items

Frontiers in Psychology | www.frontiersin.org 2November 2021 | Volume 12 | Article 794353

fpsyg-12-794353 November 26, 2021 Time: 12:51 # 3

Jerkovi ´

c et al. HADS Psychometrics

(HADS-D). The subject gives answers to each question on a 4-

point (0–3) Likert scale and answering how he/she has been

feeling in the past week. Items 1, 3, 5, 7, 9, 11, 13 belong to

the anxiety subscale, while items: 2, 4, 6, 8, 10, 12, 14 belong to

the depression subscale. The total score is obtained by summing

the scores within each subscale. According to Pais-Ribeiro et al.

(2018) interpretation, the score 0–7 represents “normal,” 8–10

“mild,” 11–14 “moderate,” and 15–21 “severe.” In the present

study, the cut-oﬀ score of ≥8 and of ≥11 was used for HADS

subscales (Botega et al., 1995;Bjelland et al., 2002;Honarmand

and Feinstein, 2009;Brennan et al., 2010;Watson et al., 2014;

Litster et al., 2016).

Multiple Sclerosis Impact Scale-29

The MSIS-29 is a self-report scale capturing MS disease’s impact

from a patient’s physical and psychological perspective (Hobart

et al., 2001;Rogi´

c Vidakovi´

c et al., 2021). The MSIS-29 is a

self-report scale capturing MS disease’s impact from a patient’s

physical and psychological perspective. The scale is structured

into two subscales, a 20-item scale for measuring physical impact

and a 9-item scale for measuring the psychological impact of the

disease. The “physical impact” subscale consists of items from 1

to 20. The subscale of “psychological impact” consists of items

from 21 to 29. The patient is instructed to read each statement

about the disease’s impact on his/her everyday life in the past

2 weeks. For each statement, the patient’s task is to circle the

TABLE 1 | Characteristics of study participants.

Control subjects

(N= 999)

pwMS (N= 179)

Age in years (mean ±SD) 39.9 ±10.2 41.6 ±11.3

Age (range) 20–74 19–75

Sex

Female 81% 84%

Male 19% 16%

MS type

RRMS 70.4%

SPMS 7.8%

PPMS 10.6%

Not known 11.2%

Years of MS disease (mean ±SD) 8.7 ±7.2

EDSS (median ±IQR, range) 3.5 ±3.5, 0–9

EDSS* 2.5 ±2.5

EDSS** 6 ±2

Self-report scales (mean ±SD)

HADS-A 6.5 ±3.6 8.8 ±4.1

HADS-D 5.1 ±3.1 7.8 ±3.9

MSIS-29 PHYS 46.6 ±17.2

MSIS-29 PSY 24.3 ±8.8

SD, standard deviation; IQR, interquartile range; EDSS, Expanded Disability

Status Scale; EDSS*, fully preserved mobility 0–4.5; EDSS**, partially or

fully impaired mobility 5–9.5; RRMS, relapsing-remitting multiple sclerosis;

SPMS, secondary progressive multiple sclerosis; PPMS, primary progressive

multiple sclerosis; HADS-A, HADS Anxiety subscale; HADS-D, HADS Depression

subscale; MSIS-29 PHYS, MSIS-29 Physical subscale; MSIS-29 PSY, MSIS-29

Psychological subscale.

TABLE 2 | Score classiﬁcation percentages of HADS anxiety and depression

subscales for pwMS and control subjects.

Control subjects pwMS

HADS-A (%) HADS-D (%) HADS-A (%) HADS-D (%)

(0–7) normal 65.7 79.7 41.4 50.3

(8–10) mild 21.3 12.9 22.9 21.8

(11–14) moderate 9.6 7.2 26.8 24.0

(15–21) severe 3.4 0.2 8.9 3.9

≥8 34.3% 20.3% 58.6% 49.8%

≥11 13% 7.4% 35.7% 27.9%

HADS-A, HADS Anxiety subscale; HADS-D, HADS Depression subscale.

number that best describes his/her condition and answering on

a 5-point Likert scale (1 = not at all, 2 = a little, 3 = moderately,

4 = quite a bit, and 5 = extremely. The patient’s scores on

two subscales generated by summing individual items can be

transformed to a scale of 0–100, with higher scores indicating a

more severe disease burden.

Translation and Cultural Adaptation

Croatian translation of the HADS questionnaire was used in the

evaluation of anxiety and depression in patients suﬀering from

oncological (Miljanovi´

c et al., 2017) and neurological (Vuleti´

et al., 2011;Ostoji´

c et al., 2014) diseases or other conditions

(Filipovic-Grcic et al., 2010). Recently, HADS was used in the

general Croatian population during the COVID-19 infection

(Gali´

c et al., 2020). Among the mentioned studies that used the

Croatian translation of HADS, two studies stated the origin of the

translated version of the HADS questionnaire. Miljanovi´

c et al.

(2017) used the purchased Croatian translation of HADS from

Mapi Research Trust, and Gali´

c et al. (2020) used the translated

Croatian version of HADS from Pokrajac-Bulian et al. (2015).

The reason why our group initiated the HADS translation

procedure is the fact that the translation of HADS from Mapi

Research Trust is not entirely in the spirit of the Croatian

language according to the authors’ opinion, and all authors of this

study agreed not to use it in the present study. Also, the Croatian

version of HADS from Mapi Research Trust is not publicly free of

charge to the research community. Further, since Pokrajac-Bulian

et al. (2015) did not detail the process of translating HADS into

Croatian, having a relatively small sample size of obese people,

and the main aim of the study was not the validation of HADS in

the Croatian population, we did not consider it appropriate.

Therefore, our group translated HADS following current

recommendations, methodological approaches, and guidelines

in the process of translating, adapting, and cross-validating

instruments (Sousa and Rojjanasrirat, 2011). One author of

this study (MRV) and Professor of English language (Professor

Dalibora Behmen – DB, from the University of Split School of

Medicine), both natives in the Croatian language, translated the

HADS from English to Croatian. Next, the English language

professor (DB) compared both translated versions of HADS in

the Croatian language and produced the ﬁnal version of the

questionnaires. Another independent English language professor

(University of Split) who had no insight into the original English

Frontiers in Psychology | www.frontiersin.org 3November 2021 | Volume 12 | Article 794353

fpsyg-12-794353 November 26, 2021 Time: 12:51 # 4

Jerkovi ´

c et al. HADS Psychometrics

FIGURE 1 | Path diagram for the conﬁrmatory factor analysis of HADS with standardized regression weights.

version translated the last Croatian version of the questionnaires

back into the English language, completing the ﬁnal adaptation of

the Croatian version of HADS used in this study (Supplementary

Material).

Validation Procedure

Internal consistency of HADS was estimated by Cronbach’s alpha

coeﬃcients and inter-item correlations. CFA was carried out to

test the validity of the two-factor and one-factor models. Data

were analyzed by using the generalized least square (GLS) method

and the maximum likelihood (ML) estimator. Several criteria

[ML Chi-square, root mean square (RMS) standardized residual,

Steiger Lind RMSEA, and McDonald non-centrality index] are

reported with an emphasis on the root mean square error of

approximation (RMSEA), the most commonly used ﬁt index.

Convergent validity was demonstrated by the correlation between

HADS and MSIS-29 subscales. Concurrent validity was assessed

by comparisons between a group of pwMS and control subjects.

A receiver operating characteristic (ROC) curve was used to

determine the optimum cut-oﬀ score for each HADS subscales –

the score that yielded the best balance between sensitivity

and speciﬁcity. Furthermore, comparisons were also provided

between published data on psychometric properties of the HADS

(Watson et al., 2014;Pais-Ribeiro et al., 2018). Pais-Ribeiro et al.

(2018) oﬀered psychometric properties of HADS, analyzing a

sample of 380 pwMS (63.9% female; mean age 40.0 ±10.9 years;

range: 16–71 years) from the outpatient Neuroimmunology

Clinic at a Central Hospital in Porto, Portugal, while Watson et al.

(2014) included 34 pwMS (71%) female and (29%) male; mean

age 48.5 (11.1) from England.

The incremental validity of HADS was assessed by the

hierarchical regression model using the MSIS-29 and relevant

demographic and disease-related factors as criterion variables.

Age, sex, EDSS, type of MS, duration of the disease were entered

into the ﬁrst step, while the scores on HADS subscales were added

in the second step.

Statistical Analyses

Parameters of skewness and kurtosis were tested for HADS

and MSIS-29 scales. Results indicated acceptable values for

the parametric statistic. Mean value comparisons between our

study and published studies using HADS (Watson et al., 2014;

Pais-Ribeiro et al., 2018) in pwMS and diﬀerences between

relevant disease-related variables were carried using t-tests,

Chi-square test, Mann–Whitney Utest, Kruskal–Wallis test,

and variance analysis (ANOVA). The post hoc Bonferroni

test was calculated when using multiple comparisons. Levene’s

test was used to assess the equality of variances between

Frontiers in Psychology | www.frontiersin.org 4November 2021 | Volume 12 | Article 794353

fpsyg-12-794353 November 26, 2021 Time: 12:51 # 5

Jerkovi ´

c et al. HADS Psychometrics

groups. Correlation analyses were conducted using Pearson’s r

coeﬃcient and Spearman rank-order correlation (ρ). Descriptive

statistics of relevant participants’ characteristics and applied

scales were summarized by N, percentage, mean and standard

deviations, median, and interquartile range. Psychometric

properties were examined by estimating internal consistency,

factor structure, convergent, concurrent, and incremental validity

of the HADS. In all calculations, a p-value of <0.05 was

considered statistically signiﬁcant. Data analysis was performed

using the software Statistica 12.

RESULTS

Overview Results

The demographic characteristics, disease-related variables, and

mean results on self-rating scales (HADS and MSIS-29) of pwMS

and healthy subjects are shown in Table 1. No signiﬁcant sex

(χ2= 0.05, p= 0.82, p>0.05) and age (t=−4.84, df = 1390,

p>0.05) diﬀerences were found between pwMS and control

subjects. The scores on HADS depression (t=−2.34, df = 177,

p<0.05) and MSIS-29 physical (t=−2.94, df = 177, p<0.01)

subscales varied signiﬁcantly by MS type in pwMS. People

with RRMS type (MeanHADS−D= 7.4; MeanMSIS−PHYS = 43.9)

were less depressed and had better physical health than people

with SPMS (MeanHADS−D= 10.0; MeanMSIS−PHYS = 55.9)

and PPMS (MeanHADS−D= 10.1; MeanMSIS−PHYS = 56.4). For

women, HADS scores on depression subscale varied signiﬁcantly

with MS type. The women with RRMS were less depressed

[χ2(df =3)= 8.81; p<0.05] than women with SPMS and PPMS.

However, the sex diﬀerences were found in pwMS in achievement

on the HADS depression subscale, indicating that the male

participants have a higher depression score than females with MS

(t=−2.10, df = 177, p<0.05). Further, in pwMS, signiﬁcant

diﬀerences were found between diﬀerent age groups (19–34; 35–

39; and 40–75 years) for HADS depression subscales (F= 12.34;

p<0.001) and MSIS-29 physical impact subscale (F= 12.16;

p<0.001). Post hoc results suggest an increase in depression

and poorer physical health in older pwMS than younger pwMS

(pyounger vs.older <0.001; pmidle age vs.older = 0.04; p<0.05;

pmidle age vs.younger = 0.04; p<0.05).

Further, the participants who suﬀer from MS for a more

extended period (more than 11 years) have poorer physical

health on the MSIS-29 than those who are younger and suﬀer

from MS for a shorter period, less than 5 years (F= 3.29,

p<0.05). Furthermore, when levels of physical health and

depression were compared for types of MS (1-participants with

RRMS; 2-participants with other types of MS/SPMS, PPMS,

MS type not provided), a signiﬁcant diﬀerence was also found

(tdepression = 2.34, df = 177, p<0.05; tphysical =−2.54, df = 177,

p<0.001). Participants with RRMS had better physical health

and were less depressed than people with SPMS, PPMS, and those

who did not provide information on MS type.

Table 2 presents the score classiﬁcation percentages of HADS

anxiety and depression subscales for pwMS and control subjects.

According to score classiﬁcation for the HADS depression

subscale, 49.8% of the pwMS exhibited a score of ≥8 compared

to 20.3% of control subjects. For HADS anxiety score, 58.6%

of pwMS presented a score ≥8, compared to 34.3% of control

subjects. Moreover, based on the score of ≥11, for the HADS

depression subscale, 27.9% of pwMS exhibited moderate or

severe depression compared to 7.4% of control subjects. For

the HADS anxiety subscale, 35.7% of pwMS presented a score

≥11 compared to 13% of control subjects. The prevalence of

depression in pwMS seems to be higher in comparison to anxiety.

Psychometric Properties of the Hospital

Anxiety and Depression Scale (HADS)

Internal Consistency

Expressed by Cronbach’s Acoeﬃcients, both HADS subscales

(αHADS−A= 0.82 to αHADS−D= 0.83) and MSIS-29 subscales

(αMSIS−PHYS = 0.82 to αMSIS−PSY = 0.81) had excellent internal

consistency. Values for both HADS and MSIS-29 scales are

considered indicative of good reliability. Inter-item correlations

for HADS and MSIS-29 scales were >0.3, meaning that all items

on each subscale correlate very well with the scale overall.

Factor Analysis of the Hospital Anxiety and

Depression Scale (HADS)

Indicated by almost all obtained ﬁtting parameters except for a

slightly higher ratio between Chi-square and corresponding df

(Kenny, 2020), CFA conﬁrmedthe original structure of the HADS

in general (Figure 1 and Table 3). Namely, HADS, as expected,

shows a primarily two-factor structure (separate dimensions of

anxiety and depression) with mutually signiﬁcantly correlated

factors. HADS-A subscale explained 18.66% of factor variance

and with HADS-D subscale 21.85% of the variance. The CFA

for the one-factor solution was also reported (Table 3), but all

ﬁt indices support the retention of the two-factor solution. The

Steiger Lind RMSEA index was used as the main and most

commonly used criteria for accepting models. Cut-oﬀ RMSEA

TABLE 3 | Fit indices for one-factor and two-factor model of HADS (CFA).

One-factor

solution

Two-factor

solution

ML Chi-square 752.03 (df = 77) 128.315 (df = 28)

RMS standardized residual 0.059 0.024

Steiger Lind RMSEA 0.107 0.051

McDonald non-centrality index 0.643 0.93

Rms, root mean square; Rmsea, root mean square error of approximation; Ml,

maximum likelihood.

TABLE 4 | Pearson correlation coefﬁcient for HADS and MSIS-29 scale (N= 179).

HADS-A HADS-D MSIS-29 PHYS MSIS-29 PSY

HADS-A – 0.54** 0.33** 0.69**

HADS-D – 0.54** 0.61**

MSIS-29 PHYS – 0.57**

MSIS-29 PSY –

**p <0.01; HADS-A, HADS Anxiety subscale; HADS-D, HADS Depression

subscale; MSIS-29 PSY-MSIS-29 Psychological subscale.

Frontiers in Psychology | www.frontiersin.org 5November 2021 | Volume 12 | Article 794353

fpsyg-12-794353 November 26, 2021 Time: 12:51 # 6

Jerkovi ´

c et al. HADS Psychometrics

FIGURE 2 | Box plots showing medians and interquartile range of HADS scores in pwMS and control subjects. HADS-A, HADS Anxiety subscale; HADS-D, HADS

Depression subscale.

value of <0.05 indicates a “close ﬁt,” and that <0.08 suggests

a reasonable model–data ﬁt (e.g., Browne and Cudeck, 1993;

Jöreskog and Sörbom, 1993).

Convergent Validity of the Hospital Anxiety and

Depression Scale (HADS)

Convergent validity was demonstrated by the correlations of

the HADS subscales and the MSIS-29 subscales (Table 4) for

pwMS. HADS anxiety and depression subscales have a signiﬁcant

moderate correlation (r= 0.54; p<0.001). Moreover, both HADS

subscales are correlated with MSIS-29 subscales, noting that the

correlations of HADS subscales are higher with the psychological

MSIS-29 subscale (r= 0.61–0.69; p<0.01) compared to the

physical MSIS-29 subscale (r= 0.33–0.54; p<0.01). Correlation

coeﬃcients between HADS subscales and MSIS-29 subscales

indicate weak and moderate correlations.

Concurrent Validity

Concurrent validity was demonstrated by diﬀerences between

MS and control subjects. HADS mean values for pwMS were

signiﬁcantly higher (Mann–Whitney Utest; zanxiety = 6.98,

p<0.01; zdepression = 8.588, p<0.01) than those reported in

control subjects (Figure 2). A non-parametric test was done

because Levene’s test for homogeneity of variances was signiﬁcant

(both HADS-A and HADS-D). Further, compared to the results

of the current study with Watson et al. (2014) and Pais-Ribeiro

et al. (2018), depression and anxiety were not equally represented

(Table 5). The results on both subscales were signiﬁcantly higher

in our sample than those presented by Pais-Ribeiro et al. (2018),

and the diﬀerence is signiﬁcantly more pronounced when it

comes to HADS-D. Compared to Watson et al. (2014), there were

no signiﬁcant diﬀerences in depression levels, while the diﬀerence

in anxiety exists (small eﬀect size).

Receiver operating characteristic analysis (Table 6) indicated

that for the HADS-A, the highest value of the Youden Index

(J= 0.245) was obtained for a cut-oﬀ point of >7 and the HADS-

D at the cut-oﬀ point of >6 (J= 0.328). For the HADS-A, the

statistically signiﬁcant AUC was 0.664 (p<0.001) with a 95%

conﬁdence interval of 0.635–0.692. For the HADS-D, AUC was

0.702 (p<0.001) with 95% conﬁdence interval 0.675–0.728.

Both parameters (Jand AUC) indicate that the HADS-A and

the HADS-D have a signiﬁcant diagnostic validity for group

diﬀerentiation.

Incremental Validity

Table 7 represents the results of multiple hierarchical regression

analyses and the incremental validity of the HADS. Results

indicate whether HADS-A and HADS-D contribute to the

explanation of MSIS-29 variance (incremental validity) in

relation to some examined sociodemographic variables,

MS type, and EDDS.

For the physical impact on the MSIS-29, the ﬁrst set of

predictor variables (age, sex, EDSS, MS type, and disease

duration) only sex had a signiﬁcant βcoeﬃcient. Step 2,

which included HADS subscales, revealed that these variables

TABLE 5 | The HADS results from the present study and comparisons between

published studies.

HADS-A HADS-D

Present study

N= 179

Mean (SD) 8.82 (4.11) 7.80 (3.99)

Watson et al.

(2014) N= 34

Mean (SD) 7.2 (5.4) 8.1 (5.9)

t1.99 0.37

df 211 211

p0.04; p<0.05 0.71; p>0.05

Pais-Ribeiro

et al. (2018)

N= 380

Mean (SD) 7.94 (4.31) 5.63 (4.01)

t2.29 5.98

df 557 557

p0.02; p<0.5 0.00; p<0.001

HADS-A, HADS Anxiety subscale; HADS-D, HADS Depression subscale.

Frontiers in Psychology | www.frontiersin.org 6November 2021 | Volume 12 | Article 794353

fpsyg-12-794353 November 26, 2021 Time: 12:51 # 7

Jerkovi ´

c et al. HADS Psychometrics

TABLE 6 | Psychometric properties of HADS-A and HADS-D at different cut-off

scores (ROC analysis).

Sensitivity (95% CI) Speciﬁcity (95% CI) +LR −LR

HADS-A scores

≤6 65.92 (58.5–72.8) 54.44 (51.1–57.7) 1.45 0.63

≤7 58.66 (51.1–66.0) 65.39 (62.2–68.5) 1.69 0.63

≤8 48.04 (40.5–55.6) 73.38 (70.4–76.2) 1.81 0.71

≤9 43.02 (35.7–50.6) 80.94 (78.2–83.4) 2.26 0.70

≤10 35.75 (28.7–43.2) 87.19 (84.8–89.3) 2.79 0.74

≤11 27.37 (21.0–34.5) 90.25 (88.1–92.1) 2.81 0.80

≤12 20.11 (14.5–26.7) 93.32 (91.5–94.9) 3.01 0.86

HADS-D scores

≤6 61.45 (53.9–68.6) 71.37 (68.5–74.2) 2.15 0.54

≤7 49.72 (42.2–57.3) 79.58 (76.9–82.0) 2.43 0.63

≤8 43.58 (36.2–51.2) 85.39 (83.0–87.5) 2.98 0.66

≤9 35.20 (28.2–42.7) 89.19 (87.1–91.0) 3.26 0.73

≤10 27.93 (21.5–35.1) 92.49 (90.7–94.0) 3.72 0.78

≤11 21.23 (15.5–28.0) 96.30 (94.9–97.4) 5.73 0.82

≤12 12.85 (8.3–18.7) 98.70 (97.8–99.3) 9.87 0.88

+, LR likelihood ratio for a positive result; −, LR likelihood ratio for a negative result;

CI, conﬁdence interval.

contribute to the explanation of an additional 18% of physical

impact variance. Among these predictors, only HADS depression

had signiﬁcant β, which is positive, meaning that the greater

depression is accompanied by greater physical impact (Table 7).

For the psychological impact on the MSIS-29, age, among

predictors included in the ﬁrst step, signiﬁcantly predicted

psychological impact, accounting for 13% of the variance.

Simultaneously, HADS depression and anxiety subscales entered

in the second step explained 40% of the psychological

impact variance. Anxiety and depression subscale signiﬁcantly

contributed to the explanation of the criterion variable. For both

criterion variables (MSIS-29 physical and psychological impact)

HADS has been shown to have signiﬁcant incremental validity

in the explanation of MSIS-29, especially when it comes to the

second criterion, MSIS-29 psychological impact. The additional

contribution of physical impact is 13%, and for psychological

impact, even 40%.

DISCUSSION

Anxiety and depressive disorders are among the most common

psychiatric illnesses highly comorbid with each other and

considered to belong to the broader category of internalizing

disorders (Kalin, 2020). More than 50% of the patients with

major depression have signiﬁcant anxiety and were considered

to have anxious depression (Fava et al., 2004;Beijers et al.,

2019). When looking into a healthy population compared to

pwMS in terms of developing mood disorders, the risk of

depression, anxiety, and stress are higher in MS patients than

in healthy subjects (Pham et al., 2018). The etiology of MS

disease is not yet known and factors such as immune system

deﬁciency, genetic predisposition, lack of vitamin D, Epstein-Barr

virus, family background, geographical region, stress, and lifestyle

play a role in this disease (Dehghani and Kazemi Moghaddam,

2015). Besides mood disorders, relevant clinical symptoms of MS

include disturbances in motor functions (e.g., tremor, weakness,

and spasticity), sensory deﬁcits (e.g., pain), visual impairments

(e.g., diplopia and optic neuritis), vascular dysfunctions, obesity,

and cognitive impairments (e.g., attention deﬁcits, working

memory impairments, information processing). Karimi et al.

(2020) investigated 87 MS patients in Iran and showed that 47.1%

had moderate depression, 39.1% had moderate anxiety, and

44.8% had moderate stress. A study in Canada (Pham et al., 2018)

TABLE 7 | Multiple hierarchical regression analyses for the incremental validity of HADS and relevant variables on MSIS-29 subscales.

MSIS-29 PHYS MSIS-29 PSY

Step 1 Step 2 Step 1 Step 2

Predictors β β β β

Step 1 Age 0.07 0.05 −0.37** −0.40*

Sex 0.32** 0.22* 0.16 0.01

Duration of the disease 0.17 0.10 0.11 0.01

Type MS 0.11 0.14 0.01 0.04

EDSS 0.05 0.08 0.05 0.09

Step 2 R20.19 0.13

F(5,80) = 3.71 p<0.001 F(5,80) = 2.50 p<0.04

HADS-A 0.14 0.20*

HADS-D 0.36** 0.54**

R20.37 0.54

F(7,79) = 6.52 p<0.001 F(7,78) = 13.00 p<0.001

1R20.18 0.40

F(7,79) = 11.18 p<0.001 F(7,78) = 34.36 p<0.001

HADS-A, HADS Anxiety subscale; HADS-D, HADS Depression subscale; MSIS-29 PHYS, MSIS-29 Physical subscale; MSIS-29 PSY, MSIS-29 Psychological subscale;

β, standardized regression coefﬁcient; R2, coefﬁcient of determination; 1R2, change in the coefﬁcient of determination; *p <0.05, CI = 95%; **p <0.01, CI = 98%.

Frontiers in Psychology | www.frontiersin.org 7November 2021 | Volume 12 | Article 794353

fpsyg-12-794353 November 26, 2021 Time: 12:51 # 8

Jerkovi ´

c et al. HADS Psychometrics

showed 30% of MS patients suﬀered from anxiety, and 16.3%

were aﬀected with depression. The results of a study in the

United States (Boeschoten et al., 2017) revealed 20.6% of MS

patients suﬀered from depression. A signiﬁcant factor responsible

for MS relapses is stressful life events (Brown et al., 2005;

Stamoula et al., 2021). From a clinical point of view, it is therefore

recommended to monitor psychological constructs such as

depression, anxiety, and stress in pwMS (Glaser et al., 2019).

According to a literature search, it is evident that scales such

as DASS-21 (Lovibond P.F. and Lovibond S.H., 1995;Lovibond

S.H. and Lovibond P.F., 1995) and HADS (Zigmond and Snaith,

1983) were mainly used for detecting depression, anxiety, and

stress in pwMS. Recently psychometric properties for DASS-

21 were published in pwMS (Rogi´

c Vidakovi´

c et al., 2021),

while psychometric properties for HADS in pwMS have been

available on diﬀerent languages from earlier years (Honarmand

and Feinstein, 2009;Atkins et al., 2012;Watson et al., 2014;

Marrie et al., 2018;Pais-Ribeiro et al., 2018). What it has to bear in

mind is that HADS was not initially developed in pwMS. Instead,

it is created as a self-report rating scale for evaluating depression

and anxiety in patients with a general medical condition, but can

be regarded as a useful screening instrument to detect potential

psychological disturbances in pwMS (Honarmand and Feinstein,

2009;Watson et al., 2014).

By exploring the factor structure of the HADS, the present

study conﬁrmed a two-dimensionality of the HADS in a large

community and patient samples (Mykletun et al., 2001;Norton

et al., 2013), as well as in samples of pwMS (Pais-Ribeiro et al.,

2018). Internal consistency, using Cronbach’s alpha, for the two

dimensions was good, 0.80 for anxiety and 0.81 for depression

in the study of Pais-Ribeiro et al. (2018), while in the present

study, the Cronbach’s alpha, for the two dimensions was also

good, 0.82 for anxiety and 0.83 for depression. A systematic

review study conducted by Cosco et al. (2012) pointed out that

previous ﬁndings on the latent structure of the HADS have

been somewhat inconsistent factor structure with 25 of the

50 reviewed studies revealing a two-factor structure, 5 studies

revealing unidimensional, 17 studies revealing three-factor, and

2 studies revealing four-factor structures. According to the

ﬁndings of Cosco et al. (2012), diﬀerent latent variable analysis

methods gained correspondingly diﬀerent structures: exploratory

factor analysis studies revealed primarily two-factor structures,

CFA studies revealed primarily three-factor structures, and

item response theory studies revealed primarily unidimensional

structures. Regarding factor structure of HADS in MS research,

Pais-Ribeiro et al. (2018) conducted CFA and exploratory factor

analysis and provided support for the bifactor model. The present

study conﬁrmed a two-factor structure, and several ﬁt indices that

were used support the retention of the two-factor solution.

Parameters of ROC analysis indicate that the HADS-A and

the HADS-D have a signiﬁcant diagnostic validity for group

diﬀerentiation. Although the HADS depression scale shows

slightly better concurrent validity than HADS anxiety, the

accuracy of both measures to distinguish emotional disorder

is not very high. Therefore, the present study provided data

for the optimum cut-oﬀ score of >7 for HADS-A and a cut-

oﬀ score of >6 for HADS-D. The cut-oﬀ score of >7 for

HADS-A is similar to ﬁndings of Nicholl et al. (2001) and

Honarmand and Feinstein (2009), while the cut-oﬀ score of >6

for HADS-D was slightly lower compared to other studies using

HADS in pwMS (Honarmand and Feinstein, 2009;Watson et al.,

2014). When looking into studies using HADS in diﬀerent

samples of patients (not including pwMS) like cancer patients

or psychiatric illnesses, the sensitivity and speciﬁcity of HADS-

A and HADS-D with a threshold of 8+were most often found

to be in the range of 0.70–0.90. The variation in optimal

cut-oﬀ values and sensitivity and speciﬁcity might be due to

diﬀerences in HADS translations used, samples and procedures

in administration, and method analysis of HADS (Bjelland et al.,

2002;Cosco et al., 2012).

Both HADS subscales had excellent internal consistencies and

good convergent validity expressed by inter-correlations between

the HADS and the MSIS-29 subscales. Results of regression

analysis suggest that the HADS showed incremental validity in

relation to age, sex, MS type, and EDSS.

Further, we have to acknowledge several limitations of the

study. The possible limitation of the study would be the time of

conducting the survey. Namely, the study was conducted during

the COVID-19 pandemic (1 year after the ﬁrst lockdown in

Croatia) and a series of earthquakes that hit Croatia, causing

speciﬁc problems regarding the governmental social distancing

measures and collective trauma eﬀects. Although the study was

conducted during COVID-19 disease and strong earthquakes

in the eastern part of Croatia (Perinja and Zagreb region), we

assume that COVID-19 and earthquakes did not signiﬁcantly

aﬀect the HADS results in pwMS and control subjects. Gali´

et al. (2020) assessed depression and anxiety in the general

population with HADS 3 weeks after the ﬁrst registered cases

of COVID-19 in Croatia. In line with the study of Gali´

c et al.

(2020), observed values of depression were similar to the results

of control subjects in the present study, with less pronounced

anxiety in the present study. Further, a comparison with the

previous studies shows a higher prevalence of depression and

anxiety in pwMS independently of speciﬁc external factors not

related to the MS disease (Dahl et al., 2004;Karimi et al., 2020).

Another possible limitation is that HADS was not used as a

paper–pencil assessment but rather as an online survey. The

advantage of the online survey was the possibility to reach a

higher number of MS patients. The paper–pencil assessment of

HADS would last longer since we could access the MS patients

once a week at the University Hospital of Split during the

regular control examinations at the Department of Neurology.

An approximate number of MS patients that we could reach

weekly would be approximately three to ﬁve. The second

problem was that during regular control visits at the Department

of Neurology, the MS patients are not registered at speciﬁc

hours but are intermingled with other patients having other

neurological diseases. Therefore, we believe by conducting an

online survey, we reached a satisfactory number of MS patients in

a shorter period and got a more appropriate sample size avoiding

possible erroneous ﬁndings which might occur in the process of

determining psychometric properties of the HADS, in particular

the identiﬁcation of the correct structure of the questionnaire

(e.g., number of dimensions and items in each dimension).

Frontiers in Psychology | www.frontiersin.org 8November 2021 | Volume 12 | Article 794353

fpsyg-12-794353 November 26, 2021 Time: 12:51 # 9

Jerkovi ´

c et al. HADS Psychometrics

CONCLUSION

The HADS is shown to be a reliable and valid patient-self report

scale that captures meaningful psychological and physical clinical

correlates of MS disease.

DATA AVAILABILITY STATEMENT

The original contributions presented in the study are included

in the article/Supplementary Material, further inquiries can be

directed to the corresponding author.

ETHICS STATEMENT

The studies involving human participants were reviewed and

approved by the School of Medicine, University of Split. The

patients/participants provided their written informed consent to

participate in this study.

AUTHOR CONTRIBUTIONS

AJ: conceptualization, data curation, formal analysis,

methodology, project administration, resources, and

writing – original draft. APr: supervision, methodology,

and writing – original draft. MM, KD, and ZÐ: supervision and

writing – original draft. JV, AM, A ´

C, VK, and LK: methodology,

project administration, and resources. APo: methodology and

project administration. MRV: conceptualization, data curation,

formal analysis, methodology, project administration, resources,

supervision, and writing – original draft. All authors contributed

to the article and approved the submitted version.

SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found

online at: https://www.frontiersin.org/articles/10.3389/fpsyg.

2021.794353/full#supplementary-material

REFERENCES

Abiodun, O. A. (1994). A validity study of the hospital anxiety and depression scale

in general hospital units and a community sample in Nigeria. Br. J. Psychiatry

165, 669–672. doi: 10.1192/bjp.165.5.669

Atkins, L., Newby, G., and Pimm, J. (2012). Normative data for the Hospital

Anxiety and Depression Scales (HADS) in multiple sclerosis. Soc. Care

Neurodisabil. 3, 172–178. doi: 10.1108/20420911211286579

Beck, A. T., and Steer, R. A. (1990). Beck Anxiety Inventory (BAI) Manual. Oxford:

Pearson.

Beijers, L., Wardenaar, K. J., Bosker, F. J., Lamers, F., van Grootheest, G., de

Boer, M. K., et al. (2019). Biomarker-based subtyping of depression and anxiety

disorders using latent class analysis. A NESDA study. Psychol.Med. 49, 617–627.

doi: 10.1017/S0033291718001307

Beiske, A. G., Svensson, E., Sandanger, I., Czujko, B., Pedersen, E. D., Aarseth, J. H.,

et al. (2008). Depression and anxiety amongst multiple sclerosis patients. Eur. J.

Neurol. 15, 239–245. doi: 10.1111/j.1468-1331.2007.02041.x

Bjelland, I., Dahl, A. A., Haug, T. T., and Neckelmann, D. (2002). The validity

of the hospital anxiety and depression scale: an updated literature review.

J. Psychosom. Res. 52, 69–77. doi: 10.1016/s0022-3999(01)00296-3

Boeschoten, R. E., Braamse, A. M., Beekman, A. T., Cuijpers, P., van Oppen,

P., Dekker, J., et al. (2017). Prevalence of depression and anxiety in multiple

sclerosis: a systematic review and meta-analysis. J. Neurol. Sci. 372, 331–341.

doi: 10.1016/j.jns.2016.11.067

Botega, N. J., Bio, M. R., Zomignani, M. A., Garcia, C. Jr., and Pereira, W. A.

(1995). Mood disorders among medical in-patients: a validation study of the

hospital anxiety and depression scale (HAD). Rev. Saúde Pública 29, 355–363.

doi: 10.1590/S0034-89101995000500004

Brennan, C., Worrall-Davies, A., McMillan, D., Gilbody, S., and House, A. (2010).

The hospital anxiety and depression scale: a diagnostic meta-analysis of case-

ﬁnding ability. J. Psychosom. Res. 69, 371–378. doi: 10.1016/j.jpsychores.2010.

04.006

Brown, R. F., Tennant, C. C., Dunn, S. M., and Pollard, J. D. (2005). A Review of

stress-relapse interactions in multiple sclerosis: important features and stress-

mediating and -moderating variables [Internet]. Mult. Scler. 11, 477–484. doi:

10.1191/1352458505ms1170oa

Browne, M. W., and Cudeck, R. (1993). “Alternative ways of assessing model ﬁt,”

in Testing Structural Equation Models, eds K. A. Bollen and J. S. Long (Newbury

Park, CA: Sage).

Cosco, T., Doyle, F., Ward, M., and McGee, H. (2012). Latent structure of the

hospital anxiety and depression scale: a 10-year systematic review. J. Psychosom.

Res. 72, 180–184. doi: 10.1016/j.jpsychores.2011.06.008

Dahl, O. P., Aarseth, J. H., Myhr, K. M., Nyland, H., and Midgard, R. (2004).

First published anxiety and depression in multiple sclerosis. A comparative

population-based study in Nord-Trøndelag County, Norway. Acta Neurol.

Scand. 109, 378–384. doi: 10.1111/j.1600-0404.2004.00244.x

Dehghani, R., and Kazemi Moghaddam, V. (2015). Potential causes of the in

creased prevalence of multiple sclerosis in Iran: a review study. Pars J. Med.

Sci. 13, 17–25.

el-Rufaie, O. E., and Absood, G. H. (1995). Retesting the validity of the Arabic

version of the Hospital Anxiety and Depression (HAD) scale in primary

health care. Soc. Psychiatry Psychiatr. Epidemiol. 30, 26–31. doi: 10.1007/BF0078

4431

Fava, M., Alpert, J. E., Carmin, C. N., Wisniewski, S. R., Trivedi, M. H., Biggs,

M. M., et al. (2004). Clinical correlates and symptom patterns of anxious

depression among patients with major depressive disorder in STAR∗D. Psychol.

Med. 34, 1299–1308. doi: 10.1017/s0033291704002612

Fiest, K. M., Fisk, J. D., Patten, S. B., Tremlett, H., Wolfson, C., McKay, K. A.,

et al. (2015). Comorbidity is associated with pain-related activity limitations in

multiple sclerosis. Mult. Scler. Relat. Disord. 4, 470–476. doi: 10.1016/j.msard.

2015.07.014

Filipovic-Grcic, I. F., Tonkovic, D. T., Grubisin, J. G., Peri´

c, M. P., and Majeric

Kogler, V. M. (2010). Hospital anxiety depression scale in our surgical ICU.

Crit. Care. 14:499. doi: 10.1186/cc8731

Flint, A., and Rifat, S. (1996). Validation of the hospital anxiety and depression scale

as a measure of severity of geriatric depression. Int. J. Geriatr. Psychiatry 11,

991–994. doi: 10.1002/(SICI)1099-1166(199611)11:11<991::AID-GPS423< 3.0.

CO;2-8

Gali´

c, M., Mustapi´

c, L., Šimuni´

c, A., Si´

c, L., and Cipolletta, S. (2020). COVID-19

related knowledge and mental health: case of Croatia. Front. Psychol. 11:567368.

doi: 10.3389/fpsyg.2020.567368

Glaser, A., Stahmann, A., Meissner, T., Flachenecker, P., Horáková, D., Zaratin,

P., et al. (2019). Multiple sclerosis registries in Europe – an updated mapping

survey. Mult. Scler. Relat. Disord. 27, 171–178. doi: 10.1016/j.msard.2018.09.

032

Hobart, J., Lamping, D., Fitzpatrick, R., Riazi, A., and Thompson, A. (2001).

The Multiple Sclerosis Impact Scale (MSIS-29); a new patient-based outcome

measure. Brain 124, 962–973. doi: 10.1093/brain/124.5.962

Honarmand, K., and Feinstein, A. (2009). Validation of the hospital anxiety

and depression scale for use with multiple sclerosis patients. Mult. Scler. 15,

1518–1524. doi: 10.1177/1352458509347150

Ibbotson, T., Maguire, P., Selby, P., Priestman, T., and Wallace, L. (1994). Screening

for anxiety and depression in cancer patients – the eﬀects of disease and

treatment. Eur. J. Cancer 30, 37–40. doi: 10.1016/s0959-8049(05)80015- 2

Frontiers in Psychology | www.frontiersin.org 9November 2021 | Volume 12 | Article 794353

fpsyg-12-794353 November 26, 2021 Time: 12:51 # 10

Jerkovi ´

c et al. HADS Psychometrics

Johnson, G., Burvill, P. W., Anderson, C. S., Jamrozik, K., Stewartwynne, E. G., and

Chakera, T. M. H. (1995). Screening instruments for depression and anxiety

following stroke – experience in the Perth Community Stroke Study. Acta

Psychiatr. Scand. 91, 252–257. doi: 10.1111/j.1600-0447.1995.tb09778.x

Jöreskog, K. G., and Sörbom, D. (1993). Structural Equation Modeling with the

SIMPLIS Command Language. Chicago, IL: Scientiﬁc Software.

Kalin, N. H. (2020). The critical relationship between anxiety and depression. Am.

J. Psychiatry 177:5. doi: 10.1176/appi.ajp.2020.20030305

Karimi, S., Andayeshgar, B., and Khatony, A. (2020). Prevalence of Anxiety,

depression, and stress in patients with multiple sclerosis in Kermanshah-Iran: a

cross-sectional study. BMC Psychiatry20:166. doi: 10.1186/s12888-020-02579-z

Kenny, D. A. (2020). Measuring Model Fit. Available online at: http://www.

davidakenny.net/cm/ﬁt.htm (accessed November 1, 2021)

Litster, B., Fiest, K. M., Patten, S. B., Fisk, J. D., Walker, J. R., Graﬀ, L. A., et al.

(2016). Screening tools for anxiety in people with multiple sclerosis: a systematic

review. Int. J. MS Care 18, 273–281. doi: 10.7224/1537-2073.2016- 004

Lovibond, P. F., and Lovibond, S. H. (1995). The structure of negative emotional

states: comparison of the Depression Anxiety Stress Scales (DASS) with the

beck depression and anxiety inventories. Behav. Res. Ther. 33, 335–343. doi:

10.1016/0005-7967(94)00075-u

Lovibond, S. H., and Lovibond, P. F. (1995). Manual for the Depression Anxiety &

Stress Scales, 2nd Edn. Sydney, NSW: Psychology Foundation.

Lublin, F. D., Reingold, S. C., Cohen, J. A., Cutter, G. R., Sørensen, P. S., Thompson,

A. J., et al. (2014). Deﬁning the clinical course of multiple sclerosis: the 2013

revisions. Neurology 83, 278–286. doi: 10.1212/WNL.0000000000000560

Marrie, R. A., Reingold, S., Cohen, J., Stuve, O., Trojano, M., Sorensen, P. S.,

et al. (2015). The incidence and prevalence of psychiatric disorders in

multiple sclerosis: a systematic review. Mult. Scler. 21, 305–317. doi: 10.1177/

1352458514564487

Marrie, R. A., Zhand, L., Lix, L. M., Graﬀ, L. A., Walker, J. R., Fisk, J. D., et al.

(2018). The validity and reliability of screening measures for depression and

anxiety disorders in multiple sclerosis. Mult. Scler. Relat. Disord. 20, 9–15.

doi: 10.1016/j.msard.2017.12.007

Maters, G. A., Sanderman, R., Kim, A. Y., and Coyne, J. C. (2013). Problems in

cross-cultural use of the hospital anxiety and depression scale: “no butterﬂies in

the desert”. PLoS One 8:e70975. doi: 10.1371/journal.pone.0070975

McDonald, W. I., Compston, A., Edan, G., Goodkin, D., Hartung, H. P., Lublin,

F. D., et al. (2001). Recommended diagnostic criteria for multiple sclerosis:

guidelines from the International Panel on the diagnosis of multiple sclerosis.

Ann. Neurol. 50, 121–127. doi: 10.1002/ana.1032

Miljanovi´

c, M., Sindik, J., Milunovi´

c, V., Kralj Škoc, V., Braš, M., and Ðor¯

devi´

V. (2017). Factor structure and cut-oﬀ scores of the Hospital Anxiety and

Depression Scale (HADS) in a Croatian sample of adult patients suﬀering

from advanced cancer. Psychiatr. Danub. 29, 451–458. doi: 10.24869/psyd.2017.

451

Mitchell, A. J., Meader, N., and Symonds, P. (2010). Diagnostic validity of the

Hospital Anxiety and Depression Scale (HADS) in cancer and palliative settings:

a metaanalysis. J. Aﬀect. Disord. 126, 335–348. doi: 10.1016/j.jad.2010.01.067

Mykletun, A., Stordal, E., and Dahl, A. A. (2001). Hospital Anxiety and Depression

(HAD) scale: factor structure, item analyses and internal consistency in a large

population. Br. J. Psychiatry 179, 540–544. doi: 10.1192/bjp.179.6.540

Nicholl, C. R., Lincoln, N. B., Francis, V. M., and Stephan, T. F. (2001). Assessment

of emotional problems in people with multiple sclerosis. Clin. Rehabil. 2001,

657–668. doi: 10.1191/0269215501cr427oa

Norton, S., Cosco, T., Doyle, F., Done, J., and Sacker, A. (2013). The hospital anxiety

and depression scale: a meta conﬁrmatory factor analysis. J. Psychosom. Res. 74,

74–81. doi: 10.1016/j.jpsychores.2012.10.010

Ostoji´

c, D., Vidovi´

c, D., Baˇ

cekovi´

c, A., Breˇ

ci´

c, P., and Juki´

c, V. (2014). Prevalence

of anxiety and depression in caregivers of Alzeheimer’s dementia patients. Acta

Clin. Croat. 53, 17–21.

Pais-Ribeiro, J. J., da Silva, A. M., Vilhena, E., Moreira, I., Santos, E., and Mendonça,

D. (2018). The hospital anxiety and depression scale, in patients with multiple

sclerosis. Neuropsychiatr. Dis. Treat. 14, 3193–3197. doi: 10.2147/NDT.S184260

Pham, T., Jetté, N., Bulloch, A. G., Burton, J. M., Wiebe, S., and Patten, S. B. (2018).

The prevalence of anxiety and associated factors in persons with multiple

sclerosis. Mult. Scler. Relat. Disord. 19, 35–39. doi: 10.1016/j.msard.2017.11.003

Pokrajac-Bulian, A., Kuki´

c, M., and Baši´

c-Markovi´

c, N. (2015). Quality of life as a

mediator in the association between body mass index and negative emotionality

in overweight and obese non-clinical sample. Eat.Weight Disord. 20, 473–481.

doi: 10.1007/s40519-015- 0208-x

Rogi´

c Vidakovi´

c, M., Šimi´

c, N., Poljiˇ

canin, A., Nikoli´

c Ivaniševi´

c, M., Jerkovi´

A., and Ðogaš, Z. (2021). Psychometric properties of the Croatian version

of the depression, anxiety, and stress scale-21 and multiple sclerosis impact

scale-29 in multiple sclerosis patients. Mult. Scler. Relat. Disord. 50:102850.

doi: 10.1016/j.msard.2021.102850

Sousa, V. D., and Rojjanasrirat, W. (2011). Translation, adaptation and validation

of instruments or scales for use in cross-cultural health care research: a clear and

user-friendly guideline. J. Eval. Clin. Pract. 17, 268–274. doi: 10.1111/j.1365-

2753.2010.01434.x

Stamoula, E., Siaﬁs, S., Dardalas, I., Ainatzoglou, A., Matsas, A., Athanasiadis, T.,

et al. (2021). Antidepressants on multiple sclerosis: a review of in vitro and

in vivo models. Front. Immunol. 12:677879. doi: 10.3389/ﬁmmu.2021.677879

Vuleti´

c, V., Ležai´

c, Ž, and Morovi´

c, S. (2011). Post-stroke fatigue. Acta Clin. Croat.

50, 341–344.

Watrowski, R., and Rohde, A. (2014). Validation of the Polish version of the

hospital anxiety and depression scale in three populations of gynecologic

patients. Arch. Med. Sci. 10, 517–524. doi: 10.5114/aoms.2013.36520

Watson, T. M., Ford, E., Worthington, E., and Lincoln, N. B. (2014). Validation of

mood measures for people with multiple sclerosis. Int. J. MS. Care 16, 105–109.

doi: 10.7224/1537-2073.2013- 013

Wichowicz, H. M., and Wieczorek, D. (2011). Screening post-stroke depression

using the hospital anxiety and depression scale. Psychiatr Pol. 45, 505–514.

Zigmond, A. S., and Snaith, R. P. (1983). The hospital anxiety and depression

scale. Acta Psychiatr. Scand. 67, 361–370. doi: 10.1111/j.1600-0447.1983.tb09

716.x

Conﬂict of Interest: The authors declare that the research was conducted in the

absence of any commercial or ﬁnancial relationships that could be construed as a

potential conﬂict of interest.

Publisher’s Note: All claims expressed in this article are solely those of the authors

and do not necessarily represent those of their aﬃliated organizations, or those of

the publisher, the editors and the reviewers. Any product that may be evaluated in

this article, or claim that may be made by its manufacturer, is not guaranteed or

endorsed by the publisher.

c, Prorokovi´

c, Matijaca, Vuko, Poljiˇ

canin, Masteli´

Curkovi´

c Kati´

c, Košta, Kustura, Doli´

c, Ðogaš and Rogi´

c Vidakovi´

c. This is an open-

access article distributed under the terms of the Creative Commons Attribution

License (CC BY). The use, distribution or reproduction in other forums is permitted,

provided the original author(s) and the copyright owner(s) are credited and that the

original publication in this journal is cited, in accordance with accepted academic

practice. No use, distribution or reproduction is permitted which does not comply

with these terms.

Frontiers in Psychology | www.frontiersin.org 10 November 2021 | Volume 12 | Article 794353

fpsyg-13-814458 January 26, 2022 Time: 12:22 # 1

MINI REVIEW

published: 01 February 2022

doi: 10.3389/fpsyg.2022.814458

Edited by:

Casandra Isabel Montoro,

University of Jaén, Spain

Reviewed by:

Nieves Valencia Naranjo,

University of Jaén, Spain

Muhammad Nawaz Tunio,

Greenwich University, Pakistan

Margarida Pocinho,

University of Madeira, Portugal

*Correspondence:

Huaruo Chen

hchen164@jh.edu

Specialty section:

This article was submitted to

Psychology for Clinical Settings,

a section of the journal

Frontiers in Psychology

Received: 13 November 2021

Accepted: 10 January 2022

Published: 01 February 2022

Citation:

Wen Y, Li K, Chen H and Liu F

(2022) Life Design Counseling:

Theory, Methodology, Challenges,

and Future Trends.

Front. Psychol. 13:814458.

doi: 10.3389/fpsyg.2022.814458

Life Design Counseling: Theory,

Methodology, Challenges, and

Future Trends

Ya Wen1, Kai Li2, Huaruo Chen3,4*and Fei Liu3,5

1School of Teacher Education, Nanjing Xiaozhuang University, Nanjing, China, 2School of Philosophy, Wuhan University,

Wuhan, China, 3School of Education Science, Nanjing Normal University, Nanjing, China, 4Center for Research and Reform

in Education, Johns Hopkins University, Baltimore, MD, United States, 5School of Education Science, Huaiyin Normal

University, Huaian, China

With the rapid development of society and the dramatic change of environment,

previous career counseling focusing on personal choice has been difﬁcult to meet

individuals’ needs. It is very meaningful and valuable to introduce the ideology of Life

Design Counseling (LDC). In this mini review, we introduce and analyze the theory and

methodology of LDC. This review puts forward challenges in the ﬁeld of LDC, including

the lack of attention to clients from multiple backgrounds and professional counselors,

the lack of diversiﬁed methods in the intervention process, and the lack of diversiﬁed

research. The theoretical research, practical research, and the integration of theory and

practice of LDC still need to be further concerned by researchers.

Keywords: life design counseling, theory, methodology, challenges, trends

INTRODUCTION

The current living environment changes faster than ever before, resulting in the trend of personal

career development from static and linear to dynamic and nonlinear. Personal future work

and employment are becoming more and more unstable in this world dominated by Volatility,

Uncertainty, Complexity, and Ambiguity (VUCA) (Canzittu, 2020). Savickas et al. (2009) proposed

the third paradigm of career intervention, which is called the “Life Design Paradigm,” to deal with

the drastic changes in the current social environment (Savickas et al., 2009). Based on this paradigm,

Life Design Counseling (LDC) refers to a kind of counseling that helps clients deﬁne their career

as a story, narrate their personal working life with continuity and coherence, discover life themes,

create life meaning, construct identity, formulate adaptive actions, and pursue the life expected by

individuals (Savickas and Pouyaud, 2016;Venter and Maree, 2020;Wong, 2021). LDC aims to help

clients explore possible careers, reshape personal narrative identity through meaning construction,

clarify self-concept, establish a life purpose and create a meaningful life (Savickas, 2012;Wehmeyer

et al., 2018;Wen et al., 2020a). Compared with other career counseling methods, LDC provides

a new perspective for individual or group counseling to formulate long-term career planning and

has the potential for cross-cultural research (Savickas et al., 2018). Therefore, this mini review aims

to introduce the theory, methodology, challenges, and future trends of LDC, so that researchers

Frontiers in Psychology | www.frontiersin.org 1February 2022 | Volume 13 | Article 814458

fpsyg-13-814458 January 26, 2022 Time: 12:22 # 2

Wen et al. Life Design Counseling

engaged in this speciﬁc ﬁeld can more eﬀectively understand

and master the research trends and future development trends

in this ﬁeld, and expand the application scope of LDC. In order

to ensure the quality of the literature, LDC was used as the

search term to conduct a full search on the Web of Science

in this study. The search scope is limited to English articles.

Speciﬁcally, the literature search was conducted from January

2000 to November 2021. In addition, this study only included

standard research papers, excluding news, conferences and other

types of documents.

WHAT IS LIFE DESIGN COUNSELING?

Life Design Counseling is a kind of career intervention,

which originates from Guichard’s self-construction theory and

Savikcas’s career construction theory (Guichard, 2009;Maree,

2019). After entering the 21st century, individuals need to

face more complex social relationships and work environments,

especially to design their own lives. The premise of designing life

is self-construction. In response to the above changes, Guichard

put forward self-construction theory, which mainly points out

that self-construction is a subjective identity system of past,

present, and future (Guichard, 2009;Guichard et al., 2012). With

the development of career theory, in order to explore career from

a more integrated perspective, Savickas et al. proposed Career

Construction Theory. Career Construction Theory attempts to

look at individual career from an overall perspective, aiming

to answer three core questions, namely “What kind of career

will be established?” “How to construct a career? ” and “Why

build a career?” (Savickas, 2008;Savickas and Pouyaud, 2016).

Speciﬁcally, “What kind of career will be established?” reﬂects

the relationship between personal interests, abilities, values and

career development from the perspective of social division of

labor (Rudolph et al., 2019). “How to construct a career?”

is manifested as the interaction between individuals and the

environment in the current mobile society and ﬂexible work

organization, emphasizing that individuals reshape themselves in

reﬂection and cope with external challenges based on their career

adaptability. “Why build a career” is the fundamental issue of

Career Construction Theory, which truly reﬂects that individuals

regard their own careers as creating their own life stories and

strung the stories into a line around the theme of life. In general,

career is a macro narrative about the role of work in one’s life

which was deﬁned in career construction theory. Both those two

theories involve social constructivism, emphasize the signiﬁcance

of understanding and intervening narrative thinking in the

process of construction, and highlight the value of reﬂexivity

based on individual in-depth thinking of past and present

experience for individual behavior (Guichard, 2016b;Savickas,

2016). In short, whether it is the theory of Self-Construction

Theory or Career Construction Theory, reﬂexivity emphasizes

the formation of the individual based on current problems

to guide the next step in life. However, there are also some

diﬀerences between the two theories: self-construction theory

focuses on the survival direction of individuals, emphasizes

the dynamic development of individuals based on subjective

identity formal system (SIFS), and constructs self-concept in

the social environment (Guichard, 2016a). LDC based on Self-

Construction Theory adopts a relatively open interview form,

pays attention to the role of clients’ self-construction in meaning,

identity, and future planning, focuses on helping clients reshape

their narrative identity, and projects the new possibility of

self-construction into their career role.Therefore, it is more

suitable for teenagers or emerging adults (Di Fabio, 2014;

Guichard, 2016a). While Career Construction Theory points

out that career can be a dynamic construction process, and

subjective self and external world can adapt to each other.

LDC based on Career Construction Theory helps clients to

focus on their own career through highly structured interviews

and narration around personal interests and models, increase

their control over their future career positioning, stimulate

their curiosity to explore possible self and possible future

scenes, and build their conﬁdence in career development

(Maree, 2014). Counselors help clients reconstruct the meaning,

identity, and intention of life, as well as ﬁnd the life theme

and answer the meaning of life, which is more applicable

for mature adults (Guichard, 2016a;Hartung and Santilli,

2018).

In summary, LDC means that counselors help clients give

meaning to their lives and social relationships through various

stories and ﬁnd their understanding of life themes as well

throughout this process (Cardoso et al., 2016b;Wong, 2021).

LDC can be regarded as an intervention method combining

Self-Construction and Career Construction, which reﬂects the

integration of personal personality characteristics, development

process and life story (Di Fabio and Maree, 2012;Wong, 2021).

LDC emphasizes that individuals reﬂect on themselves and

construct meaning based on continuous narration by considering

the overall impact of the surrounding system, so as to generate

expectations and motivation for the future (Cardoso et al., 2016a;

Venter and Maree, 2020). Speciﬁcally, LDC focuses on people’s

interaction between themselves and the environment, establishes

life trajectory through narration, and considers educational

investment, work and life roles and meaning in life, so as to deal

with the uncertainty of the future (Nota et al., 2016;Cardoso et al.,

2019). In short, life-long, holistic, contextual and preventive are

important characteristics of LDC (Savickas et al., 2009;Venter

and Maree, 2020). LDC aims to help clients reshape their lives

through narration based on personal needs, interests, abilities and

experiences (Maree, 2020b). Clients can rewrite their narrative

identity, clarify their self-concept, establish life goals and explore

possible life tracks (Savickas et al., 2009;Cardoso et al., 2019;

Maree, 2020a).

HOW TO USE LIFE DESIGN

COUNSELING?

Life Design Counseling is an emerging method of career

intervention. The research on LDC is mainly divided into three

categories: process intervention, process evaluation, and results

evaluation. According to these three categories, some pioneer

scholars have made a comprehensive evaluation system of LDC.

Frontiers in Psychology | www.frontiersin.org 2February 2022 | Volume 13 | Article 814458

fpsyg-13-814458 January 26, 2022 Time: 12:22 # 3

Wen et al. Life Design Counseling

Process Intervention

The process intervention on LDC mainly includes Guichard’s Life

and Career Design Dialogues (LCDD) and Career Construction

Interview (CCI) (Pouyaud et al., 2016;Barclay et al., 2019;Wong,

2021). My Career Story (MCS) is a written expression tool

developed based on CCI (Hartung and Santilli, 2018). In addition,

Interpersonal process recall (IPR) is an important auxiliary

method of LDC that is usually used to explore the experience

of clients in the process of counseling dialogue (Cardoso et al.,

2016a). Written exercises, career collages, and career portfolios

are often combined with CCI creatively(Barclay, 2019).

Speciﬁcally, LCDD is a psychological counseling and

intervention method which focuses on the dialogue between

client and counselors to help the client understand the self-

construction of subjective identity (Guichard, 2016b;Pouyaud

et al., 2016). The dialogue took place in one-on-one interviews

between clients and counselors lasting several weeks (Pouyaud

et al., 2016). CCI designed by Savickas (2011) is widely used

which is usually divided into three sessions (Cardoso et al., 2018).

The ﬁrst session is Career Construction Interview (Cardoso

et al., 2018). After the client and counselor discussed their

expectations for intervention, LDC focused on ﬁve aspects: role

models; favorite magazines, TV programs, etc.; current favorite

story; Motto to oneself; early recollections (Taylor et al., 2016;

Taylor and Savickas, 2016). For example, clients are required

to tell real or ﬁctional role models. Counselors can discuss the

similarities and diﬀerences of role models with clients, so as

to reﬂect the clients’ self-concept, including how to treat their

own identity and values. Interested magazines and TV programs

reﬂect the relationship between clients’ interests and working

environment. During the second session, counselors assist the

client in constructing a story, which aims to provide continuity

for the fragments of the client’s life in the previous stage. In the

third session, the focus of the intervention is to assist the client to

connect life themes with career planning (Cardoso et al., 2018).

As an auxiliary means, IPR aims to explore the experience of

clients in the interview process. Clients are required to watch the

video replay of the interview. Counselors need to ask clients how

to experience the important moment in the video, help clients

analyze the emotional experience of the important moment, and

promote the reﬂection of clients in the process of interpersonal

interaction with counselors. In addition, in the process of CCI,

clients can create career collages based on paper pasting. The

counselor assists the client in creating “a collage” and takes the

collage as a tool for meaning construction. The client describes

what the client attaches importance to and desires in his life

by pasting images representing personal models, pictures of

his favorite magazines and TV programs, so as to help the

client imagine and design the future and obtain the power to

realize the future.

Process Evaluation

The purpose of the process evaluation of LDC is to ﬁnd out

what changes have taken place in the counseling process. The

process evaluation of LDC can be divided into the qualitative

evaluation and quantitative evaluation. The focus of qualitative

assessment is whether the clients have changed and what

changes have taken place in the LDC process. To analyze the

changes of clients, researchers also used quantitative methods to

process evaluation.

Speciﬁcally, Innovative moments (IMs) are usually used

as a sign of changes in clients (Cardoso et al., 2020). At

present, for the qualitative evaluation of IMs in LDC, the

representative is the innovative moments coding system (IMCS)

developed by Cardoso et al. IMCS is a tool for tracking

IMs. IMs in the counseling process reﬂects the non-linear

change of the clients, which is often accompanied by the

circular ambivalence reﬂecting the self motivation of the clients.

Researchers often use IMCS together with qualitative tools

to evaluate the changes in clients’ ambivalence during the

counseling process. Representative tools include Return to the

Problem Coding System (RPCS) and Ambivalence Coding

System (ACS) (Da Silva et al., 2020).The former refers to that

the client immediately returns after generating the innovation

moment and re emphasizes the problematic self narration, which

weakens the signiﬁcance of IM. The latter is often used to identify

clients who immediately turn to problematic self narration after

describing the moment of innovation.

Many researchers have conducted comprehensive research

on LDC combined both qualitative evaluation and quantitative

evaluation. Giving full play to the advantages of qualitative

evaluation and quantitative evaluation, some scholars also try to

develop a comprehensive evaluation of LDC. For instance, Maree

Career Matrix (MCM) is used to measure the clients’ interest

and conﬁdence in the successful pursuit of a variety of careers

(Maree and Taylor, 2016). MCM with 19 categories showed

good psychometric characteristics during the standardization

process. In terms of the comprehensive evaluation, MCM

shows good psychometric characteristics in the standardization

process (Morgan and Ferreira, 2019). In addition, Maree

(2007) developed the South African version of the qualitative

questionnaire of Career Interest Proﬁle (CIP) (Di Fabio and

Maree, 2013), trying to construct narrative information from

individual career stories.

Results’ Evaluation

Results’ evaluation is to compare the status of clients after

counseling with those before counseling to ﬁnd out the

diﬀerences. From the aspect of the quantitative evaluation,

researchers found that the clients’ career uncertainty and

career pending decreased, while their preparation for academic

specialty and career decision-making, professional self-eﬃcacy,

career planning, professional identity, and professional control

trajectory, life satisfaction increased (Barclay and Stoltz, 2016a,b;

Nota et al., 2016;Cardoso et al., 2018;Di Maggio et al., 2021b).

The above research shows that through the intervention of

LDC, individuals can more accurately evaluate themselves, gather

occupational information, select goals, make plans for the future,

have more certainty about their career development, and improve

their evaluation of their own quality of life. Still, there exist some

discrepancies among diﬀerent research. Some scholars found that

career adaptability improved, and others assert that this variable

did not change (Cardoso et al., 2018;Rudolph et al., 2019). These

studies suggest that the impact of LDC on an individual’s career

Frontiers in Psychology | www.frontiersin.org 3February 2022 | Volume 13 | Article 814458

fpsyg-13-814458 January 26, 2022 Time: 12:22 # 4

Wen et al. Life Design Counseling

TABLE 1 | How the LDC is operated.

Number Category Illustration

1 Process

Intervention

Main methods 1. Career Construction Interview

(CCI)

2. My Career Story (MCS)

3. Life and Career Design

Dialogues (LCDD)

4. Interpersonal process recall

(IPR)

Additional ways 1. Written Exercises

2. Career Collages

3. Career Portfolios

2 Evaluation Process Evaluation 1. Innovative Moments Coding

System (IMCS)

2. Return to the Problem Coding

System (RPCS)

3. Ambivalence coding system

(ACS)

4. Career Interest Proﬁle (CIP)

5. Maree Career Matrix (MCM)

Results’ Evaluation 1. Vocational Certainty Scale

(VCS)

2. The Career Maturity

Inventory–Form C (CMI-FC)

3. Career Decision Self-Efﬁcacy

Scale–Short Form (CDSE-SF)

4. The Career Adapt-abilities

Scale(CAAS)

adaptability is a complex issue, and more studies are still needed

to examine the eﬀects of interventions in more detail. In general,

how the LDC is operated can be shown in Table 1.

WHAT ARE THE CHALLENGES OF LIFE

DESIGN COUNSELING?

Lack of Attention to Clients From

Multiple Backgrounds and Professional

Counselors

On the one hand, LDC needs to pay attention to the psychological

needs of more diverse cultural background groups, because the

counseling needs of diﬀerent types of clients are unique and

diﬀerent (Sampaio et al., 2021). Scholars have accumulated some

group LDC research in the early stage, such as the research

on college students (Pordelan et al., 2018), middle and high

school students (Nota et al., 2016), Italian entrepreneurs (Di

Fabio and Maree, 2012), etc. However, these groups are still

limited, and the research group based on LDC needs to be

further expanded. For example, LDC may become an eﬀective

way to help vulnerable groups develop their career. On the

other hand, the eﬀective implementation of LDC is inseparable

from professional counselors. The lack of professional LDC

counselors may bring the following negative eﬀects: it is diﬃcult

for clients and counselors to establish a working alliance

and sort out ambivalence through coherent narration and re

conceptualization, then it is hard for clients to draw a life portrait

and formulate a plan for moving forward (Cardoso et al., 2016a,b;

Barclay, 2019). In addition, long-term intervention is an

important guarantee for the eﬀectiveness of LDC (Di Fabio

and Maree, 2012;Nota et al., 2016). If there is no professional

LDC counselors, the continuous participation of clients may be

reduced (Barclay and Stoltz, 2016a). Therefore, in the practice of

LDC, it is particularly necessary to cultivate a large number of

professional counselors who are familiar with the operation mode

of LDC.

Lack of Diversiﬁed Methods in the

Intervention Process

The intervention methods for LDC are still not diversiﬁed

enough. The typical intervention methods for LDC only include

CCI, MCS, LCDD, etc. The combination of LDC and other

practical methods can give better play to the function of career

counseling. For example, in view of the close relationship between

career and psychosocial issues, how to combine LDC and

psychotherapy to solve more complex life problems of clients

is a topic worthy of attention (Cardoso, 2016). In addition, the

creative use of LDC is still a problem that needs the attention

of career practitioners. Speciﬁcally, how to combine intervention

methods of methods with art (e.g., painting) to help clients better

narrate, reconstruct their own life themes, and ﬁnd the power

for change is a issue that needs to be solved urgently (Barclay,

2019). LDC practitioners need to creatively develop intervention

methods, so that they can be applied to a wider range of

clients, help them express themselves through narration, generate

motivation for substantial change, discover life themes, ﬁnd life

goals and plan their own future. In general, counselors need

to creatively and ﬂexibly use CCI and other major intervention

methods to help clients build their future careers.

Lack of Diversiﬁed Evaluation Methods

Scholars lack research on the diversiﬁcation of LDC in the

existing research, especially in its evaluation method. First, the

evaluation object is not comprehensive enough. Current research

mainly focuses on the counseling eﬀect on clients in LDC,

while neglecting the diﬀerent inﬂuence from counselors who

plays the same important roles as well. Although clients are

an important part of the counseling subject, the evaluation of

counselors has a speciﬁc value, and researchers lack attention

to the growth of counselors’ counseling ability (Storlie et al.,

2017). The second is the evaluation method: the current research

mainly includes qualitative evaluation, quantitative evaluation,

and comprehensive evaluation (McMahon et al., 2003;Di Fabio

and Maree, 2013). However, these evaluation methods are limited

and the evaluation content is not rich. It seems that the evaluation

of LDC can be further improved by referring to other counseling

and evaluation methods as well as other evaluation contents.

FUTURE RESEARCH TRENDS

Although LDC is a relatively new way of career counseling and

opens up new ideas for counselors, the theoretical research,

practical research, and the integration of theory and practice

of LDC still need to be further concerned and promoted by

Frontiers in Psychology | www.frontiersin.org 4February 2022 | Volume 13 | Article 814458

fpsyg-13-814458 January 26, 2022 Time: 12:22 # 5

Wen et al. Life Design Counseling

TABLE 2 | Future research trends of LDC.

Number Item

1. Strengthen the in-depth

study of clients and

counselors

1. More vulnerable groups

2. The general population with ﬁner classiﬁcation

3. Improve the quality of counselors

4. Increase the number of counselors

5. Create a standardized process for counselor

training

6. Establishment a working alliance between client

and counselor

2. Extended intervention

Methods

1. Pay attention to the cutting-edge trends in the

ﬁeld of counseling methods

2. Combine the methods in LDC with art therapy

methods

3. Online and ofﬂine intervention

3. Improve intervention

evaluation methods

1. Learn from other disciplines and develop more

process assessment tools, such as big data,

NVivo, etc.

2. From the perspective of personality

characteristics, such as career optimism,

mindfulness, gratitude, benevolence, etc.

3. From the perspective of organizational contexts,

such as decent work, job satisfaction, job

burnout, thriving at work, reemployment, etc

4. Clarify the relationship between counselors and

researchers in the evaluation process and focus

on the evaluation of counselors

researchers. The main trends of future development of LDC are

shown in Table 2.

Strengthen the Needs Analysis of Clients

and the Professional Training of

Counselors of Life Design Counseling

Scholars need to create diversiﬁed counseling methods to

broaden the application scope of LDC. On one hand, scholars

especially need to pay more attention to the following groups:

(a) vulnerable groups, such as the unemployed, people with

disabilities, people who have experienced breakups or divorces,

cancer patients, aging Chinese parents who lose their only child,

patients with depression, survivors of family violence, refugees,

ex-oﬀenders, solitary person, the bereaved and the low-income

people in remote areas (Nota et al., 2014;Li, 2018;Bergeron

et al., 2021;Di Maggio et al., 2021a;Maree, 2021) and (b) the

general population with ﬁner classiﬁcation, such as public or

private primary school students in diﬀerent countries or regions,

middle school students from diﬀerent regions and nationalities,

high school students with diﬀerent school characteristics, college

students of diﬀerent majors, workers of diﬀerent occupations

(Cardoso et al., 2016a;Maree, 2020a). Researchers may help

these people achieve better career development by designing

diversiﬁed LDC studies. On the other hand, more professional

LDC counselors need to be trained, including improving the

quality of counselors, increasing the number of counselors,

and forming a complete and standardized process for training

counselors, to serve more people who need LDC help (Di Fabio

and Maree, 2012;Venter and Maree, 2020;Wen et al., 2020a).

Researchers and practitioners need to pay particular attention to

the formation and maintenance of a good relationship between

counselors and clients, because the establishment of a working

alliance between them is an important guarantee for eﬀective

LDC (Cardoso et al., 2016a, 2021;Tian et al., 2020).

Increase Multi-Dimensional and

Diversiﬁed Intervention Process

Methods of Life Design Counseling

In order to solve the lack of diversiﬁed LDC intervention

methods, career counseling practitioners can try the following

paths. First, LDC counselors can integrate other counselor

theories with LDC and ﬁnd the coincidence points between

some cutting-edge psychology and pedagogy theories and

LDC, so as to create a diversiﬁed counseling model and

serve more clients with diﬀerent needs. For example, future

researchers can try to combine PERMA (Positive Emotions,

Engagement, Relationships, Meaning, Achievements) theory in

positive psychology with LDC, so as to jointly promote the

positive change of clients (Carreno et al., 2021). Second, the

existing LDC methods can be presented in a diversiﬁed and

creative way in order to promote the self-expression and

reﬂexivity of clients. Future career researchers and practitioners

can try to combine the methods in LDC with art therapy

methods such as painting, music, dance, and writing, so as

to serve a more diverse group of individual or group clients.

Last but not least, scholars may be able to conduct online and

oﬄine LDC through computers, mobile phones, telephones, and

e-mails (Savarese et al., 2020;Carbone et al., 2021), and other

ﬂexible forms and combined with other emerging psychological

counseling methods. In general, career practitioners need to

create diversiﬁed LDC intervention methods in the future to help

clients through narrative construction, deconstruction and joint

construction of their personal career stories, ﬁnd the core life

theme, and carry out a meaningful life.

Strengthen the Research on the

Evaluation Objects and Methods of Life

Design Counseling

Future research on LDC needs to enhance the research on

the evaluation objects and methods of LDC. Strengthening the

research on the evaluation is mainly reﬂected in the following

three points: Firstly, researchers need to further broaden the

application scope of the above evaluation methods and also create

more process evaluation methods suitable for LDC because of

big data, Nvivo to serve a wider group (Alam, 2020;Wen et al.,

2020a). Secondly, in terms of personality characteristics, it should

be paid more attention to such things as the meaning of life,

mindfulness, and optimism (Ginevra et al., 2018;Wen et al.,

2020b). From the perspective of the organizational situation,

it should focus on exploring the inﬂuence mechanisms caused

by self and the outside world, such as professional identity,

job satisfaction, social support, and job burnout (Chen et al.,

2020;Shen et al., 2021). Finally, strengthen the research on

counselors in the evaluation of LDC. Scholars need to further

enrich the evaluation objects of LDC and pay attention to

the evaluation of counselors. Previous studies usually focused

Frontiers in Psychology | www.frontiersin.org 5February 2022 | Volume 13 | Article 814458

fpsyg-13-814458 January 26, 2022 Time: 12:22 # 6

Wen et al. Life Design Counseling

on the evaluation of clients. However, it needs to pay more

attention to the uniqueness of counselors and the important

value of counselors’ personal growth to the counseling eﬀect,

such as counselors’ cultural background, personality, value, and

helping counselor trainees’ personal growth and professional

development through supervision (Prosek and Michel, 2016). At

the same time, since researchers and counselors may be the same

group in some studies, future researchers need to further clarify

the responsibilities and authorities of counselors and researchers

in the LDC evaluation process (Tian et al., 2020).

CONCLUSION

Life Design Counseling is a new form of career counseling in the

VUCA era. The emergence and development of LDC are based

on Self-Construction Theory and Career Construction Theory.

At present, the use of LDC mainly includes process intervention,

process evaluation, and results evaluation. To sum up, this study

found some challenges of LDC, including lack of attention to

clients from multiple backgrounds and professional counselors,

lack of diversiﬁed methods in the intervention process, and lack

of diversiﬁed evaluation methods. Therefore, this study came up

with some suggestions about how to deal with those challenges

as follows: Firstly, facing the changeable environment, future

research should focus on strengthen the needs analysis of clients

and the professional training of counselors of LDC. Secondly,

for more individuals or groups to obtain career development

through LDC, it is necessary to increase the research on the multi-

dimensional and diversiﬁed intervention methods of LDC in

future research. Finally, future researchers need to pay attention

to strengthening the research on the evaluation objects and

methods of LDC to ensure further prove the eﬀect of LDC on

individuals or groups.

AUTHOR CONTRIBUTIONS

All authors participated in the study design. YW and HC wrote

the ﬁrst draft. KL and FL modiﬁed the manuscript.

FUNDING

This research was funded by Jiangsu Province General research

project of philosophy and Social Sciences in universities

(grant number 2021SJA0488), Jiangsu Province University’s

Advantageous Discipline Construction Project (grant number

PAPD), and China Scholarship Council Projects (grant

number 202006860031).

REFERENCES

Alam, M. K. (2020). A systematic qualitative case study: questions, data collection,

NVivo analysis and saturation. Qual. Res. Organ. Manag. Int. J. [Online ahead

of print]. doi: 10.1108/QROM-09- 2019-1825

Barclay, S. R. (2019). Creative use of the career construction interview. Career Dev.

Q. 67, 126–138. doi: 10.1002/cdq.12176

Barclay, S. R., and Stoltz, K. B. (2016a). The life design group: a case study vignette

in group career construction counseling. J. Stud. Aﬀ. Res. Pract. 53, 78–89.

doi: 10.1080/19496591.2016.1087859

Barclay, S. R., and Stoltz, K. B. (2016b). The life-design group: a case study

assessment. Career Dev. Q. 64, 83–96. doi: 10.1002/cdq.12043

Barclay, S. R., Stoltz, K. B., and Clemente, A. (2019). Assessing manifest interests

within the career construction interview. Int. J. Educ. Vocat. Guid. 19, 455–473.

doi: 10.1007/s10775-019- 09388-1

Bergeron, C. D., Boolani, A., Jansen, E. C., and Smith, M. L. (2021). Practical

solutions to address COVID-19-related mental and physical health challenges

among low-income older adults. Front. Public Health 9:674847. doi: 10.3389/

fpubh.2021.674847

Canzittu, D. (2020). A framework to think school and career guidance in a VUCA

world. Br. J. Guid. Counc. [Online ahead of print].

Carbone, G. A., Zarfati, A., Nicoli, M. S., Paulis, A., Tourjansky, G., Valenti, G.,

et al. (2021). Online psychological counselling during lockdown reduces anxiety

symptoms and negative aﬀect: insights from Italian framework. Clin. Psychol.

Psychother. [Online ahead of print]. doi: 10.1002/cpp.2608

Cardoso, P. (2016). Integrating life-design counseling and psychotherapy:

possibilities and practices. Career Dev. Q. 64, 49–63. doi: 10.1002/cdq.12040

Cardoso, P., Campos, R., Taveira, M. D. C., and Silva, F. (2021). Client factors and

working alliance: a preliminary study in career construction counselling. Br. J.

Guid. Counc. 1–9. doi: 10.1080/03069885.2021.2009769 [Epub ahead of print].

Cardoso, P., Duarte, M. E., Gaspar, R., Bernardo, F., Janeiro, I. N., and Santos,

G. (2016a). Life design counseling: a study on client’s operations for meaning

construction. J. Vocat. Behav. 97, 13–21. doi: 10.1016/j.jvb.2016.07.007

Cardoso, P., Gonçalves, M. M., Duarte, M. E., Silva, J. R., and Alves, D. (2016b).

Life design counseling outcome and process: a case study with an adolescent.

J. Vocat. Behav. 93, 58–66. doi: 10.1016/j.jvb.2016.01.002

Cardoso, P., Janeiro, I. N., and Duarte, M. E. (2018). Life design counseling

group intervention with Portuguese adolescents. J. Career Dev. 45, 183–196.

doi: 10.1177/0894845316687668

Cardoso, P., Savickas, M. L., and Gonçalves, M. M. (2019). Innovative moments in

career construction counseling: proposal for an integrative model. Career Dev.

Q. 67, 188–204. doi: 10.1002/cdq.12190

Cardoso, P. M., Savickas, M. L., and Gonçalves, M. M. (2020). Facilitating narrative

change in career construction counseling. J. Career Dev. 48, 1–14. doi: 10.1177/

0894845319898872

Carreno, D. F., Eisenbeck, N., Pérez-Escobar, J. A., and García-Montes, J. M.

(2021). Inner harmony as an essential facet of well-being: a multinational study

during the COVID-19 pandemic. Front. Psychol. 12:648280. doi: 10.3389/fpsyg.

2021.648280

Chen, H., Liu, F., Pang, L., Liu, F., Fang, T., Wen, Y., et al. (2020). Are you

tired of working amid the pandemic? The role of professional identity and

job satisfaction against job burnout. Int. J. Environ. Res. Public Health 17:9188.

doi: 10.3390/ijerph17249188

Da Silva, C. S. C., Teixeira, M. A. P., Cardoso, P., Fernández-Navarro, P.,

Gonçalves, M. M., and Duarte, M. E. (2020). Innovative moments and narrative

change in career counselling: a case study. Int. J. Educ. Vocat. Guid. 20, 635–652.

doi: 10.1007/s10775-020- 09422-7

Di Fabio, A. (2014). Life designing in 21st century: using a new, strengthened career

genogram. J. Psychol. Afr. 20, 381–384. doi: 10.1080/14330237.2010.10820389

Di Fabio, A., and Maree, J. G. (2012). Group-based life design counseling in an

Italian context. J. Vocat. Behav. 80, 100–107. doi: 10.1016/j.jvb.2011.06.001

Di Fabio, A., and Maree, J. G. (2013). Career counselling: the usefulness of the

career interest proﬁle (CIP). J. Psychol. Afr. 23, 41–49. doi: 10.1080/14330237.

2013.10820592

Di Maggio, I., Ginevra, M. C., Santilli, S., and Nota, L. (2021a). Elementary school

students’ attitudes towards peers with disabilities: the role of personal and

contextual factors. J. Intellect. Dev. Disabil. 1–9. doi: 10.3109/13668250.2021.

1920091 [Epub ahead of print].

Di Maggio, I., Montenegro, E., Little, T. D., Nota, L., and Ginevra, M. C. (2021b).

Career adaptability, hope, and life satisfaction: an analysis of adults with and

without substance use disorder. J. Happiness Stud. doi: 10.1007/s10902-021-

00405-1 [Epub ahead of print].

Frontiers in Psychology | www.frontiersin.org 6February 2022 | Volume 13 | Article 814458

fpsyg-13-814458 January 26, 2022 Time: 12:22 # 7

Wen et al. Life Design Counseling

Ginevra, M. C., Magnano, P., Lodi, E., Annovazzi, C., Camussi, E., Patrizi, P.,

et al. (2018). The role of career adaptability and courage on life satisfaction in

adolescence. J. Adolesc. 62, 1–8. doi: 10.1016/j.adolescence.2017.11.002

Guichard, J. (2009). Self-constructing. J. Vocat. Behav. 75, 251–258. doi: 10.1016/j.

jvb.2009.03.004

Guichard, J. (2016a). Life-designing counseling: a comparison of the career

construction and self-construction approaches. Psychol. Française 61, 15–29.

doi: 10.1016/j.psfr.2013.03.002

Guichard, J. (2016b). Reﬂexivity in life design interventions: comments on life and

career design dialogues. J. Vocat. Behav. 97, 78–83. doi: 10.1016/j.jvb.2016.08.

001

Guichard, J., Pouyaud, J., de Calan, C., and Dumora, B. (2012). Identity

construction and career development interventions with emerging adults.

J. Vocat. Behav. 81, 52–58. doi: 10.1016/j.jvb.2012.04.004

Hartung, P. J., and Santilli, S. (2018). My career story: description and initial

validity evidence. J. Career Assess. 26, 308–321. doi: 10.1177/1069072717692980

Li, Y. (2018). Who will care for the health of aging Chinese parents who lose their

only child? A review of the constraints and implications. Int. Soc. Work 61,

40–50. doi: 10.1177/0020872815603782

Maree, J. G. (2007). The Career Interest Proﬁle, 1st Edn. Randburg: Jopie van

Rooyen & Partners.

Maree, J. G. (2014). Career construction with a gay client: a case study. Br. J. Guid.

Counc. 42, 436–449. doi: 10.1080/03069885.2014.886670

Maree, J. G. (2019). Group career construction counseling: a mixed−methods

intervention study with high school students. Career Dev. Q. 67, 47–61. doi:

10.1002/cdq.12162

Maree, J. G. (2020b). Career construction counselling aimed at enhancing the

narratability and career resilience of a young girl with a poor sense of self-worth.

Early Child Dev. Care 190, 2646–2662. doi: 10.1080/03004430.2019.1622536

Maree, J. G. (2020a). Connecting conscious knowledge with subconscious advice

through career construction counselling to resolve career choice indecision.

S. Afr. J. Educ. 40, 1–13. doi: 10.15700/saje.v40ns1a1863

Maree, J. G. (2021). The outcomes of a mixed-methods, innovative group life

design intervention with unemployed youths. Int. J. Educ. Vocat. Guid. doi:

10.1007/s10775-021- 09514-y [Epub ahead of print].

Maree, J. G., and Taylor, N. (2016). Development of the Maree Career Matrix:

a new interest inventory. S. Afr. J. Psychol. 46, 462–476. doi: 10.1177/

0081246316641558

McMahon, M., Patton, W., and Watson, M. (2003). Developing qualitative career

assessment processes. Career Dev. Q. 51, 194–202. doi: 10.1002/j.2161-0045.

2003.tb00601.x

Morgan, B., and Ferreira, J. C. (2019). An investigation of the psychometric

properties of the Maree Career Matrix. S. Afr. J. Psychol. 49, 206–218. doi:

10.1177/0081246318782188

Nota, L., Santilli, S., and Soresi, S. (2016). A life-design-based online career

intervention for early adolescents: description and initial analysis. Career Dev.

Q. 64, 4–19. doi: 10.1002/cdq.12037

Nota, L., Soresi, S., Ferrari, L., and Ginevra, M. C. (2014). Vocational designing

and career counseling in Europe: challenges and new horizons. Eur. Psychol. 19,

248–259. doi: 10.1027/1016-9040/a000189

Pordelan, N., Sadeghi, A., Abedi, M. R., and Kaedi, M. (2018). How online career

counseling changes career development: a life design paradigm. Educ. Inf.

Technol. 23, 2655–2672. doi: 10.1007/s10639-018-9735-1

Pouyaud, J., Bangali, M., Cohen-Scali, V., Robinet, M. L., and Guichard, J. (2016).

Exploring changes during life and career design dialogues. J. Vocat. Behav. 97,

3–12. doi: 10.1016/j.jvb.2016.07.008

Prosek, E. A., and Michel, R. E. (2016). Transformative learning for counselor

trainees: implementation of the multicultural immersion experience model. Int.

J. Adv. Couns. 38, 61–76. doi: 10.1007/s10447-015-9256-1

Rudolph, C. W., Zacher, H., and Hirschi, A. (2019). Empirical developments in

career construction theory. J. Vocat. Behav. 111, 1–6. doi: 10.1016/j.jvb.2018.12.

003

Sampaio, C., Cardoso, P., Rossier, J., and Savickas, M. L. (2021). Attending to

clients’ psychological needs during career construction counseling. Career Dev.

Q. 69, 96–113. doi: 10.1002/cdq.12252

Savarese, G., Curcio, L., D’Elia, D., Fasano, O., and Pecoraro, N. (2020). Online

university counselling services and psychological problems among Italian

students in lockdown due to Covid-19. Healthcare 8:440. doi: 10.3390/

healthcare8040440

Savickas, M. (2011). Career Counseling. Washington, DC: American Psychological

Association.

Savickas, M. L. (2008). David V. Tiedeman: engineer of career construction. Career

Dev. Q. 56, 217–224. doi: 10.1002/j.2161-0045.2008.tb00035.x

Savickas, M. L. (2012). Life design: a paradigm for career intervention in the 21st

century. J. Couns. Dev. 90, 13–19. doi: 10.1111/j.1556-6676.2012.00002.x

Savickas, M. L. (2016). Reﬂection and reﬂexivity during life-design interventions:

comments on career construction counseling. J. Vocat. Behav. 97, 84–89. doi:

10.1016/j.jvb.2016.09.001

Savickas, M. L., Nota, L., Rossier, J., Dauwalder, J., Duarte, M. E., Guichard, J., et al.

(2009). Life designing: a paradigm for career construction in the 21st century.

J. Vocat. Behav. 75, 239–250. doi: 10.1016/j.jvb.2009.04.004

Savickas, M. L., Porfeli, E. J., Hilton, T. L., and Savickas, S. (2018). The student

career construction inventory. J. Vocat. Behav. 106, 138–152. doi: 10.1016/j.jvb.

2018.01.009

Savickas, M. L., and Pouyaud, J. (2016). Life design: a general model for career

intervention in the 21st century. Psychol. Fr. 61, 5–14. doi: 10.1016/j.psfr.2013.

11.003

Shen, X., Gu, X., Chen, H., and Wen, Y. (2021). For the future sustainable career

development of college students: exploring the impact of core self-evaluation

and career calling on career decision-making diﬃculty. Sustainability 13:6817.

doi: 10.3390/su13126817

Storlie, C. A., Baltrinic, E. R., Mostade, S. J., and Darby, T. (2017).

Professional functioning: exploring developmental narratives of counselor

trainees. Counselor Educ. Supervision 56, 225–240. doi: 10.1002/ceas.12074

Taylor, J. M., and Savickas, S. (2016). Narrative career counseling: my career story

and pictorial narratives. J. Vocat. Behav. 97, 68–77. doi: 10.1016/j.jvb.2016.07.

010

Taylor, N., Taylor, N., Maree, J. G., and Maree, J. G. (2016). Development of

the Maree career matrix: a new interest inventory. S. Afr. J. Psychol. 46,

462–476.

Tian, X., Hou, Z. J., Wang, D., Savickas, S., Chang, X., Cao, Y., et al. (2020).

Counselor actions to facilitate client change during life−design counseling.

Career Dev. Q. 68, 48–62. doi: 10.1002/cdq.12212

Venter, C. J., and Maree, J. G. (2020). Life-design counselling for survivors of

family violence in resource-constrained areas. S. Afr. J. Educ. 40, 1–12. doi:

10.15700/saje.v40ns2a1845

Wehmeyer, M. L., Nota, L., Soresi, S., Shogren, K. A., Morningstar, M. E., Ferrari,

L., et al. (2018). A crisis in career development: life designing and implications

for transition. Career Dev. Transit. Except. Individ. 42, 179–187. doi: 10.1177/

2165143417750092

Wen, Y., Chen, H., Li, K., and Gu, X. (2020a). The challenges of life design

counseling in the times of the coronavirus pandemic (COVID-19). Front.

Psychol. 11:1235. doi: 10.3389/fpsyg.2020.01235

Wen, Y., Chen, H., Pang, L., and Gu, X. (2020b). The relationship between

emotional intelligence and entrepreneurial self-eﬃcacy of Chinese vocational

college students. Int. J. Environ. Res. Public Health 17:4511. doi: 10.3390/

ijerph17124511

Wong, S. (2021). Life themes of a slash worker in Hong Kong identiﬁed through

a career construction interview. Br. J. Guid. Counc. 49, 524–532. doi: 10.1080/

03069885.2021.1921697

Conﬂict of Interest: The authors declare that the research was conducted in the

absence of any commercial or ﬁnancial relationships that could be construed as a

potential conﬂict of interest.

Publisher’s Note: All claims expressed in this article are solely those of the authors

and do not necessarily represent those of their aﬃliated organizations, or those of

the publisher, the editors and the reviewers. Any product that may be evaluated in

this article, or claim that may be made by its manufacturer, is not guaranteed or

endorsed by the publisher.

under the terms of the Creative Commons Attribution License (CC BY). The

use, distribution or reproduction in other forums is permitted, provided the

original author(s) and the copyright owner(s) are credited and that the original

publication in this journal is cited, in accordance with accepted academic practice.

No use, distribution or reproduction is permitted which does not comply with

these terms.

Frontiers in Psychology | www.frontiersin.org 7February 2022 | Volume 13 | Article 814458

OPINION

published: 03 March 2022

doi: 10.3389/fpsyg.2022.856949

Frontiers in Psychology | www.frontiersin.org 1March 2022 | Volume 13 | Article 856949

Edited by:

Casandra Isabel Montoro,

University of Jaén, Spain

Reviewed by:

Dominika Kwasnicka,

SWPS University of Social Sciences

and Humanities, Poland

Jennifer Paternostro,

The University of Iowa, United States

Igor Marchetti,

University of Trieste, Italy

Radion Svynarenko,

The University of Tennessee, Knoxville,

United States

*Correspondence:

Robyn E. Metcalfe

robynm@uoregon.edu

Specialty section:

This article was submitted to

Psychology for Clinical Settings,

a section of the journal

Frontiers in Psychology

Received: 17 January 2022

Accepted: 07 February 2022

Published: 03 March 2022

Citation:

Metcalfe RE (2022) Open Science

Within Pediatric Medical and Mental

Health Systems: Practical

Considerations for Behavioral Health

Researchers.

Front. Psychol. 13:856949.

doi: 10.3389/fpsyg.2022.856949

Open Science Within Pediatric

Medical and Mental Health Systems:

Practical Considerations for

Behavioral Health Researchers

Robyn E. Metcalfe*

Department of Counseling, Family and Human Services, University of Oregon, Eugene, OR, United States

Keywords: research methods, open science, reproducibility, preregistration, pediatric psychology

INTRODUCTION

The open science movement is an epistemological movement toward increasing accessibility

of data and research processes in order to improve the quality and reproducibility of science

(Hesse, 2018). A number of practices have been proposed in association with open science,

including journal clubs, systems for project workﬂow, sharing code, sharing data, use of preprints,

pre-registering studies, open-access publishing, an increased focus on statistical power, and greater

transparency in data analysis documentation (Allen and Mehler, 2019; Kathawalla et al., 2021). In

particular, researchers promoting open science practices focus on the beneﬁts to scientiﬁc inquiry

by improving reproducibility, improving public access to scientiﬁc ﬁndings, and allowing for more

active collaboration and building on previous work, among other beneﬁts (Hesse, 2018).

As technology increasingly improves connection between researchers, the pressure to engage in

speciﬁc open-science practices, such as sharing raw data with other researchers, has increased. Still,

data sharing in psychological research remains relatively rare due to a range of potential barriers

(Houtkoop et al., 2018). Researchers identify practical concerns about the process of data-sharing,

concerns about anonymity of participants, and concerns about being credited on subsequent

research projects as speciﬁc concerns related to data-sharing (Cheah et al., 2015).

Further, implementation of open science practices has been uneven across settings, with

pediatric medical and mental health system-based research lagging behind other psychology

research contexts. Fewer than 10% of studies from these hospital-based settings engage in even low-

stakes open science practices such as publishing supplemental code or promptly publishing results

in clinical trials registries (Sixto-Costoya et al., 2020; Kadakia et al., 2021), let alone high-stakes

or potentially diﬃcult-to-implement practices. This problem is compounded by low standards for

adherence to best practices by journals in these ﬁelds (Gardener et al., 2022). Despite clear potential

for clinical beneﬁt to pediatric populations when open science practices are utilized (Rubinstein

et al., 2020), there has been little exploration or published discussion about how barriers are

perceived by researchers in these systems or discussion about how previously established solutions

to barriers might apply to these researchers.

Children and families within systemic pediatric medical or mental health systems are considered

sensitive populations, deserving of particular ethical consideration in research and clinical

processes (American Psychological Association, 2017), and their behavioral health data is often

sensitive and/or identiﬁable. Goodie et al. (2013) advise that, for sensitive populations, researchers

and other stakeholders should consider the balance between providing high quality, patient-

centered services while collecting sound scientiﬁc data and decreasing the likelihood of adverse

outcomes. Further, grant-funded research in pediatric medical and mental health systems is

often costly and high-stakes, leading to particularly robust discussion of risk-management for

Metcalfe Open Science Within Health Systems

participants in these settings (Wendler, 2006). These concerns

about open science practices by behavioral health researchers are

legitimate and worthy of serious consideration. Considerations

for engaging in open science practices in these settings may

be diﬀerent than in other psychological research. Due to the

potential need for additional protections, consideration of open

science methods in the context of these populations is warranted.

FACTORS

Consideration: Protection of Participant

Privacy

Concerns about participant welfare rightfully concern behavioral

health researchers in pediatric settings who are considering open

science practices. Sharing research data may increase the risk

of compromising conﬁdential information about participants.

While many of these risks are preventable, adequate protection

requires both preventative planning and clear informed consent

by participants (Alter and Gonzalez, 2018). Some research, such

as longitudinal studies or research that includes geographic

tracking, may be particularly diﬃcult to de-identify without

compromising the research value of the dataset. Medical and

mental health data has long been known to be at particular

risk of re-identifying participants when combined with basic

demographic data (Sweeney, 2000). In pediatric settings, where

demographic information may include both children and their

parents or guardians, the potential to re-identify participants is

additionally heightened.

Data collected in medical or mental health settings also

likely falls under other legislation, such as the Health Insurance

Portability and Accountability Act (HIPAA) in the United States,

the General Data Protection Regulation (GDPR) in the European

Union, or the Personal Information Protection and Electronic

Documents Act (PIPEDA) in Canada, and any data-sharing plan

must consider this explicitly.

Actions for Researchers

Of course, researchers should ensure that suﬃcient processes

are in place to ensure participant safety and appropriately

document this with their Institutional Review Board. Researchers

should engage in robust risk-management strategies regardless

of whether or not they intend to share their data (e.g., data

encryption for personal health data, splitting datasets to separate

identiﬁable information from ID variables, etc.). Depending on

the types of information gathered by researchers, partial datasets

may be realistic for some projects, with only non-sensitive

data available for data sharing. Example repositories include

Open Science Framework (OSF), Mendeley Data, Figshare,

and Zenodo.

However, in the context of sensitive personal health data

that makes data sharing infeasible, researchers may also beneﬁt

from a selective use of open science practices. For example,

in situations where data sharing is not feasible (e.g., due to

patient conﬁdentiality concerns), researchers may still choose to

engage with other open science practices such as preregistration,

reproducible code, and preprints. Additionally, publishing

adequate summary statistics provides readers greater opportunity

to evaluate a research paper, particularly in contexts where

original datasets have not be shared. For example, in addition

to means and standard deviations, a researcher might choose

to publish variance-covariance matrices of all variables in their

work. Researchers may also choose to publish their statistical

script alongside research papers to improve reader conﬁdence in

their results and facilitate researcher learning (Mair, 2016).

Kathawalla et al. (2021) identify the least diﬃcult to most

diﬃcult open science practices as: (1) Starting an open science

journal club, (2) Developing an open science-friendly project

workﬂow, (3) Posting pre-prints, (4) Using reproducible code,

(5) Sharing data, (6) Transparent manuscript writing, (7)

Preregistering research, and (8) Publishing registered reports.

Given that this list was not designed with protected health data

in mind, research in pediatric medical and mental health systems,

this diﬃculty order may be somewhat diﬀerent, with data sharing

likely causing the most signiﬁcant concern, due to the potentially

sensitive nature of these datasets. Even in cases where no data

sharing is feasible, researchers can likely engage in some of the

other open science practices endorsed by Kathawalla et al. (2021).

Consideration: Propriety of Information

Data collected in pediatric medical and mental health settings

are often collected at great monetary and temporal expense.

Researchers in these settings may intend to produce a large

number of publications with a single dataset and worry that

procedures such as data sharing may result in reduced ability to

maximize their collected materials for their own research output.

Concerns about the ability to adequately utilize one’s own dataset,

procedure, or planned analyses before opening it up to other

researchers are valid.

Actions for Researchers

In addition to choosing some of the other open science

practices discussed above, researchers may consider using project

management tools that enable timelocks (e.g., preregistering a

hypothesis and not publicly releasing it until there has been

suﬃcient time to complete all relevant work) to avoid the

possibility of getting “scooped,” a colloquial term referring to

instances where one researcher publishes a novel project in

advance of another researcher, who was already working on

this idea. If data sharing is possible, it may be reasonable to

require that researchers request access in order to access the full

dataset. In these cases, sharing variables or codebook material

may be more feasible. Notably, there are likely advantages of

data sharing, when feasible to do so, for overall research impact.

For example, linking data in a repository increases researchers’

citation count by 25% (Colavizza et al., 2020). Researchers may

also wish to license their data, code, and materials to ensure that

it is used appropriately (e.g., a Creative Commons license).

Consideration: Data Ownership and

Permissions

Ownership and legal issues may disrupt the possibility of

data sharing even among willing participants. In many cases,

the grantee for ownership of research data is the university

or research hospital, not the individual researcher (Alter and

Frontiers in Psychology | www.frontiersin.org 2March 2022 | Volume 13 | Article 856949

Metcalfe Open Science Within Health Systems

Gonzalez, 2018). Other times, research may be published that is

clearly not within the purview of the project leader. For example,

a secondary data analysis by a graduate student, completed with

permission from the data steward, is almost certain to be derived

from a dataset that is not the intellectual property of the student.

In these cases, a researcher may be unsure of their ability to share

their data, or simply unable to do so due to restrictions imposed

by the data owner.

Previously-collected data in pediatric settings may also lack

adequate informed consent procedures to facilitate data-sharing,

or complications may arise in creating pathways for sharing

sensitive or protected information in these settings. These

concerns may be more likely to come up for large scale

studies operating out of medical centers within major pediatric

health systems.

Actions for Researchers

Researchers in leadership roles may ﬁnd value in drafting

guidelines for open science practices within their organization in

collaboration with their research ethics team to improve access to

data sharing options. For example, if researchers have historically

used consents that prohibit data sharing, a relatively common

practice in large healthcare facilities, choosing a more ﬂexible

option with informed consent of participants may allow for more

open science-related workability in future projects.

Importantly, Campbell et al. (2019) highlight the

complications of setting up adequate informed consent

procedures for highly sensitive information or for populations

that have historically been wronged by psychological or medical

researchers. They recommend a tiered consent approach to

allow parents or guardians to choose the level of data sharing

that feel comfortable with. Notably, children in pediatric

settings are not able to oﬀer informed consent to participation

in research. Based on developmental stage, children may be

able to provide assent in conjunction with parental consent.

Thus, child health and mental health data deserves additional

considerations related to consent when children reach the

age of majority, such as policies around re-contacting to

establish consent for data-sharing when children become adults

(Brothers et al., 2014). Despite these potential complications,

setting up standard procedures to facilitate the conversations

around open science practices can improve the reproducibility of

future research.

DISCUSSION

Generally, the beneﬁts of open science practices have the

potential to be robust. However, legitimate concerns may limit

the ability of some researchers operating within healthcare

settings to engage in certain practices such as data sharing.

Although steps can be taken to encourage data sharing when

feasible, importantly, there are a range of potential actions in

addition to data-sharing that researchers can take to improve

both the accessibility and reproducibility of science. This article

aims to encourage interested but hesitant researchers in pediatric

medical and mental health systems to consider manageable steps

in the direction of open science.

In particular, researchers are encouraged to:

1. Explicitly consider which open science practices are feasible

for their project and speciﬁc datasets with large health and

mental health systems.

2. Proactively plan for large scale research trials to ensure

rigorous data reporting standards while balancing the rights

of children and families who participate in research.

3. Connect with peers to create a culture that encourages open

science within pediatric-focused institutions including active

supports for researchers.

Future research should examine setting-speciﬁc barriers or

hesitations to engaging with open science practices as well

as facilitators and the impacts of potential interventions (e.g.,

creating an open science journal club) on the reproducibility of

research within institutions.

AUTHOR CONTRIBUTIONS

The author conﬁrms being the sole contributor of this work and

has approved it for publication.

REFERENCES

Allen, C., and Mehler, D. M. A. (2019). Open science challenges,

beneﬁts and tips in early career and beyond. PLoS Biol. 17, e3000246.

doi: 10.1371/journal.pbio.3000246

Alter, G., and Gonzalez, R. (2018). Responsible practices for data sharing. Am.

Psychol. 73, 146–156. doi: 10.1037/amp0000258

American Psychological Association (2017). Ethical Principles of Psychologists

and Code of Conduct. Available online at: https://www.apa.org/ethics/code/

(accessed January 16, 2022).

Brothers, K. B., Lynch, J. A., Aufox, S. A., Connolly, J. J., Gelb, B.

D., Holm, I. A., et al. (2014). Practical guidance on informed consent

for pediatric participants in a biorepository. Mayo Clinic Proc. 89,

1471–1480. doi: 10.1016/j.mayocp.2014.07.006

Campbell, R., Goodman-Williams, R., and Javorka, M. (2019). A trauma-

informed approach to sexual violence research ethics and open

science. J. Interpers. Violence 34, 4765–4793. doi: 10.1177/08862605198

71530

Cheah, P., Tangseefa, D., Somsaman, A., Chunsuttiwat, T., Nosten, F., Day, N.,

et al. (2015). Perceived beneﬁts, harms and views about how to share data

responsibly: a qualitative study of experiences with and attitudes towards data-

sharing among research staﬀ and community representatives in Thailand.

J. Empirical Res. Human Res. Ethics 10, 278–289. doi: 10.1177/15562646155

92388

Colavizza, G., Hrynaszkiewicz, I., Staden, I., Whitaker, K., and McGillivray,

B. (2020). The citation advantage of linking publications to

research data. PLoS ONE 15, e0230416. doi: 10.1371/journal.pone.

0230416

Gardener, A. D., Hicks, E. J., Jacklin,C., Tan, G., Cashin, A. G., Lee, H., et al. (2022).

Open science policies of medical and health sciences journals before and during

the COVID-19 pandemic: a repeat cross-sectional study. MedRxIv [Preprint].

doi: 10.1101/2022.01.26.22269868

Goodie, J. L., Kanzler, K. E., Hunter, C. L., Glotfelter, M. A., and Bodart,

J. J. (2013). Ethical and eﬀectiveness considerations with primary care

behavioral health research in the medical home. Famil. Syst. Health 31, 86–95.

doi: 10.1037/a0031852

Frontiers in Psychology | www.frontiersin.org 3March 2022 | Volume 13 | Article 856949

Metcalfe Open Science Within Health Systems

Hesse, B. W. (2018). Can psychology walk the walk of open science? Am. Psychol.

73, 126–137. doi: 10.1037/amp0000197

Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V. M., Nichols, T.

E., and Wagenmakers, E. (2018). Data sharing in psychology: a survey

on barriers and preconditions. Adv. Methods Pract. Psychol. Sci. 2018,

70–85. doi: 10.1177/2515245917751886

Kadakia, K. T., Beckman, A. L., Ross, J. S., and Krumholz, H. M. (2021).

Leveraging open science to accelerate research. New Engl J Med. 384,

e61. doi: 10.1056/NEJMp2034518

Kathawalla, U., Silverstein, P., and Syed, M. (2021). Easing into open science:

a guide for graduate students and their advisors. Collabra Psychol. 7,

18684. doi: 10.1525/collabra.18684

Mair, P. (2016). Thou shalt be reproducible! A technology perspective. Front.

Psychol. 7, 1079. doi: 10.3389/fpsyg.2016.01079

Rubinstein, Y. R., Robinson, P. N., Gahl, W. A., Avillach, P., Baynam, G.,

Cederroth, H., et al. (2020). The case for open science: rare diseases. JAMIA

Open, 3, 472–486. doi: 10.1093/jamiaopen/ooaa030

Sixto-Costoya, A., Aleixandre-Benavent, R., Lucas-Dominguez, R., and Vidal-

Infer, A. (2020). The emergency medicine facing the challenge of open science.

Data 5, 28. doi: 10.3390/data5020028

Sweeney, L. (2000). Simple Demographics Often Identify People Uniquely.

Carnegie Mellon University, Technical Report. Available online at: https://

dataprivacylab.org/projects/identiﬁability/paper1.pdf (accessed January 16,

2022).

Wendler, D. (2006). Three steps to protecting pediatric research participants from

excessive risks. PLoS Clin. Trials 1, e25. doi: 10.1371/journal.pctr.0010025

Conﬂict of Interest: The author declares that the research was conducted in the

absence of any commercial or ﬁnancial relationships that could be construed as a

potential conﬂict of interest.

Publisher’s Note: All claims expressedin this article are solely those of the authors

and do not necessarily represent those of their aﬃliated organizations, or those of

the publisher, the editors and the reviewers. Any product that may be evaluated in

this article, or claim that may be made by its manufacturer, is not guaranteed or

endorsed by the publisher.

terms of the Creative Commons Attribution License (CC BY). The use, distribution

or reproduction in other forums is permitted, provided the original author(s) and

the copyright owner(s) are credited and that the original publication in this journal

is cited, in accordance with accepted academic practice. No use, distribution or

reproduction is permitted which does not comply with these terms.

Frontiers in Psychology | www.frontiersin.org 4March 2022 | Volume 13 | Article 856949

Frontiers in Psychology | www.frontiersin.org 1 March 2022 | Volume 13 | Article 834419

ORIGINAL RESEARCH

published: 07 March 2022

doi: 10.3389/fpsyg.2022.834419

Edited by:

Carmen María Galvez-Sánchez,

University of Jaén, Spain

Reviewed by:

Claudia I. Iacob,

University of Bucharest, Romania

Serena Grumi,

Neurological Institute Foundation

Casimiro Mondino (IRCCS), Italy

*Correspondence:

Alberto Romano

alberto.romano01@universitadipavia.it

†These authors have contributed

equally to this work

Specialty section:

This article was submitted to

Psychology for Clinical Settings,

a section of the journal

Frontiers in Psychology

Received: 13 December 2021

Accepted: 24 January 2022

Published: 07 March 2022

Citation:

Zwilling M, Romano A, Favetta M,

Ippolito E and Lotan M (2022) Impact

of a Remotely Supervised Motor

Rehabilitation Program on Maternal

Well-Being During the COVID-19

Italian Lockdown.

Front. Psychol. 13:834419.

doi: 10.3389/fpsyg.2022.834419

Impact of a Remotely Supervised

Motor Rehabilitation Program on

Maternal Well-Being During the

COVID-19 Italian Lockdown

MotiZwilling

1†, AlbertoRomano

2,3,4*†, MartinaFavetta

3, ElenaIppolito

5 and MeirLotan

6,7†

1 Department of Economics and Business Administration, Ariel University, Ari’el, Israel, 2 Department of Health System

Management, Ariel University, Ari’el, Israel, 3 Movement Analysis and Robotics Laboratory, Unit of Neurorehabilitation,

Department of Neuroscience, Bambino Gesù Children’s Hospital, Rome, Italy, 4 Centro AIRETT Ricerca e Innovazione (CARI),

Research and Innovation Airett Center, Verona, Italy, 5 SMART Learning Center, Milan, Italy, 6 Department of Physiotherapy,

Ariel University, Ari’el, Israel, 7 Israeli Rett Syndrome National Evaluation Team, Sheba Hospital, Ramat-Gan, Israel

COVID-19 Lockdown was particularly challenging for most mothers of people with

intellectual disabilities, including those with Rett syndrome (RTT), leading to feelings of

abandonment from healthcare services of their children. Within those days, telerehabilitation

has represented a valid alternative to support physical activity and treatment, supporting

parents in structuring their children’s daily routine at home. This article aims to describe

the well-being level of two groups of mothers of girls and women with RTT who were

involved in a home-based remotely supervised motor rehabilitation program, respectively,

before and during the COVID-19 Italian lockdown. Forty participants with classic RTT

were recruited before the lockdown and randomly assigned to two groups that performed

the intervention immediately before (Group1) and during (Group2) the lockdown,

respectively. The intervention included an individualized daily physical activity program

carried out for 12 weeks by participants’ parents and fortnightly supervised throughout

Skype contacts to plan, monitor, and accommodate individual activities in the participant’s

life at home. The short form Caregivers Well-Being Scale was collected for the mothers

in each group12 weeks before intervention (T1), at intervention initiation (T2), immediately

after intervention termination (T3), as well as at 12 weeks after intervention termination

(T4). Mothers of participants in the Group1 showed a stable level of well-being across

all four evaluations with a slight improvement during the lockdown, without signicant

change. Similarly, the well-being level of mothers in the Group2 showed a statistically

signicant increase in their well-being between T2 and T3 (during the lockdown) and its

reduction to the pre-intervention level between T3 and T4 (after the lockdown). The results

suggest that the lockdown did not negatively affect the participants’ mothers’ well-being,

leading to its improvement. Moreover, the proposed intervention could have supported

the mothers in managing the new daily routine at home, positively affecting maternal

well-being.

Keywords: Rett Syndrome, telerehabilitation, parental well-being, exercise therapy, parents, COVID-19, home

exercise program

Frontiers in Psychology | www.frontiersin.org 2 March 2022 | Volume 13 | Article 834419

Zwilling et al. Telerehabilitation Maternal Well-Being COVID-19 Lockdown

INTRODUCTION

Rett Syndrome (RTT) is a rare neurological disorder observed

mainly in females (Amir etal., 1999, 2000). RTT is characterized

by normal birth and apparently normal psychomotor development

during the rst 6–18 months of life (Epstein, 1995). e disorder’s

trademark is the repetitive stereotypical hand movements

appearing aer the child has entered the typical regression

phase of RTT. Additional characteristics at the breakthrough

of the disease include autistic-like behavior, panic-like attacks,

breathing disturbances, sleeping problems, gait ataxia and apraxia,

and acquired microcephaly (Hagberg et al., 1983). Aer this

period of rapid functional deterioration, the disorder progresses

relatively stable, although the child with RTT could develop

dystonia and musculoskeletal deformities (Sponseller, 2001) as

she grows old. Seizures occur in 50–85% (Epstein, 1995; Glaze

et al., 2010) of individuals with RTT. Females with RTT typically

survive into adulthood, and their estimated life expectancy is

49 years, suggesting the need for life-long support. Due to the

complexity of the disorder, RTT is a particularly challenging

disorder even for highly trained rehabilitation therapists due

to its severity and complexity (Perks et al., 1994).

e challenges presented by the individual with RTT and

their expected longevity necessitate constant individual

adaptations of intervention in all educational and therapeutic

areas. e literature reported that brain development continues

aer the onset of RTT (Kaufmann et al., 2005). Moreover,

the cognitive and communicative abilities of the individual

with RTT do not deteriorate over the years. erefore, setting

proper and achievable rehabilitative, educational and

communicational goals is vital for the individual with RTT

(Cass et al., 2003).

Within the last decade, researchers published new therapeutic

strategies for treating each aspect of the disability associated

with RTT. ese developments include technologically supported

strategies for cognitive (Fabio etal., 2020, 2021; Dovigo etal.,

2021), communicative (Fabio etal., 2013, 2018b; Stasolla etal.,

2014; Vessoyan etal., 2018; Fabio, 2019), and motor (Romano

etal., 2020, 2021; Lotan etal., 2021a,b) rehabilitative evaluations

and interventions. Technology-supported strategies resulted in

adequate support for skills acquisition and improvements in

all these elds. e technology-supported strategies available

for people with RTT mainly refer to telerehabilitation

strategies. Telerehabilitation, also known as net therapy, virtual

rehabilitation, mobile rehabilitation, or remote rehabilitation

(RR), delivers services over distance via technologies addressing

therapeutic issues by presenting remote services to individuals

with DD (Caprì etal., 2021). Clinically, the term ‘telerehabilitation’

encompasses a range of rehabilitation and habilitation services

that include evaluation, assessment, monitoring, prevention,

intervention, supervision, education, consultation, and coaching.

Technologies used to deliver rehabilitation and habilitation

services may incorporate but are not limited to video and

audio conferencing, chat messaging, wearable technologies,

sensor technologies, patient portals or platforms, mobile health

applications, virtual reality, robotics, and therapeutic gaming

technologies (Richmond etal., 2017). e benets of telehealth

include improved access to rehabilitation services and specialists,

preventing unnecessary delays in care and support (Cason and

Cohn, 2014).

Life-long rehabilitation interventions should be performed

to maintain individuals with RTT as functional as possible

along with their life span, accompanying the person through

the disease evolution (George et al., 1988; Lotan and Hanks,

2006; Lotan et al., 2010) and overcoming all the medical and

functional impediments mentioned above. Yet, the complexity

of the disorder necessitates treatment delivery to the individual

with RTT and her family with utmost prociency and intensity.

Previous ndings suggest that an intensive intervention

program can enhance the abilities of individuals with RTT

in numerous areas, such as learning skills (Elefant, 2001;

Koppenhaver etal., 2001a,b; Elefant and Wigram, 2005), new

skills (Demeter, 2000; Jacobsen et al., 2001; Leonard et al.,

2001), literacy (Koppenhaver et al., 2001a; Hetzroni et al.,

2002; Fabio et al., 2013), communication abilities (Sigafoos

et al., 2000; Elefant, 2001; Hetzroni et al., 2002; Lindberg,

2006; Wine, 2009; Fabio etal., 2018a), cognitive abilities (Fabio

etal., 2011, 2016, 2019, 2020, 2021), manual abilities (Sullivan

et al., 1995; Pizzamiglio et al., 2008), osteoporosis (Zysman

et al., 2006), functional abilities (Lotan et al., 2004; Downs

etal., 2008; Maciques Rodríguez and Lotan, 2011; Lotan etal.,

2021a), orthopedic issues (Legrand etal., 1997; McClure etal.,

1998; Elefant and Lotan, 2004; Lotan et al., 2004, 2005;

Maciques Rodríguez and Lotan, 2011), and sensory issues

(Pizzamiglio et al., 2008; Drobnyk et al., 2019). In addition,

such programs have been found eective when suggested to

individuals with RTT of all ages (Gillberg, 1997; Demeter,

2000; Lindberg, 2006).

As the primary caregivers, the parents and family play a

vital role in ensuring the health and well-being of children.

e focus of health and developmental services has evolved

from a child-centered, traditional “medical” model to a family-

centered “developmental” model. Bly suggests that as: “e

more involved the family becomes, the more consistent

therapeutic management becomes” (Bly, 1999). In this framework,

those who coordinate services consider the essential contributions

of the family unit and the ability of families to adapt to new

challenges. e pediatric healthcare professional must involve

family members in all areas of planning, delivery, and evaluation

of health and developmental services. Communication between

parents and pediatric healthcare professionals should beopen,

comprehensible, culturally sensitive, and sincere (Ramey and

Ramey, 1996). erefore, the family’s involvement is a

fundamental resource. However, in planning the involvement

of parents in the therapeutic path of their child with disability,

therapists cannot ignore the impact of that decision on the

parents’ stress level (Smith et al., 2001). Stressful experiences

are considered as person-environment interactions, in which

both external stressors and available psychological,

socioeconomic, and cultural resources inuence the person’s

appraisal of the stressor (Lazarus and Folkman, 1984). Stress

was found as a multi-dimensional construct that can

be operationalized in various ways, oen related to having

children with developmental disabilities (Crnic et al., 2009).

Zwilling et al. Telerehabilitation Maternal Well-Being COVID-19 Lockdown

Frontiers in Psychology | www.frontiersin.org 3 March 2022 | Volume 13 | Article 834419

Furthermore, it is known that stress levels negatively inuence

the well-being of parents of children with developmental

disabilities (Cramm and Nieboer, 2011). In general, it can

be determined that exposure to prolonged or chronic

psychological distress has resulted in adverse health outcomes,

ranging from inadequate sleep to negative psychological and

physiological well-being (Lee, 2013).

It is known that the process of caring for a child with

RTT is, by itself, a dicult task for the family, requiring the

activation of psychological, social, and economic resources,

and may represent a source of stress for family members

(Downs and Leonard, 2016). e risk of increased parental

stress level arises from the beginning of the disorder when

parents must face the onset of RTT in a child who has

shown an initial normal development. Moreover, additional

medical comorbidities can arise during childhood, requiring

increased attention and constant adaptation of the parents’

caring strategies, all within a framework of uncertainty regarding

their daughter’s life expectancy (Mori et al., 2019). Recent

articles explored the stress levels of parents of girls with RTT.

Authors reported an increased stress level in this population

of parents, with mothers showing a higher stress level than

fathers (Perry etal., 1992; Pari etal., 2020). Moreover, parental

stress level and health-related quality of life were found to

correlate with the degree of clinical severity (Sarajlija et al.,

2013; Pari et al., 2020). Nevertheless, the stress level appears

higher for those parents who had taken care of a girl with

RTT for many years and were, on average older, showing a

cumulative eect (Mori etal., 2019; Pari etal., 2020). Despite

this regression in stress levels, several authors agree that many

families with RTT have found functional strategies to cope

with the strains of their particular parenting (Perry et al.,

1992; Mori et al., 2019). e existing literature agrees in

arming the need for solid support for these parents, which

should follow them across their whole parental experience

(Perry etal., 1992; Pari etal., 2020). Considering the delicate

balance of the need for an intense, long-lasting structured

rehabilitative program while maintaining a healthy familial

infrastructure, it is necessary to investigate the eects of

parental involvement in their daughters’ therapeutic activities

on their well-being.

In February 2020, rst COVID-19 patients were recognized

in Italy. From that moment, the number of aected people

started to increase all over the country and abroad and, on

March 11th, 2020, the world health organization declared the

COVID-19 global pandemic (World Health Organization,

2020). On the same date, the Italian government ordered the

rst national lockdown closing the majority of the working

places, schools, and places of worship and prohibiting all

recreational activities. During the lockdown, people have been

conned to their houses with the possibility to go out for

only a few specic reasons. According to the literature, many

Italian parents experienced signicant parenting-related

exhaustion and well-being levels reduction with increased

anxiety, with mothers more severely aected (Cusinato etal.,

2020; Marchetti etal., 2020; Bentenuto etal., 2021). Coherent

results were found related to mothers of people with disabilities.

An Italian study reported increased sleep diculties and

feelings of reduced external support for mothers of children

with X-Fragile syndrome. However, their perceived self-ecacy

as caregivers did not change during the lockdown (Di Giorgio

et al., 2021).

Feelings of abandonment, powerlessness, and fear for their

and their children’s health have been reported in numerous

studies that have explored the experiences of mothers of both

children and adults with intellectual disabilities during the

lockdown (Asbury et al., 2021; Embregts et al., 2021; Patel

etal., 2021; Rogers etal., 2021). Moreover, in an online survey

administered with 527 Italian parents of children with autism

spectrum disorder, 93.9% of parents reported that the COVID-19

emergency was a challenging period with diculties in free

time management and structured activities development. e

same study reported a low level of specialist support for medical

and behavioral needs (Colizzi et al., 2020). Coherently, Bova

et al. (2021) analyzed data from interviews conducted with

parents of 514 Italian children with neurological disorders who

reported that 67.7% of programmed specialist appointments

were canceled during the lockdown and about half (49.5%)

of children who usually received rehabilitation continued

it remotely.

On the other hand, a study from the United Kingdom

reported that well-being in families of children with intellectual

disabilities measured before and during/immediately aer the

lockdown did not show any dierence. e authors concluded

that the general belief of the lockdown’s negative impact on

these families could be as straightforward as expected (Bailey

et al., 2021). Sporadic similar results were available in other

articles that reported benets for parents due to changes in

their daily routines. For these parents, the house connement

represents a possibility to spend more time with their family,

strengthening the parent-child relationship and contributing

to their well-being (Bentenuto et al., 2021; Embregts et al.,

2021; Rogers et al., 2021).

Reports suggest the need for support by most parents of

people with intellectual disabilities. Remote rehabilitation has

represented a valuable strategy to continue the treatment and

support the families in those dicult days (Assenza et al.,

2020; Rabanifar and Abdi, 2021). However, patients who have

received remote rehabilitation intervention perceived dierences

in the quality of service and preferred traditional in-person

treatment to service delivery via remote rehabilitation (Milani

et al., 2021). is article aims to describe the well-being level

of two groups of mothers of girls and women with RTT who

were involved in a home-based remotely supervised motor

rehabilitation program, respectively, before and during the

COVID-19 Italian lockdown.

MATERIALS AND METHOD

Ethical Approval

Declaration of Helsinki principles was followed while conducting

this research. e research protocol was approved by Ariel

University IRB (AU-HEA-ML-20190326-1) and explained to

Zwilling et al. Telerehabilitation Maternal Well-Being COVID-19 Lockdown

Frontiers in Psychology | www.frontiersin.org 4 March 2022 | Volume 13 | Article 834419

all participants’ parents, who signed an informed consent form

aer understanding the protocol and agreeing to participate.

Participants

Participants were recruited from the Italian Rett syndrome

Association database (AIRett). To be included in the current

investigation, participants must be genetically diagnosed with

classic RTT and reside with their parents. Moreover, participants’

parents must have approved their availability to follow a physical

rehabilitation activity program with their daughter for one

non-consecutive hour a day, 5 days a week, for 3 months.

Candidates for participation were excluded if they presented

a neurological or psychomotor developmental decit comorbidity

other than RTT. All candidates were approved to participate

by a specialist doctor certied (due to unstable health conditions,

e.g., ongoing or recurrent infections, severe gastrointestinal

disorders, and drug-resistant epilepsy with multiple

daily seizures).

Study Design

A randomized between-groups comparison design was applied.

Participants were randomly divided into Group1 and Group2.

Both groups followed the same A-A-B-A-A protocol. Group2

started the intervention program 3 months aer Group1. Letter

“A” represents the evaluation meetings that occurred 3 months

apart. Letter “B” represents the intervention phase. Before the

COVID-19 outbreak, when the study was planned, the authors

intended to analyze the intervention eects on participants in

Group1 and Group2 together. However, the Italian COVID-19

lockdown occurred before the beginning of the Group 2

intervention phase occasioning the opportunity to investigate

the intervention impact on maternal well-being within the

context of the lockdown limitations. erefore, for the current

article, the research protocol planned before the lockdown

initiation was not changed due to the COVID-19 outbreak,

but the collected data were analyzed for the two groups

independently and compared. e research timetable is outlined

in Figure 1.

e Participatory Action Research (PAR) method was used.

Within the PAR model, participants (in this case, the family

members, caregivers, and referral therapists of the person with

RTT) are involved throughout the whole research process

(Mackenzie et al., 2012). ey are asked to participate with

the researchers in planning their involvement in the research,

identifying problems, and nding solutions to disentangle them

through the direct application of research ndings in a practical

context (Ison, 2014). PAR design is characterized by three

recurring stages: inquiry, action, and reection (Kemmis and

McTaggart, 2005). In the current article, the PAR process was

developed through the implementation of numerous and

continuous cycles of:

• Assessment (of each participant’s therapeutic needs);

• Mutual goal attainment (individualized therapeutic goals

were set in mutual agreement with each participant’s parents

taking into consideration the participant need and her parents

expectation, needs, and availability within the

family’s framework);

• Action (the family members implemented the program);

• Reection (each program was discussed with participant’s

family members within the bi-weekly supervised Skype

meetings between a trained supervisor and parents); and

• Evaluation (if necessary, the program was modied and

re-implemented).

e theoretical framework of PAR allows the participants

to actively participate in the research protocol construction

they are involved in resulting in greater intervention eectiveness

(Newig, 2007). us, PAR researchers collaborate with the

participants to obtain changes and identify new solutions

according to their family’s needs and desires (Ozanne and

Saatcioglu, 2008). Even considering the lower level of evidence

of PAR design compared to the randomized controlled trial,

it best-suited studies that provide a protocol requiring a high

level of collaboration between researchers and participants, as

is the present investigation (Mackenzie etal., 2012). Moreover,

it is the researchers’ intention that the participants’ parents

fully understand the process of planning positive physical

activity for their daughter (similar to those proposed in the

present research) becoming able to develop them even aer

the research concludes to maintain long-lasting well-being and

functional status of the participants with RTT.

Procedure

All participants’ parents signed an informed consent explaining

the research protocol before the research initiation. Each

participant was evaluated four times at their house. Evaluation

meetings were conducted with 3 months apart from each other

(± 1 month, T1–T4). Participants’ mothers’ well-being was

assessed during each evaluation meeting.

FIGURE1 | Research timetable. T1–T4 represent the evaluations meeting occurred with 3 months apart. BP, Baseline Phase; IP, Intervention Phase; and WOP,

Wash-Out Phase.

Zwilling et al. Telerehabilitation Maternal Well-Being COVID-19 Lockdown

Frontiers in Psychology | www.frontiersin.org 5 March 2022 | Volume 13 | Article 834419

During the rst evaluation meeting (T1), anamnestic

information related to participants’ medical situation was collected

(e.g., epilepsy, bone density condition, feeding problems, sleep

disorders, and other comorbidities typically associated with

RTT) together with the extent and type of ongoing physical

therapeutic interventions (such as physiotherapy, hippotherapy,

hydrotherapy, and others). At the end of this meeting, the

researchers compiled a dra of the individualized rehabilitation

goals in collaboration with participants’ parents to bepursued

in the intervention phase. e objectives set followed the

SMART principles to best suit each participant’s potential.

Under SMART principles, goals should bespecic, measurable,

attainable, realistic, and timely (Bovend’Eerdt etal., 2009). No

change was made between the rst and second evaluation

meetings (baseline phase) to the participants’ and their parents’

daily routines.

In the second evaluation meeting (T2), identied treatment

goals were re-discussed with participants’ parents, and corrections

were applied if needed. At the end of the second (T2) evaluation

meeting, an individualized motor activity program was designed

for each participant and discussed with the family. Each activity

program was developed to pursue the identied intervention

goals through easily constructed physical activities that did

not require professional competencies to be carried out.

Aer the second evaluation meeting termination, participants’

parents were asked to conduct the activities of their daughter’s

program within her daily routine, for one non-continuous hour

a day, 5 days a week, for 3 months. Researchers helped the

parents to plan the therapeutic activities development within

their weekly routines and habits. Aer 2 weeks from the delivery

of the program, necessary for familiarization with the activities,

each family started to participate in remotely conducted

supervision meetings with a researcher experienced in the

rehabilitation of people with RTT. Supervision meetings occurred

fortnightly and lasted for a maximum of 1 h. A videoconference

platform (Skype) was used to conduct the supervision meetings.

ese meetings continued until the end of the intervention

phase. e rst supervision meeting was mainly dedicated to

clarifying any doubts about the practical execution of the

activity programs, guiding the parents in the activity development,

and modifying them if necessary to meet the families’ needs.

e subsequent supervision meetings aimed at supporting the

adherence to the program and their execution by answering

parents’ questions, adapting the program to emerging needs,

solving problems, rearranging the timetable, adapting the

proposed exercises, evaluating and sharing the achievement of

objectives and, if necessary, setting new goals. Due to the

experience of the researchers in RTT, the weekly meetings did

not refer only to the program implementation but also to

general issues related to RTT, raised by the parents.

Aer the 3 months of programs implementation, the third

evaluation meeting (T3) was conducted. In this meeting, the

level of achievement of identied goals was assessed, and the

parents’ considerations relating to the intervention phase

were collected.

Between the third (T3) and fourth (T4) evaluation meeting

(wash-out phase), the remote supervisions were suspended,

and parents were informed that they could, at their discretion,

continue or interrupt the program. e researcher took this

choice to support the continuation of the activities learned,

promoting the participants’ physical tness.

Measures

RTT Severity Level

Rett Assessment Rating Scale (RARS; Fabio et al., 2005) was

administered at T1 to assess the participants’ level of RTT

clinical severity. is is a 31-item scale aimed to score many

specic RTT phenotypic characteristics. Each item is scored

on a four-point scale from one to four. Intermediate scores

(e.g., score of 2.5) can be attributed to the subject to make

the scale more sensitive to the typical variability of RTT. In

the theoretical framework of this scale, RTT severity is

conceptualized as a continuum between mild decit (lower

score) to severe symptoms (higher score; Vignoli etal., 2010).

RARS standardization procedure for the Italian population with

RTT was conducted involving a sample of 220 individuals

with RTT. Solid psychometric values were proved for this scale

(Fabio et al., 2005, 2014; Vignoli et al., 2010; Romano et al.,

2020). e results obtained at this scale will be only briey

discussed within the present article. e results obtained from

this scale will be presented to provide the reader with a more

precise description of the participants’ disease severity.

Mothers’ Well-Being

e change in the well-being level across the protocol was

evaluated with the short form of the Caregiver Well-Being

Scale (CWBS-SF; Tebb et al., 2013). is is a 16-items scale

targeted to address areas relevant to caregivers that allow

obtaining information related to their well-being. e scale

covers basic needs (meeting the biopsychosocial needs to sustain

life) and activities of daily living (regarding the implementation

of the biopsychosocial needs). A daily need is presented for

each item, and the parent is asked to assign a score based

on the level hefeels he has satised that need in the previous

3 months. e score is attributed on a ve-point Likert scale

from 1 (the need was never or almost never satised) to 5

(the need was almost always satised). e scores of each

item were averaged together for the subsequent analysis. is

scale showed an overall internal consistency of 0.83 (Tebb

etal., 2013). e CWBS-SF was administered with participants’

mothers only as few fathers participated in all the

evaluation sessions.

Statistical Analyses

e Shapiro-Wilk normality test was used to assess the

normality of the data distributions. As most analyzed data

sets were not normally distributed, the non-parametric statistic

was used to analyze the obtained results. Friedman’s test

was run to compare the CWBS-SF scores obtained from the

mothers in each group and all together at the four evaluation

points. Post-hoc analysis with Wilcoxon signed-rank tests

was conducted for pairwise comparisons. Mann-Whitney U

Test was used to compare the results obtained by Group 1

Zwilling et al. Telerehabilitation Maternal Well-Being COVID-19 Lockdown

Frontiers in Psychology | www.frontiersin.org 6 March 2022 | Volume 13 | Article 834419

with those achieved by Group 2 at each time point for all

the outcome measures. e Spearman rank correlation

coecient was used to explore the relations between

participants’ mothers’ well-being (CWBS-SF score) and

participants’ and their mothers’ age and level of participants’

clinical severity at T1 (RARS score). e threshold for

signicance for the analyses above has been assumed as

α = 0.05. No correction for multiple comparisons was applied

to avoid missing signicant results (Armstrong, 2014).

RESULTS

Main Descriptive Statistics

Forty-two families were involved in the rst evaluation (T1).

Two families (4.8%) did not complete the research protocol.

One drop-out happened in Group 1 and was due to health

problems of the participant’s mother that arose during the baseline

period. e other drop-out occurred in Group2 and concerned

a family living in a rural area with negative external involvement

by local healthcare professionals who gave them contradicting

advice about the proposed program for their child with RTT.

erefore, data of 40 participants with RTT were used for the

subsequent analysis. Participants’ and parents’ ages and RTT

severity levels measured with RARS were collected in Tab l e  1 .

At the rst evaluation meeting (T1), 11 participants were

younger than 10 years, 19 were aged between 10 and 20 years,

and 10 were older than 20 years. Seven participants attended

motor rehabilitative intervention for at least 4 h a week.

Twenty-six subjects attended such interventions between 1

and 3 h per week. Five participants were not involved in any

motor rehabilitative treatment. All the rehabilitative

interventions were suspended during the lockdown. All

participants resided at home with their parents. All the

participants and their parents were born in Italy. e

participants’ and parents’ daily routines and the amount of

parents’ working hours varied widely within our sample. In

three families, the parents were divorced, and the participants

lived with their mother. A weak correlation was found between

maternal well-being and both participants’ and mothers’ ages

at T1 (p = 0.001, rho = 0.491 and p = 0.006, rho = 0.429,

respectively), T2 (p = 0.005, rho = 0.433 and p = 0.032,

rho = 0.339, respectively), and T4 (p < 0.001, rho = 0.530 and

p = 0.011, rho = 0.398, respectively). No correlation emerged

between the maternal well-being level and participants’ RTT

severity level.

Differences Regarding Mothers’

Well-Being

CWBS-SF was assessed to investigate the impact of program

implementation on participants’ mothers’ well-being. Descriptive

statistics of CWBS-SF are collected in Tab le 1. e variation

of maternal well-being within the current project diered

consistently between Group 1 and Group 2 (see Figure 2).

However, when analyzed with the Mann-Whitney U Test,

no statistical dierence emerged between mothers in Group1

and Group 2. Mothers in the Group 1 showed a stable level

of well-being across all four evaluations with a slight improvement

of median well-being level between T1 and T4. ese changes

were not statistically signicant at any conducted analysis.

Conversely, the well-being level of mothers in the Group 2

showed a signicant change across the four evaluations at the

Friedman test (p = 0.012). Signicant increases in mothers’ well-

being were found between T1 and T2 and between T2 and

T3 (p = 0.002; p = 0.013, respectively), but a statistically signicant

reduction between T3 and T4 (p = 0.031) was found. Looking

at participants’ mothers altogether (see Figure3), the Friedman

test identied no signicant score change among the four

evaluation meeting. However, the maternal well-being showed

a signicant increment between T1 and T2 (p = 0.018) and

between T2 and T3 (p = 0.050) and then slightly reduced at

T4, but without reaching the statistical signicance with any

of the other evaluations.

Moreover, on average, the CWBS-SF score in our group

remained within the range described as acceptable for

families without children with disabilities (average: 3.6—

range: 2.7–4.9; Tebb et al., 2013). No correlation emerged

between the mothers’ well-being level and the other

outcome measures.

TABLE1 | Descriptive statistics of participants’ and parents’ age, RTT severity level (RARS), and CWBS-SF score.

Participants’ age Mothers’ age RARS score CWBS-SF

T1 T2 T3 T4

All participants

(No. 40)

Mean (SD) 15.7(9.7) 50.0(9.8) 67.3(9.8) 3.4(0.6) 3.5(0.6) 3.6(0.7) 3.5(0.8)

Median 13.3 48.4 67.8 3.3 3.5 3.8 3.5

Max–Min 40.3–2.8 75.1–29.1 82.5–45.5 4.9–2.2 5–2.4 4.8–2.1 4.8–2

Group1 (No. 17) Mean (SD) 16.4(7.9) 53.4(7.1) 66.5(10.7) 3.6(0.6) 3.6(0.6) 3.6(0.7) 3.7(0.7)

Median 13.3 51.4 68.0 3.6 3.7 3.8 3.8

Max–Min 38.2–5.4 67.5–39.5 82.5–45.5 3.6(0.6) 3.6(0.6) 3.6(0.7) 3.7(0.7)

Group2 (No. 23) Mean (SD) 15.1(11.0) 48.8(10.8) 67.8(9.2) 3.3(0.7) 3.4(0.6) 3.7(0.7) 3.3(0.8)

Median 13.3 46.7 67.0 3.1 3.4 3.9 3.3

Max–Min 40.3–2.8 75.1–29.1 82.5–51.5 3.3(0.7) 3.4(0.6) 3.7(0.7) 3.3(0.8)

SD, Standard deviation; RTT, Rett syndrome; RARS, Rett assessment rating scale; and CWBS-SF, Caregiver well-being scale—short form.

Zwilling et al. Telerehabilitation Maternal Well-Being COVID-19 Lockdown

Frontiers in Psychology | www.frontiersin.org 7 March 2022 | Volume 13 | Article 834419

DISCUSSION

is article described the well-being of mothers of girls with

RTT and how it was aected by their enrollment in their

daughter’s rehabilitation program during the COVID-19

lockdown in Italy. A motor activity program was given to

each involved family to be carried out at home and was

fortnightly remotely supervised through video calls. Parental

well-being was assessed before and aer treatment. However,

on March 9th, the Italian government established the rst

national lockdown to face the COVID-19 outbreak. is has

led to the interruption of most working and recreational activities

and to the obligation for all Italian citizens to stay at home

and go out only if strictly necessary. In this phase, rehabilitation

and assistance facilities for people with disabilities and schools

were also closed. e lockdown and the related restrictions

continued until May 4th, when most work activities were

resumed, but without the reopening of shops, restaurants, cafes,

and places of worship that occurred on May 18th. On the

same day, some, but not all, rehabilitation activities for people

with disabilities were also resumed. However, the limitations

to some recreational activities, such as cinemas and theaters,

and the attendance of summer camps for children continued

until June 11th, and school attendance did not start again

until September 2020. e restriction progression caused the

parents to spend 2 months at home with their families without

working activities. Subsequently, aer the recovery of the working

activity (May 4th), the parents’ situation has changed. ey

were asked to go back to work within a context of social

distancing, reduced availability of recreational activity, fear of

contagion, and with their child at home from school and

rehabilitation facilities.

In this study, two groups of people with RTT and their

families followed the same research protocol starting with

3 months apart. For this reason, the lockdown occurred in a

dierent phase of the protocol for the two groups. For Group1,

the restrictions began within the wash-out phase while, for

Group 2, the lockdown started during the intervention phase

(see Figure1). e mothers’ well-being scores across the four

evaluation meetings diered for the two groups. e mothers

in Group 1, aer stable well-being scores in the rst three

evaluation meeting, showed a slight increase at T4 (during

the lockdown). Similarly, the well-being of mothers in the

Group2 increased at baseline and, more markedly, during the

intervention phase, before going back to the pre-intervention

level at T4. ese results suggested that the lockdown did not

negatively aect the mothers’ well-being level but increased

it. is eect can be explained by the fact that the lockdown

situation allowed the mothers to spend more time with their

families in the absence of working activities. ese results are

correlated to the ndings of Bailey et al. (2021) and with

sporadic results from other studies (Bentenuto et al., 2021;

Embregts et al., 2021; Rogers et al., 2021).

Nevertheless, within the lockdown, mothers in Group 2

showed a more markedly improved well-being than those in

Group1. ese can beexplained by the presence of the remote

supervision meeting that occurred during the lockdown for

Group2 but before it for Group 1. e literature reports that

parents of children with intellectual disabilities suered the

reduction in healthcare professional support they received

through the lockdown (Asbury et al., 2021; Embregts et al.,

2021; Patel et al., 2021; Rogers et al., 2021). is did not

happen in the case of mothers in group 2 who received an

organized program to implement and constant follow-up talks

where they could unload their fears and concerns.

Furthermore, the positive correlation between mothers’ and

participants’ age and maternal well-being suggests that older

mothers have higher well-being levels than younger ones

contrasting with published literature (Perry et al., 1992; Pari

et al., 2020). e fact that both articles investigated parental

stress while our study focused on maternal well-being could

explain this dierence. However, both Perry et al. (1992) and

Pari et al. (2020) suggested that families with RTT frequently

nd suitable strategies to cope with the strains related to their

parental role. In this light, our ndings agree with the researches

mentioned above, as shown by the average well-being level of

parents in our groups that remained in the range reported

for parents without children with disabilities (Tebb etal., 2013).

Group 1

Group 2

June

2019

September

2019

December

2019

April

2020

July

2020

***

FIGURE2 | Caregivers’ Well-Being Scale—Short Form (CWBS-SF) score of

each group at each evaluation point. The box inferior limits represent rst

quartile of the distribution, and the upper limits the third quartile (median

excluded). The lines across the box show the median score of each group.

The crosses inside the boxes mark the mean value of each dataset. The

whiskers indicate the minimum and maximum distribution values (outliers

excluded). The dots above the upper and below the lower whiskers represent

the distribution outliers identied through Tukey’s method (data points that lie

above 1.5 times the interquartile range under the rst quartile or over the third

quartile). In June 2019, Group1 was evaluated for the rst time (T1). In

September 2019, the second evaluation was conducted for Group1 (T2),

and the initial assessment occurred for Group2 (T1). In December 2019,

Group1 was evaluated for the third time (T3) and Group2 for the second

time (T2). In April 2020, Group1 was assessed for the last time (T4), and the

third evaluation was conducted for Group2 (T3). For Group2, the last

assessment occurred in July 2020 (T4). The black arrow indicates the

initiation of the Italian COVID-19 national lockdown, and the grey arrow

indicates its end for a total of 2 months. *p ≤ 0.05.

Zwilling et al. Telerehabilitation Maternal Well-Being COVID-19 Lockdown

Frontiers in Psychology | www.frontiersin.org 8 March 2022 | Volume 13 | Article 834419

However, these results may also relate to the fact that all

participating families remained protected from the COVID-19

contagion and were nancially well established, thereby reducing

the burden associated with the pandemic.

e current investigation presents some limitations. First,

only one measure of maternal well-being was used. For more

solid results, more evaluation tools should beused to investigate

more dimensions of maternal well-being and stress level in the

future. Moreover, the amount of time each parent spent in their

daughter’s program was not collected, but weasked them never

to exceed 1 h of treatment per day for 5 days a week. Furthermore,

a relatively small sample was enrolled in this study, challenging

the external validity of the obtained results. Additionally, each

group’s participants’ data were analyzed together, preventing

further ne-grained analyses. As a correlation between the

participants’ ages and their mothers’ well-being level was identied,

the age eect may have aected the whole group’s well-being

outcome. erefore, future studies are needed to evaluate the

impact of the participants’ and parents’ ages on the well-being

of parents involved in activity programs, such as the one presented.

Plus, the maternal working status changes due to the lockdown

were not assessed within the present study limiting the validity

of the discussion of the results. However, most workplaces were

closed, and people were forbidden to leave their houses (except

for a few reasons) within those months in Italy. erefore, the

authors reasonably assumed that, on average, the participants’

mothers’ working hours were reduced during the lockdown.

Similarly, it would have been interesting to analyze the amount

of social support the mother received in the dierent phases

of the present study and their impact on maternal well-being.

Finally, only mothers’ well-being was analyzed due to fathers’

missing data. Future research should investigate these variables

individually for each parent to understand the dierences between

the impact of the treatment on both mothers and fathers.

CONCLUSION

e results obtained at the CWBS-SF hint that the availability

of an activity program and the conducted remote supervision

calls positively aected maternal well-being in a challenging

period, such as the COVID-19 lockdown. Moreover, as the

researchers were highly familiarized with RTT, their bi-weekly

calls enabled the parents to consult on other issues which

were not directly connected with the motor elements of the

program. It is the authors’ opinion that the strategies and

suggestions given to parents within the current project supported

the daily caring of their daughters, supporting the maternal

well-being in accordance with existing literature, not specically

related to RTT (Singer et al., 2007; Todd et al., 2010).

DATA AVAILABILITY STATEMENT

e original contributions presented in the study are included

in the article/supplementary les, and further inquiries can

be directed to the corresponding author.

ETHICS STATEMENT

e studies involving human participants were reviewed and

approved by the Ariel University Institutional Review Boards,

Ariel University, Ari’el, Israel. Written informed consent to

participate in this study was provided by the participants’ legal

guardian/next of kin.

AUTHOR CONTRIBUTIONS

ML obtained the funds. MZ and ML coordinated the project.

ML and AR conducted participants’ evaluations. AR organized

the participants’ evaluations and carried out all remote supervision

meetings. EI and MF collected the data that MZ and AR

analyzed. MZ, ML, and AR wrote the article. All authors have

read the article and suggested improvements and changes until

agreement was reached.

FUNDING

e International Rett Syndrome Foundation funded the

described project within the HeART grant no. 3610.

ACKNOWLEDGMENTS

We would like to thank the International Rett Syndrome

Foundation for supporting this research project nancially.

T1 T2 T3 T4

Evaluation sessions

All participants

FIGURE3 | Caregivers’ Well-Being Scale—Short Form (CWBS-SF) score of

all participants together at each evaluation point. The box inferior limits

represent rst quartile of the distribution, and the upper limits the third quartile

(median excluded). The lines across the box show the median score of each

group. The crosses inside the boxes mark the mean value of each dataset.

The whiskers indicate the minimum and maximum distribution values.

*p ≤ 0.05.

Zwilling et al. Telerehabilitation Maternal Well-Being COVID-19 Lockdown

Frontiers in Psychology | www.frontiersin.org 9 March 2022 | Volume 13 | Article 834419

Wewould also like to thank the Italian Rett Association (AIRett)

and the head of the organization Lucia Dovigo for supporting

this project connecting researchers and involved families, and

constantly supporting the experimentation of therapeutic

strategies based on telerehabilitation. Finally, we thank the

parents and individuals with RTT involved in this project.

REFERENCES

Amir, R. E., Van Den Veyver, I. B., Schultz, R., Malicki, D. M., Tran, C. Q.,

Dahle, E. J., et al. (2000). Inuence of mutation type and X chromosome

inactivation on Rett syndrome phenotypes. Ann. Neurol. 47, 670–679. doi:

10.1002/1531-8249(200005)47:5<670::AID-ANA20>3.0.CO;2-F

Amir, R. E., Van den Veyver, I. B., Wan, M., Tran, C. Q., Francke, U., and

Zoghbi, H. Y. (1999). Rett syndrome is caused by mutations in X-linked

MECP2, encoding methyl-CpG-binding protein 2. Nat. Genet. 23, 185–188.

doi: 10.1038/13810

Armstrong, R. A. (2014). When to use the Bonferroni correction. Ophthalmic

Physiol. Opt. 34, 502–508. doi: 10.1111/opo.12131

Asbury, K., Fox, L., Deniz, E., Code, A., and Toseeb, U. (2021). How is

COVID-19 aecting the mental health of children with special educational

needs and disabilities and their families? J. Autism Dev. Disord. 51, 1772–1780.

doi: 10.1007/S10803-020-04577-2/TABLES/4

Assenza, C., Catania, H., Antenore, C., Gobbetti, T., Gentili, P., Paolucci, S.,

et al. (2020). Continuity of care During COVID-19 lockdown: A survey

on stakeholders’ experience With Telerehabilitation. Front. Neurol. 11:617276.

doi: 10.3389/FNEUR.2020.617276

Bailey, T., Hastings, R. P., and Totsika, V. (2021). COVID-19 impact on

psychological outcomes of parents, siblings and children with intellectual

disability: longitudinal before and during lockdown design. J. Intellect. Disabil.

Res. 65, 397–404. doi: 10.1111/jir.12818

Bentenuto, A., Mazzoni, N., Giannotti, M., Venuti, P., and de Falco, S. (2021).

Psychological impact of Covid-19 pandemic in Italian families of children

with neurodevelopmental disorders. Res. Dev. Disabil. 109:103840. doi:

10.1016/J.RIDD.2020.103840

Bly, L. (1999). Baby Treatment Based on NDT Principles. Tucson, AZ: erapy

Skill Builders.

Bova, S. M., Basso, M., Bianchi, M. F., Savaré, L., Ferrara, G., Mura, E., et al.

(2021). Impact of COVID-19 lockdown in children with neurological

disorders in Italy. Disabil. Health J. 14:101053. doi: 10.1016/J.

DHJO.2020.101053

Bovend’Eerdt, T. J. H., Botell, R. E., and Wade, D. T. (2009). Writing SMART

rehabilitation goals and achieving goal attainment scaling: A practical guide.

Clin. Rehabil. 23, 352–361. doi: 10.1177/0269215508101741

Caprì, T., Nucita, A., Iannizzotto, G., Stasolla, F., Romano, A., Semino, M.,

et al. (2021). Telerehabilitation for improving adaptive skills of children

and young adults with multiple disabilities: a systematic review. Rev. J.

Autism Dev. Disord. 8, 244–252. doi: 10.1007/s40489-020-00214-x

Cason, J., and Cohn, E. R. (2014). Telepractice: An overview and best

practices. Perspect. Augment. Altern. Commun. 23, 4–17. doi: 10.1044/

aac23.1.4

Cass, H., Reilly, S., Owen, L., Wisbeach, A., Weekes, L., Slonims, V., et al.

(2003). Findings from a multidisciplinary clinical case series of females

with Rett syndrome. Dev. Med. Child Neurol. 45, 325–337. doi:

10.1111/j.1469-8749.2003.tb00404.x

Colizzi, M., Sironi, E., Antonini, F., Ciceri, M. L., Bovo, C., and Zoccante, L.

(2020). Psychosocial and behavioral impact of COVID-19in autism Spectrum

disorder: An online parent survey. Brain Sci. 10:341. doi: 10.3390/

BRAINSCI10060341

Cramm, J. M., and Nieboer, A. P. (2011). Psychological well-being of caregivers

of children with intellectual disabilities: using parental stress as a mediating

factor. J. Intellect. Disabil. 15, 101–113. doi: 10.1177/1744629511410922

Crnic, K., Arbona, A. P., Baker, B., and Blacher, J. (2009). Chapter 1 mothers

and fathers together. Int. Rev. Res. Dev. Dis. 37, 3–30. doi: 10.1016/

S0074-7750(09)37001-9

Cusinato, M., Iannattone, S., Spoto, A., Poli, M., Moretti, C., Gatta, M., et al.

(2020). Stress, resilience, and well-being in Italian children and their parents

during the COVID-19 pandemic. Int. J. Environ. Res. Public heal. 17:8297.

doi: 10.3390/IJERPH17228297

Demeter, K. (2000). Assessing the developmental level in Rett syndrome: An

alternative approach? Eur. Child Adolesc. Psychiatry 9, 227–233. doi: 10.1007/

s007870070047

Di Giorgio, E., Polli, R., Lunghi, M., and Murgia, A. (2021). Impact of the

COVID-19 Italian lockdown on the physiological and psychological well-

being of children with fragile X syndrome and their families. Int. J. Environ.

Res. Public heal. 18:5752. doi: 10.3390/IJERPH18115752

Dovigo, L., Caprì, T., Iannizzotto, G., Nucita, A., Semino, M., Giannatiempo, S.,

et al. (2021). Social and cognitive interactions rough an interactive school

service for RTT patients at the COVID-19 time. Front. Psychol. 12:2417.

doi: 10.3389/fpsyg.2021.676238

Downs, J. A., Bebbington, A., Jacoby, P., Msall, M. E., Mcllroy, O., Fyfe, S.,

et al. (2008). Gross motor prole in rett syndrome as determined by video

analysis. Neuropediatrics 39, 205–210. doi: 10.1055/s-0028-1104575

Downs, J., and Leonard, H. (2016). Quantitative and qualitative insights into

the experiences of childrenwith Rett syndrome and their families. Wien.

Med. Wochenschr. 166, 338–345. doi: 10.1007/s10354-016-0494-6

Drobnyk, W., Rocco, K., Davidson, S., Bruce, S., Zhang, F., and Soumerai, S. B.

(2019). Sensory integration and functional reaching in children With Rett

syndrome/Rett-related disorders. Clin. Med. Insights Pediatr. 13:7195. doi:

10.1177/1179556519871952

Elefant, C. (2001). “Speechless yet communcative: revealing the person behind

the disablity of Rett syndrome through clinical research on songs in music

therapy,” in Music erapy in Europe. e 5th European Music erapy

Congress (Napoli, April 2001), eds. D. Aldridge, G. Di Franco, E. Ruud,

and T. Wigram (Rome: Ismez), 113–128.

Elefant, C., and Lotan, T. (2004). Rett syndrome: dual intervention – music and

physical therapy. Nord. J. Music er. 13, 172–182. doi: 10.1080/08098130409478114

Elefant, C., and Wigram, T. (2005). Learning ability in children with Rett

syndrome. Brain Dev. 27, S97–S101. doi: 10.1016/j.braindev.2005.03.020

Embregts, P., Heerkens, L., Frielink, N., Giesbers, S., Vromans, L., and Jahoda, A.

(2021). Experiences of mothers caring for a child with an intellectual disability

during the COVID-19 pandemic in the Netherlands. J. Intellect. Disabil.

Res. 65, 760–771. doi: 10.1111/JIR.12859

Epstein, L. G. (1995). Rett Syndrome--Clinical and Biological Aspects: Studies

on 130 Swedish Females. London, NW: Cambridge University Press.

Fabio, R. A. (2019). Attention measures of patients with Rett syndrome need

to overcome the challenges in evaluating the oculomotor function using

electronystagmography. Eur. J. Paediatr. Neurol. 23:232. doi: 10.1016/j.

ejpn.2019.02.008

Fabio, R. A., Billeci, L., Crifaci, G., Troise, E., Tortorella, G., and Pioggia, G.

(2016). Cognitive training modies frequency EEG bands and

neuropsychological measures in Rett syndrome. Res. Dev. Disabil. 53, 73–85.

doi: 10.1016/j.ridd.2016.01.009

Fabio, R. A., Castelli, I., Marchetti, A., and Antonietti, A. (2013). Training

communication abilities in Rett syndrome through reading and writing.

Front. Psychol. 4:911. doi: 10.3389/fpsyg.2013.00911

Fabio, R. A., Colombo, B., Russo, S., Cogliati, F., Masciadri, M., Foglia, S.,

et al. (2014). Recent insights into genotype-phenotype relationships in patients

with Rett syndrome using a ne grain scale. Res. Dev. Disabil. 35, 2976–2986.

doi: 10.1016/j.ridd.2014.07.031

Fabio, R. A., Gangemi, A., Capri, T., Budden, S., and Falzone, A. (2018a).

Neurophysiological and cognitive eects of Transcranial direct current

stimulation in three girls with Rett syndrome with chronic language

impairments. Res. Dev. Disabil. 76, 76–87. doi: 10.1016/j.ridd.2018.03.008

Fabio, R. A., Gangemi, A., Semino, M., Vignoli, A., Canevini, M. P., Priori, A.,

et al. (2020). Eects of combined Transcranial direct current stimulation

with cognitive training in girls with Rett syndrome. Brain Sci. 10:276. doi:

10.3390/brainsci10050276

Fabio, R. A., Giannatiempo, S., and Caprì, T. (2019). Attention and Identication

of the Same and the Similar Visual Stimuli in Rett Syndrome. Life Span

Disabil. 22, 113–127.

Zwilling et al. Telerehabilitation Maternal Well-Being COVID-19 Lockdown

Frontiers in Psychology | www.frontiersin.org 10 March 2022 | Volume 13 | Article 834419

Fabio, R. A., Giannatiempo, S., Oliva, P., and Murdaca, A. M. (2011). e

increase of attention in Rett syndrome: A pre-test/post-test research design.

J. Dev. Phys. Disabil. 23, 99–111. doi: 10.1007/s10882-010-9207-z

Fabio, R. A., Giannatiempo, S., Semino, M., and Caprì, T. (2021). Longitudinal

cognitive rehabilitation applied with eye-tracker for patients with Rett

syndrome. Res. Dev. Disabil. 111:103891. doi: 10.1016/j.ridd.2021.103891

Fabio, R. A., Magaudda, C., Caprì, T., Towey, G. E., and Martino, G. (2018b).

Choice behavior in Rett syndrome: e consistency parameter. Life Span

Disabil. 21, 47–62.

Fabio, R. A., Martinazzoli, C., and Antonietti, A. (2005). Costruzione e

standardizzazione dello strumento “RARS” (Rett assessment rating scale).

Ciclo. Evol. Disabil. 8, 257–281.

George, J., Binns, V. E., Clayden, A. D., and Mulley, G. P. (1988). Aids and

adaptations for the elderly at home: underprovided, underused, and

undermaintained. Br. Med. J. Clin. Res. 296, 1365–1366. doi: 10.1136/

bmj.296.6633.1365

Gillberg, C. (1997). Communication in Rett syndrome complex. Eur. Child

Adolesc. Psychiatry 6(Suppl. 1), 21–22.

Glaze, D. G., Percy, A. K., Skinner, S., Motil, K. J., Neul, J. L., Barrish, J. O.,

et al. (2010). Epilepsy and the natural history of Rett syndrome. Neurology

74, 909–912. doi: 10.1212/WNL.0b013e3181d6b852

Hagberg, B., Aicardi, J., Dias, K., and Ramos, O. (1983). A progressive syndrome

of autism, dementia, ataxia, and loss of purposeful hand use in girls: Rett’s

syndrome: report of 35 cases. Ann. Neurol. 14, 471–479. doi: 10.1002/

ana.410140412

Hetzroni, O., Rubin, C., and Konkol, O. (2002). e use of assistive technology

for symbol identication by children with Rett syndrome. J. Intellect. Develop.

Disabil. 27, 57–71. doi: 10.1080/13668250120119626-1

Ison, R. (2014). “Systems thinking and practice for action research,” in e

SAGE Handbook of Action Research. eds. R. Peter and B. Hilary (London:

SAGE publications), 139–158.

Jacobsen, K., Viken, Å., and Von Tetzchner, S. (2001). Rett syndrome and

ageing: A case study. Disabil. Rehabil. 23, 160–166. doi:

10.1080/09638280150504234

Kaufmann, W. E., Johnston, M. V., and Blue, M. E. (2005). MeCP2 expression

and function during brain development: implications for Rett syndrome’s

pathogenesis and clinical evolution. Brain Dev. 27, S77–S87. doi: 10.1016/j.

braindev.2004.10.008

Kemmis, S., and McTaggart, R. (2005). “Participatory action research:

communicative action and the public sphere,” in e Sage handbook of

qualitative research. eds. N. K. Denzin and Y. S. Lincoln (ousand Oaks,

CA: SAGE Publications Ltd), 559–603.

Koppenhaver, D. A., Erickson, K. A., Harris, B., McLellan, J., Skotko, B. G.,

and Newton, R. A. (2001a). Storybook-based communication intervention

for girls with Rett syndrome and their mothers. Disabil. Rehabil. 23, 149–159.

doi: 10.1080/09638280150504225

Koppenhaver, D. A., Erickson, K. A., and Skotko, B. G. (2001b). Supporting

communication of girls with rett syndrome and their mothers in storybook

reading. Int. J. Disabil. Dev. Educ. 48, 395–410. doi: 10.1080/10349120120094284

Lazarus, R. S., and Folkman, S. (1984). Stress, Appraisal, and Coping. (New

York: Springer publishing company).

Lee, J. (2013). Maternal stress, well-being, and impaired sleep in mothers of

children with developmental disabilities: a literature review. Res. Dev. Disabil.

34, 4255–4273. doi: 10.1016/j.ridd.2013.09.008

Legrand, F., Danion-Grilliat, A., and Gras-Vincendon, A. (1997). Rett syndrome.

Ann. Med. Psychol. 155, 102–112.

Leonard, H., Fyfe, S., Leonard, S., and Msall, M. (2001). Functional status,

medical impairments, and rehabilitation resources in 84 females with Rett

syndrome: a snapshot across the world from the parental perspective. Disabil.

Rehabil. 23, 107–117. doi: 10.1080/09638280150504180

Lindberg, B. (2006). Understanding Rett Syndrome: A Practical Guide for Parents,

Teachers, and erapists. (Vienna, Austria: Hogrefe Publishing.

Lotan, M., Downs, J., and Elefant, C. (2021a). A pilot study delivering

physiotherapy support for Rett syndrome using a Telehealth framework

suitable for COVID-19 lockdown. Dev. Neurorehabil. 24, 429–434. doi:

10.1080/17518423.2021.1914762

Lotan, M., and Hanks, S. (2006). Physical therapy intervention for individuals

with Rett syndrome. Scientic World J. 6, 1314–1338. doi: 10.1100/

tsw.2006.187

Lotan, M., Ippolito, E., Favetta, M., and Romano, A. (2021b). Skype Supervised,

Individualized, Home-Based Rehabilitation Programs for Individuals With

Rett Syndrome and eir Families–Parental Satisfaction and Point of View.

Front. Psychol. 12:3995. doi: 10.3389/FPSYG.2021.720927/BIBTEX

Lotan, M., Isakov, E., and Merrick, J. (2004). Improving functional skills and

physical tness in children with Rett syndrome. J. Intellect. Disabil. Res. 48,

730–735. doi: 10.1111/j.1365-2788.2003.00589.x

Lotan, M., Merrick, J., and Carmeli, E. (2005). Managing scoliosis in a young

child with Rett syndrome: a case study. Sci. World J. 5, 264–273. doi: 10.1100/

tsw.2005.33

Lotan, M., Merrick, J., Kandel, I., and Morad, M. (2010). Aging in persons

with Rett syndrome: An updated review. Sci. World J. 10, 778–787. doi:

10.1100/tsw.2010.79

Maciques Rodríguez, E., and Lotan, M. (2011). “Rett syndrome: Hippotherapy

intervention,” in Rett Syndrome: erapeutic Interventions. eds. M. Lotan

and J. Merrick (New York, NY: Nova Science Publishers), 247–277.

Mackenzie, J., Tan, P. L., Hoverman, S., and Baldwin, C. (2012). e value

and limitations of participatory action research methodology. J. Hydrol. 474,

11–21. doi: 10.1016/J.JHYDROL.2012.09.008

Marchetti, D., Fontanesi, L., Mazza, C., Di Giandomenico, S., Roma, P., and

Verrocchio, M. C. (2020). Parenting-related exhaustion During the Italian

COVID-19 lockdown. J. Pediatr. Psychol. 45, 1114–1123. doi: 10.1093/JPEPSY/

JSAA093

McClure, M. K., Battaglia, C., and McClure, R. J. (1998). e relationship of

cumulative motor asymmetries to scoliosis in Rett syndrome. Am. J. Occup.

e r. 52, 196–204. doi: 10.5014/ajot.52.3.196

Milani, G., Demattè, G., Ferioli, M., Dallagà, G., Lavezzi, S., Basaglia, N.,

et al. (2021). Telerehabiltation in Italy During the COVID-19 lockdown:

A feasibility and acceptability study. Int. J. Telerehabil. 13:e6334. doi:

10.5195/ijt.2021.6334

Mori, Y., Downs, J., Wong, K., and Leonard, H. (2019). Longitudinal eects

of caregiving on parental well-being: the example of Rett syndrome, a severe

neurological disorder. Eur. Child Adolesc. Psychiatry 28, 505–520. doi: 10.1007/

s00787-018-1214-0

Newig, J. (2007). Does public participation in environmental decisions lead to

improved environmental quality?: towards an analytical framework. Int. J.

Sustain. Commun. 1, 51–71.

Ozanne, J. L., and Saatcioglu, B. (2008). Participatory action research. J. Consum.

Res. 35, 423–439. doi: 10.1086/586911

Pari, E., Cozzi, F., Rodocanachi Roidi, M. L., Grange, F., Toshimori, K., and

Ripamonti, E. (2020). Parenting girls with Rett syndrome: An investigation

on self-perceived levels of stress. J. Appl. Res. Intellect. Disabil. 33, 1348–1356.

doi: 10.1111/jar.12755

Patel, V., Perez-Olivas, G., Kroese, B. S., Rogers, G., Rose, J., Murphy, G., et al.

(2021). e experiences of Carers of adults With intellectual disabilities

During the rst COVID-19 lockdown period. J. Policy Pract. Intellect. Disabil.

18, 254–262. doi: 10.1111/JPPI.12382

Perks, B. A., Mackintosh, R., Stewart, C. P. U., and Bardsley, G. I. (1994). A

survey of marginal wheelchair users. J. Rehabil. Res. Dev. 31, 297–302

Perry, A., Sarlo-McGarvey, N., and Factor, D. C. (1992). Stress and family

functioning in parents of girls with Rett syndrome. J. Autism Dev. Disord.

22, 235–248. doi: 10.1007/BF01058153

Pizzamiglio, M. R., Nasti, M., Piccardi, L., Zotti, A., Vitturini, C., Spitoni, G.,

et al. (2008). Sensory-motor rehabilitation in Rett syndrome: A case report.

Focus Autism. Dev. Disabl. 23, 49–62. doi: 10.1177/1088357607311446

Rabanifar, N., and Abdi, K. (2021). Letter to editor: Telerehabilitation: A useful

and appropriate approach for people with disability in Covid-19 pandemic.

Med. J. Islam Repub. Iran 35:18. doi: 10.47176/MJIRI.35.18

Ramey, C., and Ramey, S. (1996). “Early intervention: optimizing development

for children with disabilities and risk conditions,” in Disorders of Development

& Learning: A Practical Guide to Assessment and Management. ed. M.

Wolraich (Philadelphia: Mosby Incorporated), 141–158.

Richmond, T., Peterson, C., Cason, J., Billings, M., Terrell, E. A., Lee, A. C.

W., et al. (2017). American telemedicine Association’s principles for delivering

Telerehabilitation services. IJT 9, 63–68. doi: 10.5195/ijt.2017.6232

Rogers, G., Perez-Olivas, G., Stenfert Kroese, B., Patel, V., Murphy, G., Rose, J.,

et al. (2021). e experiences of mothers of children and young people

with intellectual disabilities during the rst COVID-19 lockdown period.

J. Appl. Res. Intellect. Disabil. 34, 1421–1430. doi: 10.1111/jar.12884

Zwilling et al. Telerehabilitation Maternal Well-Being COVID-19 Lockdown

Frontiers in Psychology | www.frontiersin.org 11 March 2022 | Volume 13 | Article 834419

Romano, A., Caprì, T., Semino, M., Bizzego, I., Di Rosa, G., and Fabio, R. A.

(2020). Gross motor, physical activity and musculoskeletal disorder evaluation

tools for Rett syndrome: A systematic review. Dev. Neurorehabil. 23, 485–501.

doi: 10.1080/17518423.2019.1680761

Romano, A., Di Rosa, G., Tisano, A., Fabio, R. A., and Lotan, M. (2021).

Eects of a remotely supervised motor rehabilitation program for individuals

with Rett syndrome at home. Disabil. Rehabil. 20, 1–11. doi: 10.1080/09638288.

2021.1949398

Sarajlija, A., Djuric, M., Kisic-Tepavcevic, D., and Tepavcevic, D. K. (2013).

Health-related quality of life and depression in Rett syndrome caregivers.

Vojnosanit. Pregl. 70, 842–847. doi: 10.2298/VSP1309842S

Sigafoos, J., Woodyatt, G., Tucker, M., Roberts-Pennell, D., and Pittendreigh, N.

(2000). Assessment of potential communicative acts in three individuals with

Rett syndrome. J. Dev. Phys. Disabil. 12, 203–216. doi: 10.1023/A:1009461704556

Singer, G. H. S., Ethridge, B. L., and Aldana, S. I. (2007). Primary and secondary

eects of parenting and stress management interventions for parents of

children with developmental disabilities: a meta-analysis. Ment. Retard. Dev.

Disabil. Res. Rev. 13, 357–369. doi: 10.1002/mrdd.20175

Smith, T. B., Oliver, M. N., and Innocenti, M. S. (2001). Parenting stress in

families of children with disabilities. Am. J. Orthopsychiatry 71, 257–261.

doi: 10.1037/0002-9432.71.2.257

Sponseller, P. (2001). Orthopedic update in Rett syndrome. Rett Gaz. 1, 4–5.

Stasolla, F., De Pace, C., Damiani, R., Di Leone, A., Albano, V., and Perilli, V.

(2014). Comparing PECS and VOCA to promote communication opportunities

and to reduce stereotyped behaviors by three girls with Rett syndrome.

Res. Autism Spectr. Disord. 8, 1269–1278. doi: 10.1016/j.rasd.2014.06.009

Sullivan, M. W., Laverick, D. H., and Lewis, M. (1995). Brief report: fostering

environmental control in a young child with rett syndrome: A case study.

J. Autism Dev. Disord. 25, 215–221. doi: 10.1007/BF02178506

Tebb, S. S., Berg-Weger, M., and Rubio, D. M. (2013). e caregiver well-being

scale: developing a short-form rapid assessment instrument. Heal. Soc. Work

38, 222–230. doi: 10.1093/hsw/hlt019

Todd, S., Bromley, J., Ioannou, K., Harrison, J., Mellor, C., Taylor, E., et al.

(2010). Using group-based parent training interventions with parents of

children with disabilities: A description of process, content and outcomes

in clinical practice. Child Adolesc. Ment. Health 15, 171–175. doi: 10.1111/j.

1475-3588.2009.00553.x

Vessoyan, K., Steckle, G., Easton, B., Nichols, M., Mok Siu, V., and McDougall, J.

(2018). Using eye-tracking technology for communication in Rett syndrome:

perceptions of impact. Augment. Altern. Commun. 34, 230–241. doi:

10.1080/07434618.2018.1462848

Vignoli, A., Fabio, R. A., La Briola, F., Giannatiempo, S., Antonietti, A.,

Maggiolini, S., et al. (2010). Correlations between neurophysiological,

behavioral, and cognitive function in Rett syndrome. Epilepsy Behav. 17,

489–496. doi: 10.1016/j.yebeh.2010.01.024

Wine, J. (2009). Resource for the Development of a Communication Program

for Girls with Rett Syndrome. Israel: Ramat-Gan Rett Syndr. Found.

World Health Organization (2020). Virtual Press Conference on COVID-19.

March 11, 2020.

Zysman, L., Lotan, M., and Ben-Zeev, B. (2006). Osteoporosis in Rett syndrome:

a study on normal values. ScienticWorldJournal 6, 1619–1630. doi: 10.1100/

tsw.2006.266

Conict of Interest: e authors declare that the research was conducted in

the absence of any commercial or nancial relationships that could beconstrued

as a potential conict of interest.

Publisher’s Note: All claims expressed in this article are solely those of the

authors and do not necessarily represent those of their aliated organizations,

or those of the publisher, the editors and the reviewers. Any product that may

be evaluated in this article, or claim that may be made by its manufacturer, is

not guaranteed or endorsed by the publisher.

access article distributed under the terms of the Creative Commons Attribution

License (CC BY). e use, distribution or reproduction in other forums is permitted,

provided the original author(s) and the copyright owner(s) are credited and that

the original publication in this journal is cited, in accordance with accepted academic

practice. No use, distribution or reproduction is permitted which does not comply

with these terms.

BRIEF RESEARCH REPORT

published: 14 March 2022

doi: 10.3389/fpsyt.2022.797150

Frontiers in Psychiatry | www.frontiersin.org 1March 2022 | Volume 13 | Article 797150

Edited by:

Christos Theleritis,

National and Kapodistrian University

of Athens, Greece

Reviewed by:

Domenico De Berardis,

Mental Health Center (CSM) and

Psychiatric Service of Diagnosis and

Treatment (SPDC), Italy

Lucie Bartova,

Medical University of Vienna, Austria

*Correspondence:

Tarek Okasha

tarek.okasha@gmail.com

Specialty section:

This article was submitted to

Public Mental Health,

a section of the journal

Frontiers in Psychiatry

Received: 26 October 2021

Accepted: 04 February 2022

Published: 14 March 2022

Citation:

Abdel-Wahab M, Okasha T,

Shaheen M, Nasr M, Molokheya T,

Omar AE, Rabie MA, Samy V,

Hamed H and Ali M (2022) Clinical

Guidelines of the Egyptian Psychiatric

Association for the Management of

Treatment-Resistant Unipolar

Depression in Egypt.

Front. Psychiatry 13:797150.

doi: 10.3389/fpsyt.2022.797150

Clinical Guidelines of the Egyptian

Psychiatric Association for the

Management of Treatment-Resistant

Unipolar Depression in Egypt

Momtaz Abdel-Wahab 1, Tarek Okasha 2

*, Mostafa Shaheen 3, Mohamed Nasr3,

Tarek Molokheya 4, Abd ElNasser Omar 5, Menan A. Rabie 5, Victor Samy 6, Hany Hamed7

and Mohamed Ali 8

1Department of Psychiatry, Kasr El-Aini Medical School, Cairo University, Giza, Egypt, 2Okasha Institute of Psychiatry,

Medical School, Ain Shams University, Cairo, Egypt, 3Department of Psychiatry, Kasr El-Aini Medical School, Cairo

University, Cairo, Egypt, 4Department of Psychiatry, Alexandria Medical School, Alexandria University, Alexandria, Egypt,

5Department of Psychiatry, Ain Shams Medical School, Ain Shams University, Cairo, Egypt, 6Department of Psychiatry,

Banha Medical School, Banha University, Benha, Egypt, 7Department of Psychiatry, Beni-Suef Medical School, Beni-Suef

University, Beni-Suef, Egypt, 8Faculty of Pharmacy, German University of Cairo, Cairo, Egypt

Background: Major depressive disorder (MDD) is a public health burden that creates a

strain not only on individuals, but also on the economy. Treatment-resistant depression

in the course of major depressive disorder represents a clinically challenging condition

that is deﬁned as insufﬁcient response to two or more antidepressant trails with

antidepressants of the same or different classes that were administered at adequate

daily doses for at least 4 weeks.

Objective/Hypothesis: To develop a treatment guideline for Treatment Resistant

Depression (TRD).

Methodology: Experts in the ﬁeld gathered and reviewed the available evidence about

the subject. Then, a series of meetings were held to create recommendations that can

be utilized by Egyptian psychiatrists.

Results: The guidelines provide recommendations in various clinical settings. It

evaluates different situations, such as patients at risk of resistance, those with resistance

and recommends strategies to resolve the clinical case.

Conclusion: The consensus guidelines will improve the outcomes of patients, as

they provide recommendations across various domains that are of concern for the

practicing psychiatrist.

Keywords: depression, treatment resistance, guidelines, clinical psychiatry, neuropharmacology

INTRODUCTION

MDD is deﬁned as a disorder of having one or more major depressive episodes in a person’s

life with the absence of manic or hypomanic symptoms; to meet the criteria of MDD ﬁve out of

nine symptoms, two of which must be low mood and anhedonia (loss of pleasure) persisting for a

2-week period. The nine symptoms include low mood, loss of interest or pleasure, disturbed sleep,

change of eating pattern or weight, agitation or psychomotor slowness, unexplained fatigue, feeling

worthless or guilty, inability to concentrate and thoughts of death or suicide (1).

Abdel-Wahab et al. TRD: A Guideline for Egyptian Psychiatrists

Major depressive disorder (MDD) is a major public

health concern. It is projected to be the ﬁrst cause of

disease burden worldwide by 2030 (2). According to the

World Health Organization (WHO), depression aﬀects

over 260 million individuals worldwide (3). Results of a

recent systematic review by Odejimi et al. reported that the

prevalence of depression in Egypt ranges between 23.7 and

74.5% (4).

This mental disorder results from the interaction of several

factors, including psychological, social and biological factors (5).

According to the Diagnostic and Statistical Manual of

Mental Disorders, 5th Edition (DSM-5), a person is diagnosed

with MDD when he or she consistently depressed mood

or anhedonia, along with ﬁve of the following symptoms;

diﬃculty in concentration, appetite changes, decreased

energy, sleep disturbance, suicidal thoughts and tendencies,

concentration diﬃculties, or psychomotor agitation or

depression (6).

In medicine, the term “resistance” is used to describe the

failure of standard treatment; although there is no distinct

deﬁnition for TRD as clinical practice is ever evolving and

because treatment “failure” is sometimes judged by physicians

themselves when their patients do not respond as much as they

expected them to. In short, TRD is described as depression that

is unresponsive to antidepressant drug treatment at adequate

dosing for an adequate amount of time (5).

Response to traditional treatment has been assessed by several

studies (6). One major trial highlighted that the remission rate

after step 1 of treatment was 36.8%, with the remission rate

decreasing after each step of treatment, reaching only 13% at

the fourth step (7,8). In other words, the more advancement

in treatment lines, the more likely is the patient to relapse. The

same study concluded that the overall remission rate was 67%,

highlighting that about 1 in every 3 patients fails treatment (9).

Several psycho-pharmacotherapeutic strategies have been

suggested to overcome TRD, whereby augmentation treatment

represents the currently recommended ﬁrst-choice in case of

insuﬃcient response to the initial antidepressant treatment.

In accordance with available international evidence, second-

generation antipsychotics or lithium should be preferably

employed (6).

Many patients are labeled as treatment resistant mistakenly

when they are actually pseudo-resistant cases of depression; most

common causes of this case are due to sub-therapeutic dosing

or non-adherence to medication. According to APA guidelines it

is important for physicians to assess any patient comorbidities,

modifying doses of ﬁrst line treatment if it failed after 4 weeks,

considering patient history when adding second line treatment to

ensure that the patient will positively respond to the medication

and only initiating second line treatment after unsatisfactory

response at 4 weeks (9).

“According to available international evidence, this

publication will cover treatment options and recommendations

from the Egyptian Psychiatric Association for the management

of TRD in the course of MDD in Egypt.”

METHODS

Expert recommendations were determined after a thorough

examination of available literature to identify and assess

the recent global updates about the subject. Prior to the

meeting, two senior members of the committee performed

a comprehensive search on the PubMed database to identify

the available literature relevant to the topic using keyword,

such as “treatment-resistant depression”, “major depressive

disorders” and their derivates. Then, those senior members

prepared a questionnaire using the Delphi technique and invited

committee members to assess the literature and respond to

the questionnaire.

Afterwards, responses of clinicians to the questionnaire were

captured, analyzed and ranked. Then, another round of questions

was dispensed in a second meeting based on the outcomes

of the ﬁrst committee gathering and questionnaire responses.

The second round was when a consensus was reached by all

committee members, therefore there was no need for further

meetings (10).

RESULTS

Panel Description

An expert panel of 8 professors of psychiatry from diﬀerent

universities and the ministry of health representing the

Egyptian Psychiatric Association drafted the following

guidelines. Information about panel members can be found in

the Appendix.

Deﬁnition of Resistant Depression

According to a meta-analysis by Gaynes et al. 2019, there is no

clear-cut deﬁnition for TRD, however, most experts unanimously

agree that lack of response to initial treatment is deemed as

TRD. The APA (American Psychological Association) and NICE

(National Institute for Health and Care Excellence) guidelines

also state that the next step of handling TRD also diﬀers

between experts which makes it more challenging to develop

universal treatment guidelines. The classes of medications should

be among the following: (11).

- Tricyclic antidepressants (TCAs) (equivalent to 300 mg

imipramine) (12)

- Selective serotonin reuptake inhibitors (SSRIs) (equivalent to

50 mg ﬂuoxetine) (13)

- Serotonin norepinephrine reuptake inhibitors (SNRIs)

(equivalent to 225 mg venlafaxine) (14)

Antidepressant treatment has been the mainstay treatment of

depression for years; the ﬁrst class of such drugs were tricyclic

antidepressants followed by more novel agents namely SSRIs

and SNRIs. Both classes of TCA and SSRIs are equally aﬀective

in ameliorating depressive symptoms and decreasing depression

scores as proved by a meta-analysis to compare the two classes.

However, SSRIs are preferred by patients as well as physicians as

they do not produce bothersome side eﬀects as TCAs (15).

Frontiers in Psychiatry | www.frontiersin.org 2March 2022 | Volume 13 | Article 797150

Abdel-Wahab et al. TRD: A Guideline for Egyptian Psychiatrists

SNRIs are a group of antidepressants with a dual mechanism

of action of both serotonin and adrenergic reuptake inhibition;

research has suggested that SNRIs are superior to SSRIs

in more severe depression; which insinuates their possible

eﬀectiveness in TRD. Moreover, SNRIs have established eﬃcacy

in treating depression with somatic manifestations as pain and

other physical symptoms (16). The Danish guidelines; which

incorporates the APA and NICE guidelines states that most

patients ﬁrst diagnosed with MDD are treated with SSRIs

then switched to SNRIs. The ten most prescribed treatments

for TRD mainly constitute of both SSRI’s and SNRI’s as ﬁrst,

second- and third-line treatments (17). Other antidepressants

as trazodone, vilazodone and tranylcypromine are also available

options however, are not easily accessible in the Egyptian market.

The advisory committee recommended of TRD be as follows:

Treatment-resistant depression shall be deﬁned as the failure

of 2 diﬀerent classes of antidepressant medications, given that

the medications have been used for a period of 6–8 weeks at the

desired dose.

N.B. (Nota bene) to the above deﬁnition and in order to

account for the deﬁnition of TRD in Egypt, the committee

added 6–12 monitored sessions of BST (Brain Synchronization

Therapy)/ECT (Electroconvulsive Therapy).

The panel recognized the importance of identifying the

term “pseudo-resistance”, which was deﬁned by the panel as

a patient not responding to medication because of a problem

in the diagnosis or the type of depression or the presence

of depression secondary to another psychiatric, personality or

medical disorder.

In addition to pseudo-resistance, the clinical experts

highlighted additional predictors for TRD, as follows:

- Other forms of depression such as bipolar depression.

- Other comorbid conditions, including concurrent anxiety,

drug abuse, chronic organic medical conditions, and

personality-related disorders (e.g., mood swings in borderline

personality disorder).

Assessment of Treatment-Resistant

Depression

Clinical panel experts recommended using the following set of

clinical tools and tests to identify and diagnose TRD.

- Depression severity scales such as Montgomery-Asberg

Depression Rating Scale (MADRS) and psychiatrist-rated

scales. Montgomery-Asberg Depression Rating Scale has long

been used by physicians for assessing depression, the advantage

of MADRS is that avoids the drawbacks of the HAM-D

score and is more robust. It is used mainly to detect any

patients’ response changes to antidepressant therapy with high

sensitivity and is positively correlated to change in degree of

depression (18).

-Hypomania check list: The hypomania checklist has long

been used in many countries and in diﬀerent languages to

diﬀerentiate bipolar depression from MDD. This is important

as manic/psychotic symptoms in depression should not be

confused with TRD (19).

- Suicide scale [e.g., Beck’s Scale for Suicide Ideation or Columbia

Suicide Severity Rating Scale (C-SSRS)]. Suicide ideation scales

are mainly used to monitor patient’s health and to predict the

risk of actual suicide for timely intervention (20). The BSS is

one of the most reliable tools to predict a patient’s risk and plan

for suicide. The C-SSRS is also a sensitive scale to use especially

that it is sensitive to change of suicidal ideation over treatment

time and with the use of medication (21).

- Complete blood count (CBC), liver and kidney function tests,

lipid and glycemic proﬁles: Complete blood count (CBC), liver

and kidney function test and lipid and glycemic proﬁles; all

these tests can be used and are done to eliminate any disease

that could precipitate symptoms of depression (22).

- Thyroid function test: Regarding thyroid proﬁling; it

is established through a wide volume of research that

hypothyroidism presents with some depressive symptoms, in

a study conducted by Bathla et al. 56% of males and 64% of

females with hypothyroidism presented with some symptoms

of depression and anxiety (23).

- Electrocardiogram: An electrocardiogram could be beneﬁcial

since stress (a component of MDD) is linked to heart disease.

Not only that; depression onset is often seen in up to 40% of

patients after a major cardiac event. Therefore, an ECG could

be of use to determine heart health of patients and to stratify

patients who could be at risk of developing MDD or TRD (24).

- Brain imaging techniques like magnetic resonance imaging

(MRI): Recently, brain imaging using MRIs or CT scans

have been utilized in the diagnosis of MDD especially in the

elderly. This is because some cases of depression indicate

an underlying mental disease as Parkinson’s, Alzheimer’s

and Pick’s disease; moreover, geriatric depression has been

associated with leukoencephalopathy that can be detected

using brain imaging techniques (25).

The panel highlighted the importance of periodic performance

of most of the above tests upon prescribing ADs. In addition to

the aforementioned tests, the panel also recommends performing

the following tests based on clinician’s discretion and the patient

proﬁle, as follows:

- Assessing vitamin D plasma levels

- Assessing sex hormones plasma levels

- Toxicological analysis of blood and urine samples

Principles of Management

Hospitalization: Indications

Upon careful assessment of the condition, the expert panel agreed

that psychiatric hospitalization is warranted in severe cases that

fall under the umbrella of one of the following:

- Catatonic cases

- Patients with high suicidal tendency

- Severe psychotic symptoms

- Advanced cases of MDD

- The presence of severe uncontrolled comorbid

medical conditions

- Insuﬃcient familial support to the patient

Frontiers in Psychiatry | www.frontiersin.org 3March 2022 | Volume 13 | Article 797150

Abdel-Wahab et al. TRD: A Guideline for Egyptian Psychiatrists

Therapeutic Options for

Treatment-Resistant Depression and

Comorbid Conditions

Pharmacotherapeutic Options of Comorbid

Conditions

For patients suﬀering from comorbid conditions, the expert

panel recommends the following:

Patients suﬀering from anxiety and its features are

recommended to receive benzodiazepines. Buspirone and

pregabalin are also available options. Anxiety and depression

often go hand in hand; therefore, a multi-modal treatment

approach for handling both illnesses are often recommended.

The evidence in the treatment of anxiety disorders greatly points

to SSRIs, SNRIs and benzodiazepines. Benzodiazepines have

long been used as anxiolytics as they demonstrate a relatively

high safety and tolerability proﬁle, they also have a rapid onset

of action and manage acute anxiety symptoms and somatic

complaints according to the NICE guidelines. The common issue

physicians and patients face with benzodiazepines is dependence

and abuse; thus, benzodiazepines should be preferentially used

for a short amount of time (26).

The NICE guidelines usually recommend pregabalin in

patients with anxiety if they do not tolerate SSRIs and SNRIs;

pregabalin is an anti-convulsant and is also used for neuropathic

pain; the advantage of pregabalin over benzodiazepines is that

it does not cause dependance; however, sudden discontinuation

may cause confusion. Buspirone, a non-benzodiazepine

alternative is also prescribed for anxiety symptoms; the evidence

regarding its eﬃcacy in comparison to benzodiazepines is

conﬂicting; however, it does not cause sedation, withdrawal

symptoms or addiction as benzodiazepines (27).

Patients with depression or TRD frequently suﬀer from sleep

disturbances; therefore, a sleep-aid can be used to overcome this

symptom. Benzodiazepines; although excellent sleep-aids; often

cause residual daytime sleepiness or “hangover”, high doses of

benzodiazepines also cause cognitive impairment and respiratory

depression therefore, pose a risk; in contrast to hypnotic agent

zolpidem that lacks eﬀect of residual day-time sedation or

psycho-motor impairment. Zopiclone, another hypnotic agent; is

also free of any day-time sleepiness side eﬀect but slightly impairs

psychomotor function especially at high doses. Both agents are

reported to be tolerated, eﬃcacious, safe and have low rates of

dependence or abuse (28).

Other non-benzodiazepines that also act as sedatives

are trazodone (an antidepressant from a class of serotonin

modulators) and low dose quetiapine (an atypical anti-

psychotic). Quetiapine has a wide array of indications; it has

the advantage of ameliorating anxiety symptoms so can be used

in TRD comorbid with anxiety, chronic pain and PTSD. In

depression; 27 patients in a MDD episode were administered

low dose quetiapine along with venlafaxine or escitalopram, by

the end of the 4 weeks test time; sleep parameters had deﬁnitely

improved in all patients (29). A head-to-head comparative study

between trazodone and low dose quetiapine in hospitalized

psychiatric patients at St. Helena’s Hospital suﬀering from

insomnia revealed that trazodone is a superior agent in patients

with depressive symptoms and oﬀers higher improvement in

sleep parameters than quetiapine (30).

Patients at high risk of self-inﬂicted injury are

recommended to receive lithium, benzodiazepine, or second-

generation antipsychotic medication. This is because second

benzodiazepines generation anti-psychotics as clonazepam

have a mood stabilizing eﬀect and reduce impulsivity and

mood swings (31).

Treatment Duration

The clinical committee members recommended that patients

should be maintained on their ongoing antidepressant

medication for a period of 9–12 months following the

achievement of clinical remission.

Patients suﬀering from the following conditions are

recommended to receive a longer course of treatment:

- Long period to reach remission

- History of 2 prior depression episodes

- History of early relapse after treatment discontinuation

- Presence of suicidal tendency, symptoms of psychosis,

family history of suicide or mood disorders or comorbid

psychiatric condition.

- Resistance to an antidepressant medication when given at

proper dose and adequate duration.

Therapeutic Options of Patients Suffering

From TRD

The panel experts highlighted the available treatment options for

patients suﬀering from TRD. The available therapeutic options

are captured in Figure 1.

Pharmacological Strategies in

Treatment-Resistant Depression

Switching Strategies

Switching involves shifting to another antidepressant medication,

either within the same class or from a diﬀerent class. Switching

to a medication within the same class is undertaken to obtain a

diﬀerent pharmacological property, while switching to another

class usually yields a diﬀerent neurochemical eﬀect. This strategy

is tailored to suit individual patient needs and preferences (5).

According to evidence gathered about switching treatment in

TRD; switching has certain pros over augmentation therapy; ﬁrst

it carries lower risk of drug-drug interactions, has higher patient

adherence, moreover, it is preferred for patients who suﬀered

severe side eﬀects from ﬁrst line medication and displayed partial

or no response.

Only two major trials have been conducted to observe

switching of medications in TRD, the ﬁrst was about patients

who had previously failed two antidepressants (mostly SSRIs)

and were switched to either SSRI paroxetine or SNRI venlafaxine;

response rates were 33 and 52%, respectively while remission

rates were 20 and 42%. Other studies involved switching from

SSRI to extended-release venlafaxine and switching from an SSRI

to mirtazapine or sertraline, both studies showed no signiﬁcant

diﬀerence in depressive symptoms.

Frontiers in Psychiatry | www.frontiersin.org 4March 2022 | Volume 13 | Article 797150

Abdel-Wahab et al. TRD: A Guideline for Egyptian Psychiatrists

FIGURE 1 | The available therapeutic options for TRD.

The panel experts recommend switching to be carried out in

the following situations:

- Lack of response or poor tolerance to initial treatment

- Prior response to the introduced medication

The clinical experts highlighted 3 diﬀerent types of switching

strategies, with concurrent switching being the most

recommended, except for medications belonging to monoamine

oxidase inhibitor (MAOIs) class. These strategies are captured in

Figure 2 (32).

Recommendations for switching antidepressant medications

are captured in Figure 3.

Combination Strategies

Combination strategies usually involve using 2 diﬀerent

antidepressant medications belonging to diﬀerent classes and

having diﬀerent pharmacodynamic proﬁles (33).

The panel recommends the use of combination strategy in

patients with partial response after adequate treatment with a

medication for a period of 2–4 weeks (4–6 weeks if with TCAs).

The recommended ﬁrst-line combination strategy involves

mirtazapine plus one of the following:

- Selective serotonin reuptake inhibitor.

- Serotonin–norepinephrine reuptake inhibitor.

- Tricyclic antidepressant.

Augmentation Strategies

Augmentation strategies refers to the addition of non-standard

antidepressant medications, like lithium and quetiapine,

to enhance the outcome of classical antidepressants (34).

The panel summarized the medications and lines of choice

in Table 1.

The panel recommends this strategy in patients who

demonstrate partial response after 2–4 weeks of treatment

(4–6 weeks with TCA as they have a delayed eﬀect than

newer agents).

The panel recommends adding lithium or quetiapine to

improve eﬃcacy of the antidepressant medication.

The panel recommends a second choice of thyroid hormone

supplementation in addition to serotonin–norepinephrine

reuptake inhibitors or tricyclic antidepressants, and with

selective serotonin reuptake inhibitors or mirtazapine at a

later stage.

The recommended dose of thyroid hormone supplementation

is between 25 and 60 ug/day of liothyronine (L-T3) and is

required to achieve TSH levels ranging between 0.1 and 1 ug/L.

Prior to initiating treatment, the panel recommends performing

the following assessments:

- Physical examination

- Electrocardiogram (ECG)

- Thyroid-stimulating hormone (TSH) levels.

Frontiers in Psychiatry | www.frontiersin.org 5March 2022 | Volume 13 | Article 797150

Abdel-Wahab et al. TRD: A Guideline for Egyptian Psychiatrists

FIGURE 2 | Schematic diagrams of switching strategies. (A) Concurrent switch is best suited for patients demonstrating partial response, where simultaneous

change in the dose of both medications is implemented. (B) Overlapping switch is suitable for patients who demonstrate partial response, where the dose of the

original medication is maintained until the second medication reaches its optimal dose. (C) Sequential switch is considered to be the safest switching technique, as it

is the least likely to cause any interaction. In this strategy, one medication is substituted with another. This technique is used in patients who do not respond to initial

treatment.

Treatment Sequence for Depression

Dimensions

Major depressive disorder is a multidimensional disorder that

originates from multiple etiologies. The symptom dimension

may act as predictor of antidepressant treatment response (35).

The panel recommendations for the ﬁrst and second lines

of treatment of several depression dimensions are illustrated in

Table 2.

Brain Stimulation Techniques

The panel selected ECT/BST, and repetitive transcranial

magnetic stimulation (rTMS) as the preferred brain stimulation

techniques that are either used alone or in combination with

antidepressants. The panel agreed that ECT/BST represent

eﬀective approaches to prevent relapse, either alone or in

combination. Recommendations of brain stimulation techniques

are as follows:

BST is recommended for resistant cases after failure of 3

adequately used antidepressants.

Novel Therapeutic Agents

Esketamine

Esketamine, the S enantiomer of ketamine, is a N-methyl-D-

aspartate (NMDA) receptor antagonist that has higher aﬃnity

to the receptor compared to the R enantiomer and the racemic

mixture of ketamine (36). In the form of a nasal spray, this

molecule has recently gained approval in the United States for

the indication of TRD as well as by the EMA in Europe for the

indication of TRD, bringing hope to patients who suﬀer from the

condition (37).

Esketamine, in combination with SSRIs or SNRIs, is indicated

for the treatment of TRD in patients who did not respond to least

2 diﬀerent classes of antidepressant medications (38).

Strategies to Prevent Relapse

A relapse is the return of depressive symptoms to patients.

Relapse could be early or delayed. In the former, symptoms return

is expected to be within the ﬁrst 3–12 months, while the latter

refers to the emergence of new depressive episodes following

remission or initial short-term improvement in symptoms (39).

It is of importance to have measures to prevent relapse from

the ﬁrst relapse episode. With regard to preventive strategies, the

following has been highlighted:

The panel recommends the use of ECT and lithium as eﬀective

ﬁrst-line options to prevent relapse.

The panel recommends the use of lamotrigine or quetiapine

as second-line options to prevent relapse.

The panel recommends the following strategies when patients

achieve full remission:

- Continuous assessment of patients’ adherence to treatment.

- Continuous assessment of social functioning of patients.

- Continuous assessment of quality of life (QoL) of patients.

The panel recommends the following complementary

approaches for relapse and recurrence prevention:

- Undertaking regular physical exercises and activities.

- Eating healthy food

- Control physical illness (e.g., hypertension, diabetes, etc.. . . )

DISCUSSION

Treatment resistant depression is regarded as patients failing

2 subsequent antidepressant treatments (38); in Egypt TRD’s

deﬁnition is slightly altered to the latter by the addition of

failure of 6–12 sessions of ECT or BST (Brain Stimulation

Techniques). Approximately 60–70% of depressed patients do

not respond to ﬁrst line treatment and more than a third become

treatment resistant. However, TRD should not be confused with

“pseudo-resistance”. This means that patient-related factors that

might contribute treatment failure should be taken into account

before deeming their depression as “resistant”. These factors are

numerous, however inadequate dosing, compliance, follow-up

and primary mis-diagnosis are prime examples (38). Another

review also added other patient traits; the presence of comorbid

psychiatric illnesses as OCD (Obsessive Compulsive Disorder),

bipolarity, anxiety and eating disorders as well as the presence

of psycho-somatic disorders as Fibromyalgia and IBS (Irritable

Bowel Syndrome), all of which must be assessed carefully (with

several available diagnostic tools) as they may lead to high rates of

depression recurrence and severity; higher severity of depressive

Frontiers in Psychiatry | www.frontiersin.org 6March 2022 | Volume 13 | Article 797150

Abdel-Wahab et al. TRD: A Guideline for Egyptian Psychiatrists

FIGURE 3 | Recommendations for switching antidepressant medications. If patient is responsive continue on current treatment, if not consider combination

strategies** Adapted from the French guidelines. **Combination strategy =The panel recommends the use of combination strategy in patients with partial response

after adequate treatment with a medication for a period of 2–4 weeks.

Frontiers in Psychiatry | www.frontiersin.org 7March 2022 | Volume 13 | Article 797150

Abdel-Wahab et al. TRD: A Guideline for Egyptian Psychiatrists

TABLE 1 | Therapeutic lines in augmentation strategy in non-psychotic patients.

Potentiation treatment

First choice

• Lithium with serum level must be at least 0.8 mmol/L

• Second generation anti-psychotic with antidepressant action (Quetiapine,

Asenapine, Iloperidone, Brexpiprazole, Laurasidone, Cariprazine)

Second choice

• Aripiprazole

• Tri-iodothyronine

• Lamotrigine

TABLE 2 | Treatment lines recommendations for clinical dimensions in MDD.

Dimension First choice

With marked anhedonia NDRI or SNRI

With marked psychomotor

retardation

SNRI or NDRI

With marked sleep disturbances SSRI or SNRI or Mirtazapine or agomelatine

With atypical features

(hyperphagia, hypersomnia)

SSRI or SNRI

With psychotic features SNRI in monotherapy or SSRI in combination

with an atypical 2nd generation antipsychotic

with an antidepressant action

With anxious features SSRI or SNRI or Mirtazapine or Lithium

With high suicidal risk SSRI or SNRI or Mirtazapine or Lithium or 2nd

generation antipsychotic with an

antidepressant action

Positive family history of bipolar

disorder or suicide

Mood stabilizer (Lithium)

illness and the higher rates of relapses in between remission

periods is deﬁnitely a precipitating factor for TRD (39).

The panel urges that patients are monitored for both

psychiatric and somatic diseases; this is reinforced by Kornstein’s

data who reported that patients with hypothyroidism and

depression had higher remission rates when properly

treated for under-active thyroid, this agrees with the panel’s

recommendation in augmenting therapy with T3 hormone

supplementation. Patient history and concurrent medications

must be known as some drugs such as gluco-corticosteroids

and anti-inﬂammatories are associated with causing depressive

symptoms (39). Ergo, the panel suggests using a list of tools

and tests as MRIs, blood work, organ function tests etc. to

monitor for certain patient conditions that may cause depression

treatment failure or increase severity of the illness.

The panel agrees with the international guidelines for

hospitalization indications in patients with TRD, this is because

hospitalization prevents further complications that may be

caused if TRD and/or its comorbidities are not under continuous

medical supervision (8). A study conducted in Finland on a

population of hospitalized patients as a result of depression

followed-up patients up to 24 years after discharge to evaluate

their outcomes. Of about 15,000 patients followed-up; only 2,567

died by suicide with an overall cumulative risk of suicide of

6.13%; which is considerably lower than depressed patients who

remain unhospitalized (40).

Regarding comorbid mental illnesses with depression

the panel recommends using several drugs in addition to

antidepressants; the anxiolytics used is in line with the Canadian

clinical practice guidelines; generally, benzodiazepines are ﬁt for

all anxiety disorders; Buspirone is indicated for a wide range of

anxiety diseases as panic disorder, social anxiety, OCD (Obsessive

Compulsive Disorder), GAD (Generalized Anxiety Disorder)

and PTSD (Post-traumatic Stress Disorder), Pregabalin however,

is used as second line treatment if ﬁrst line drugs are not

tolerated. For sleep disorders and self-harm comorbid with TRD,

the panel’s recommendations are based on clinical experience;

which of course at times conﬂicts with published research. A

Cochrane meta-analysis had similar documented patients who

were prescribed drugs similar to the panel’s recommendations as

atypical anti-psychotics and benzodiazepines for both self-harm

and sleep disorders (41).

Physicians however, must be aware of which drug to add

to the patient’s treatment plan and patient monitoring and

continuous follow-up is crucial to their wellbeing. In Table 3,

the panel has several recommendations of which medications to

prescribe if TRD were associated with any other clinical issue

such as sleep disturbances, suicide ideation, anxious features etc.

Mirtazapine is an excellent antidepressant with a wide array of

indications (and oﬀ-label uses) and has demonstrated superiority

to tricyclic antidepressants due to its lack of any anti-cholinergic,

adrenergic and serotonin-mediated side eﬀects; clinical evidence

has proved that it has transferred the treatment of depression.

In comparison to tricyclic antidepressants as amitriptyline,

clomipramine, doxepin and serotonin modulator trazodone, the

clinical eﬀect of mirtazapine was similar and at times superior,

not only that, but it is also strongly advocated in cases of

depression with poor sleep scores in which mirtazapine improved

drastically in comparison to placebo. According to these ﬁndings,

mirtazapine’s adverse eﬀects as dry mouth, increased appetite

and weight gain should be weighed together with its high

safety and eﬀective treatment proﬁle, when exploring other

drugs to prescribe/augment/combine with the current treatment

proﬁle (42).

Another interesting agent that can be used and is

recommended by Egyptian psychiatrists is agomelatine,

one of the ﬁrst melatonergic agents and a 5-hydroxytryptamine

receptor (5-HT2C) antagonist both of which act harmoniously

to adjust disrupted circadian rhythms (sleep cycles) typically

found in depressive illness. As mentioned above, depressed

patients must be closely monitored to avoid any consequences

unintentionally caused by treatment; agomelatine should not be

used in patients with a compromised liver and in healthy patients,

liver function tests must be routinely done at the beginning of

treatment and subsequently at 6 weeks, 12 weeks and 6 months as

recommended by the EMA (European Medicines Agency) (43).

Moving forward, a pharmacological treatment algorithm was

devised as mentioned above in Figure 3; there are several

strategies that physicians can adhere to when approaching

TRD, these include switching, combination or augmentation

strategies. A study in 2001 evaluated patients wth low response

to Fluoxetine 20 mg/day who were switched to mianserin 60

mg/day. The results were intermediate but were still sound with

depression scores lower by 1.8 in the mianserin group than those

Frontiers in Psychiatry | www.frontiersin.org 8March 2022 | Volume 13 | Article 797150

Abdel-Wahab et al. TRD: A Guideline for Egyptian Psychiatrists

TABLE 3 | The main ﬁndings of this study.

Area of interest Recommendation

Hospitalization: indications The panel members recommended the consideration of psychiatric hospitalization for the following cases.

- Catatonia

- Patients with high suicidal tendency

- Severe psychotic symptoms

- Advanced cases of MDD

- The presence of severe uncontrolled comorbid medical conditions

- Insufﬁcient familial support to the patient

Therapeutic options for treatment-resistant

depression and comorbid conditions

pharmacotherapeutic options of comorbid

conditions

For patients suffering from comorbid conditions, the expert panel recommends the following:

Patients suffering from anxiety and its features are recommended to receive benzodiazepines. Buspirone and pregabalin are

also available options.

Patients suffering from sleep disorders are recommended to receive adjunctive hypnotic medications, such as zolpidem

or zopiclone.

Patients at high risk of self-inﬂicted injury are recommended to receive lithium, benzodiazepine, or second-generation

antipsychotic medication.

Treatment duration The clinical committee members recommended that patients are maintained on their ongoing

antidepressant medication for a period of 9–12 months following the achievement of clinical remission.

Patients suffering from the following conditions are recommended to receive a longer course of treatment:

- Long period to reach remission

- History of 2 prior depression episodes

- History of early relapse after treatment discontinuation

- Presence of suicidal tendency, symptoms of psychosis, family history of suicide or mood disorders or comorbid psychiatric

condition.

- Resistance to an antidepressant medication when given at proper dose and adequate duration.

Therapeutic options of patients suffering

from TRD

The panel experts highlighted the available treatment options for patients suffering from TRD. The available therapeutic

options are captured in Figure 1.

Pharmacological strategies in treatment-resistant depression

Switching strategies The panel experts recommend switching to be carried out in the following situations:

- Lack of response or poor tolerance to initial treatment

- Prior response to the introduced medication

The clinical experts highlighted 3 different types of switching strategies, with concurrent switching being the most

recommended, except for medications belonging to monoamine oxidase inhibitor (MAOIs) class. These strategies are

captured in Figure 2 (32).

Recommendations for switching antidepressant medications are captured in Figure 3.

**Combination strategies The panel recommends the use of combination strategy in patients with partial response after adequate treatment with a

medication for a period of 2–4 weeks (4–6 weeks with TCAs).

The recommended ﬁrst-line combination strategy involves Mirtazapine plus one of the following:

- Selective serotonin reuptake inhibitor.

- Serotonin–norepinephrine reuptake inhibitor.

- Tricyclic antidepressant.

Augmentation strategies The panel recommends this strategy in patients who demonstrate partial response after 2–4 weeks of treatment (4–6 weeks

if on TCAs).

The panel recommends adding lithium or quetiapine to improve efﬁcacy of the antidepressant medication.

The panel recommends a second choice of thyroid hormone supplementation in addition to serotonin–norepinephrine

reuptake inhibitors or tricyclic antidepressants, and with selective serotonin reuptake inhibitors or mirtazapine at a later stage.

The recommended dose of thyroid hormone supplementation is between 25–60 ug/day of liothyronine (L-T3) and is required

to achieve TSH levels ranging between 0.1 and 1 ug/L. Prior to initiating treatment, the panel recommends performing the

following assessments:

- Physical examination

- Electrocardiogram (ECG)

- Thyroid-stimulating hormone (TSH) levels (Table 1).

Treatment sequence for depression

dimensions

The panel recommendations for the ﬁrst and second lines of treatment of several depression dimensions are illustrated in

Table 2.

Brain stimulation techniques (BST) The panel selected ECT/BST, and repetitive transcranial magnetic stimulation (rTMS) as the preferred brain stimulation

techniques that are either used alone or in combination with antidepressants.

The panel agreed that ECT/BST represent effective approaches to prevent relapse, either alone or in combination.

Recommendations of Brain Stimulation Techniques are as follows: BST is recommended for resistant cases after failure of 3

adequately used antidepressants.

Novel therapeutic agents Esketamine, in combination with SSRIs or SNRIs, is indicated for the treatment of TRD in patients who did not respond to

least 2 different classes of antidepressant medications (38).

Strategies to prevent relapse It is of importance to have measures to prevent relapse from the ﬁrst relapse episode. With regard to preventive strategies,

the following has been highlighted:

(Continued)

Frontiers in Psychiatry | www.frontiersin.org 9March 2022 | Volume 13 | Article 797150

Abdel-Wahab et al. TRD: A Guideline for Egyptian Psychiatrists

TABLE 3 | Continued

Area of interest Recommendation

The panel recommends the use of ECT and lithium as effective ﬁrst-line options to prevent relapse.

The panel recommends the use of lamotrigine or quetiapine as second-line options to prevent relapse. The panel

recommends the following strategies when patients achieve full remission:

- Continuous assessment of patients’ adherence to treatment.

- Continuous assessment of social functioning of patients.

- Continuous assessment of quality of life (QoL) of patients.

The panel recommends the following complementary approaches for relapse recurrence prevention:

- Undertaking regular physical exercises and activities.

- Eating healthy food

- Control physical illness (e.g., hypertension, diabetes, etc.…)

who continued on Fluoxetine. The same study also had a third

treatment arm with patients on ﬂuoxetine therapy combined

with mianserin (=combination strategy) in which the depression

score plummeted by 4.6 in the combined treatment group; this

is a commonly observed phenomenon in combination strategies

as both drugs synergistically act to create a larger overall eﬀect

in managing symptoms especially of a multi-modal disease as

depression (44).

Furthermore, augmentation strategy is adding a diﬀerent

class of medication to a current antidepressant; this was

done in a study where randomized depressed patients who

remitted from ECT.They received placebo, lithium or lithium

with nortriptyline; 84% relapsed on placebo, 60% on lithium

monotherapy and 39.1% on the combination therapy which

consolidates the fact that combination therapy with non-

antidepressant agent proves useful in treatment of depression

as well as preventing relapses (45). There is also evidence that

suggests that Lithium decreases the risk of suicide in depressed

patients (46).

There are other methods of treatment in TRD to resort to such

as ECT (a brain stimulation technique) or the use of esketamine

(a relatively novel therapeutic agent). ECT therapy has long

been recommended by the British guidelines of 2000 for severe

cases of depression especially those who have failed two or more

drugs (TRD), rTMS is recommended second to ECT but must be

done by a team of specialists. Ghasemi made a more interesting

discovery when comparing patients who received three sessions

of ECT vs. those who received low dose esketamine over 3 days;

results supported that both treatments although comparable,

esketamine had a more rapid and more pronounced resolution

of symptoms (47). The use of esketamine is not yet approved

in Egypt; although it has long been approved by the FDA in

the United States and the EMA in Europe for its relative safety

and eﬃcacy; moreover, adverse eﬀects of dissociation, vertigo

and dizziness from esketamine usually resolved on the same

day, shortly after administration; (48) future-wise; the use of

esketamine (only as nasal spray) must be warranted in Egypt

due to its pronounced eﬀect with patients feeling better within

hours of administration and lower relapse rates as maintenance

treatment all of which is vital for a TRD patient.

Following achieving remission for depression; a major

challenge is to prevent relapse of depressive episodes; the use

of lithium is widely agreed upon in patients who were suicidal,

ECT is proposed for patients with frequent relapses. Physiological

wellbeing is also an aspect to be considered; any emerging or

residual mental or physical illness should be tended to in order

to prevent relapse (46).

Like all studies, this study carries its own strengths and

limitations. Surely the ﬁrst strength of this study is that this is the

ﬁrst consensus of guidelines between Egyptian psychiatrists for

the treatment of TRD; the panel’s recommendations came from

practicing them on TRD patients in an expert clinical setting.

The limitation of this study is that the panel constituted of only

eight doctors who only represent a small group of psychiatrists

from a much larger number in Egypt; additionally, there is

no scientiﬁcally exact or oﬃcial deﬁnition of TRD or for its

treatment of TRD in Egypt or world-wide (internationally) to

base these recommendations on; they were merely observations

collected by the panel experts.

CONCLUSION

The integration of clinical practice with the latest updates of

clinical studies yields the best outcomes for patients. TRD puts

a signiﬁcant burden on the patient, therefore, it is always best

to manage the condition with careful review and step-wise

approach. The development of recent pharmaceutical options for

patients with TRD ushers a new area to tackle this condition.

DATA AVAILABILITY STATEMENT

The original contributions presented in the study are included

in the article/Supplementary Material, further inquiries can be

directed to the corresponding author/s.

AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct, and intellectual

contribution to the work, participated in its writing and approved

it for publication.

SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found

online at: https://www.frontiersin.org/articles/10.3389/fpsyt.

2022.797150/full#supplementary-material

Frontiers in Psychiatry | www.frontiersin.org 10 March 2022 | Volume 13 | Article 797150

Abdel-Wahab et al. TRD: A Guideline for Egyptian Psychiatrists

REFERENCES

1. Uher R, Payne J, Pavlova B, Perlis R. Major depressive disorder in DSM-

5: implications for clinical practice and research of changes from DSM-IV.

Depress Anxiety. (2013) 31:459–71. doi: 10.1002/da.22217

2. Bains N, Abdijadid S. Major Depressive Disorder. Treasure Island, FL:

StatPearls Publishing (2021) Available online at: https://www.ncbi.nlm.nih.

gov/books/NBK559078/ (accessed April 20, 2021).

3. Available online at: https://www.who.int/en/news-room/fact- sheets/detail/

depression (accessed December 24, 2021).

4. Odejimi O, Tadros G, Sabry N. A systematic review of the prevalence of mental

and neurocognitive disorders amongst older adults’ populace in Egypt. Middle

East Curr Psychiatry. (2020) 27:47. doi: 10.1186/s43045-020-00055-8

5. Fava M. Switching treatments for complicated depression. J Clin Psychiatry.

(2010) 71:e04. doi: 10.4088/JCP.8001tx14c

6. Voineskos D, Daskalakis ZJ, Blumberger DM. Management of treatment-

resistant depression: challenges and strategies. Neuropsychiatr Dis Treat.

(2020) 16:221–34. doi: 10.2147/NDT.S198774

7. Rush AJ, Trivedi MH, Wisniewski SR, Nierenberg AA, Stewart JW, Warden

D, et al. Acute and longer-term outcomes in depressed outpatients requiring

one or several treatment steps: a STAR∗D report. Am J Psychiatry. (2006)

163:1905–17. doi: 10.1176/ajp.2006.163.11.1905

8. Bennabi D, Charpeaud T, Yrondi A, Genty JB, Destouches S,

Lancrenon S, et al. Clinical guidelines for the management of

treatment-resistant depression: French recommendations from

experts, the French Association for Biological Psychiatry and

Neuropsychopharmacology and the fondation FondaMental. BMC Psychiatry.

(2019) 19:262. doi: 10.1186/s12888-019-2237-x

9. Hassan A, Farmer K, Brahm N, Neas B. Evaluating antidepressant

treatment prior to adding second-line therapies among patients

with treatment-resistant depression. Int J Clin Pharm. (2016)

38:429–37. doi: 10.1007/s11096-016-0272-y

10. de Villiers MR, de Villiers PJ, Kent AP. The Delphi technique

in health sciences education research. Med Teach. (2005)

27:639–43. doi: 10.1080/13611260500069947

11. Gaynes B, Lux L, Gartlehner G, Asher G, Forman-Hoﬀman V, Green J,

et al. Deﬁning treatment-resistant depression. Depress Anxiety. (2019) 37:134–

45. doi: 10.1002/da.22968

12. Available online at: https://www.accessdata.fda.gov/drugsatfda_docs/label/

2014/017090s078lbl.pdf (accessed December 24, 2021).

13. Available online at: https://www.accessdata.fda.gov/drugsatfda_docs/label/

2011/018936s091lbl.pdf (accessed December 24, 2021).

14. Available online at: https://www.accessdata.fda.gov/drugsatfda_docs/label/

2017/020699s107lbl.pdf (accessed December 24, 2021).

15. MacGillivray S. Eﬃcacy and tolerability of selective serotonin reuptake

inhibitors compared with tricyclic antidepressants in depression treated

in primary care: systematic review and meta-analysis. BMJ. (2003)

326:1014. doi: 10.1136/bmj.326.7397.1014

16. Zajecka J, Albano D. SNRIs in the management of acute major depressive

disorder. J Clin Psychiatry. (2004) 65:11–8.

17. Gronemann F, Petersen J, Alulis S, Jensen K, Riise J, Ankarfeldt M, et al.

Treatment patterns in patients with treatment-resistant depression in Danish

patients with major depressive disorder. J Aﬀect Disord. (2021) 287:204–

13. doi: 10.1016/j.jad.2021.03.029

18. Iannuzzo R, Jaeger J, Goldberg J, Kafantaris V, Sublette M.

Development and reliability of the HAM-D/MADRS interview: an

integrated depression symptom rating scale. Psychiatry Res. (2006)

145:21–37. doi: 10.1016/j.psychres.2005.10.009

19. Forty L, Smith D, Jones L, Jones I, Caesar S, Fraser C, et al.

Identifying hypomanic features in major depressive disorder

using the hypomania checklist (HCL-32). J Aﬀect Disord. (2009)

114:68–73. doi: 10.1016/j.jad.2008.07.017

20. Chioqueta A, Stiles T. Psychometric Properties of the Beck Scale for Suicide

Ideation: A Norwegian Study With University Students Trondheim: Taylor &

Francis Company (2021).

21. Posner K, Brown G, Stanley B, Brent D, Yershova K, Oquendo M, et al.

The Columbia–suicide severity rating scale: initial validity and internal

consistency ﬁndings from three multisite studies with adolescents and adults.

Am J Psychiatry. (2011) 168:1266–77. doi: 10.1176/appi.ajp.2011.10111704

22. Nobis A, Zalewski D, Waszkiewicz N. Peripheral markers of depression. J Clin

Med. (2020) 9:3793. doi: 10.3390/jcm9123793

23. Bathla M, Singh M, Relan P. Prevalence of anxiety and depressive symptoms

among patients with hypothyroidism. Indian J Endocrinol Metab. (2016)

20:468. doi: 10.4103/2230-8210.183476

24. Dhar A, Barton D. Depression and the link with cardiovascular disease. Front

Psychiatry. (2016) 7:33. doi: 10.3389/fpsyt.2016.00033

25. Zubenko G, Sullivan P, Nelson J, Belle S, Huﬀ F, Wolf G. Brain imaging

abnormalities in mental disorders of late life. Arch Neurol. (1990) 47:1107–11.

26. Dunlop B, Davis P. Combination treatment with benzodiazepines and SSRIs

for comorbid anxiety and depression: a review. Prim Care Companion J Clin

Psychiatry. (2008) 10:222–8. doi: 10.4088/PCC.v10n0307

27. Hoge E, Ivkovic A, Fricchione G. Generalized anxiety disorder: diagnosis and

treatment. BMJ. (2012) 345:e7500. doi: 10.1136/bmj.e7500

28. Wagner J, Wagner M. Non-benzodiazepines for the treatment of insomnia.

Sleep Med Rev. (2000) 4:551–81. doi: 10.1053/smrv.2000.0126

29. Wine J, Sanda C, Caballero J. Eﬀects of quetiapine on sleep in

nonpsychiatric and psychiatric conditions. Ann Pharmacother. (2009) 43:707–

13. doi: 10.1345/aph.1L320

30. Doroudgar S, Chou T, Yu J, Trinh K, Pa, J, Perry P. Evaluation of trazodone

and quetiapine for insomnia. Primary Care Compan CNS Disord. (2013)

15:PCC.13m01558. doi: 10.4088/PCC.13m01558

31. Papakostas G. Managing partial response or nonresponse. J Clin Psychiatry.

(2009) 70 (Suppl. 6):16–25. doi: 10.4088/JCP.8133su1c.03

32. Malhi GS, Hitching R, Berk M, Boyce P, Porter R, Fritz K. Pharmacological

management of unipolar depression. Acta Psychiatr Scand Suppl. (2013)

17:6–23. doi: 10.1111/ac–s.12122

33. López-Muñoz F, Álamo C, García-García P. Combination strategies in

treatment-resistant depression. In: López-Muñoz F, Srinivasan V, de Berardis

D, Álamo C, Kato T, editors. Melatonin, Neuroprotective Agents and

Antidepressant Therapy. New Delhi: Springer (2016).

34. Carvalho AF, Cavalcante JL, Castelo MS, Lima MC. Augmentation strategies

for treatment-resistant depression: a literature review. J Clin Pharm Ther.

(2007) 32:415–28. doi: 10.1111/j.1365-2710.2007.00846.x

35. Vares EA, Salum GA, Spanemberg L, Caldieraro MA, Fleck MP.

Depression dimensions: integrating clinical signs and symptoms

from the perspectives of clinicians and patients. PLoS ONE. (2015)

10:e0136037. doi: 10.1371/journal.pone.0136037

36. Swainson J, Thomas RK, Archer S, Chrenek C, MacKay MA. Baker G, et al.

Esketamine for treatment resistant depression. Expert Rev Neurother. (2019)

19:899–911. doi: 10.1080/14737175.2019.1640604

37. Fedgchin M, Trivedi M, Daly EJ, Melkote R, Lane R, Lim P, et al.

Eﬃcacy and safety of ﬁxed-dose esketamine nasal spray combined with

a new oral antidepressant in treatment-resistant depression: results of a

randomized, double-blind, active-controlled study (TRANSFORM-1). Int J

Neuropsychopharmacol. (2019) 22:616–30. doi: 10.1093/ijnp/pyz039

38. Daniel S, Papakostas G, Trivedi M. Treatment-resistant depression. J Clin

Psychiatry. (2006) 67 (Suppl 6):16-22.

39. Kornstein S, Schneider R. Clinical features of treatment-resistant depression.

J Clin Psychiatry. (2001) 62:18–25.

40. Aaltonen K, Isometsä E, Sund R, Pirkola S. Decline in suicide mortality after

psychiatric hospitalization for depression in Finland between 1991 and 2014.

World Psychiatry. (2018) 17:110–2. doi: 10.1002/wps.20501

41. Storebø OJ, Stoﬀers-Winterling JM, Völlm BA, Kongerslev MT, Mattivi

JT, Jørgensen MS, et al. Psychological therapies for people with

borderline personality disorder. Cochrane Database Syst Rev. (2020)

5:1–512. doi: 10.1002/14651858.CD012955.pub2

42. Kasper S, Praschak-Rieder N, Tauscher J, Wolf R. A risk-beneﬁt

assessment of mirtazapine in the treatment of depression. Drug

Safety. (1997) 17:251–64. doi: 10.2165/00002018-199717040-0

0005

43. de Bodinat C, Guardiola-Lemaitre B, Mocaër E, Renard P, Muñoz C,

Millan M. Agomelatine, the ﬁrst melatonergic antidepressant: discovery,

characterization and development. Nat Rev Drug Discov. (2010) 9:628–

42. doi: 10.1038/nrd3140

Frontiers in Psychiatry | www.frontiersin.org 11 March 2022 | Volume 13 | Article 797150

Abdel-Wahab et al. TRD: A Guideline for Egyptian Psychiatrists

44. Ferreri M, Lavergne F, Berlin I, Payan C, Puech A. Beneﬁts from

mianserin augmentation of ﬂuoxetine in patients with major depression

non-responders to ﬂuoxetine alone. Acta Psychiatr Scand. (2008) 103:66–

72. doi: 10.1034/j.1600-0447.2001.00148.x

45. Sackeim H, Haskett R, Mulsant B, Thase M, Mann J, Pettinati

H, et al. Continuation pharmacotherapy in the prevention of

relapse following electroconvulsive therapy. JAMA. (2001) 285:1299.

doi: 10.1001/jama.285.10.1299

46. Anderson I, Ferrier I, Baldwin R, Cowen P,Howard L, Lewis G, et al. Evidence-

based guidelines for treating depressive disorders with antidepressants: a

revision of the 2000 British Association for Psychopharmacology guidelines. J

Psychopharmacol. (2008) 22:343–96. doi: 10.1177/0269881115581093

47. Ghasemi M, Kazemi M, Yooseﬁ A, Ghasemi A, Paragomi P,

Amini H, et al. Rapid antidepressant eﬀects of repeated doses of

ketamine compared with electroconvulsive therapy in hospitalized

patients with major depressive disorder. Psychiatry Res. (2014)

215:355–61. doi: 10.1016/j.psychres.2013.12.008

48. Katz E, Hough D, Doherty T, Lane R, Singh J, Levitan B. Beneﬁt–risk

assessment of esketamine nasal spray vs. placebo in treatment-resistant

depression. Clin Pharmacol Ther. (2020) 109:536–46. doi: 10.1002/cpt.2024

Conﬂict of Interest: The authors declare that the research was conducted in the

absence of any commercial or ﬁnancial relationships that could be construed as a

potential conﬂict of interest.

Publisher’s Note: All claims expressedin this article are solely those of the authors

and do not necessarily represent those of their aﬃliated organizations, or those of

the publisher, the editors and the reviewers. Any product that may be evaluated in

this article, or claim that may be made by its manufacturer, is not guaranteed or

endorsed by the publisher.

Samy, Hamed and Ali. This is an open-access article distributed under the terms

of the Creative Commons Attribution License (CC BY). The use, distribution

or reproduction in other forums is permitted, provided the original author(s)

and the copyright owner(s) are credited and that the original publication in

this journal is cited, in accordance with accepted academic practice. No use,

distribution or reproduction is permitted which does not comply with these

terms.

Frontiers in Psychiatry | www.frontiersin.org 12 March 2022 | Volume 13 | Article 797150

Frontiers in Psychology | www.frontiersin.org 1 April 2022 | Volume 13 | Article 807935

ORIGINAL RESEARCH

published: 01 April 2022

doi: 10.3389/fpsyg.2022.807935

Edited by:

Tindara Caprì,

National Research Council of Italy

(CNR), Italy

Reviewed by:

Julian Packheiser,

Ruhr University Bochum, Germany

Natale Adolfo Stucchi, University of

Milano-Bicocca, Italy

*Correspondence:

Yasmina Crespo Cobo

yasmina.cc.mi@hotmail.com

†These authors have contributed

equally to this work

Specialty section:

This article was submitted to

Psychology for Clinical Settings,

a section of the journal

Frontiers in Psychology

Received: 02 November 2021

Accepted: 17 January 2022

Published: 01 April 2022

Citation:

Crespo Cobo Y, Kandel S,

Soriano MF and

Iglesias-Parro S (2022) Examining

Motor Anticipation in Handwriting as

an Indicator of Motor Dysfunction in

Schizophrenia.

Front. Psychol. 13:807935.

doi: 10.3389/fpsyg.2022.807935

Examining Motor Anticipation in

Handwriting as an Indicator of Motor

Dysfunction in Schizophrenia

YasminaCrespo Cobo

*†, SoniaKandel

2†, MaríaFelipaSoriano

3† and

SergioIglesias-Parro

1 Department of Methodology of Behavioral Sciences, University of Jaén, Jaén, Spain, 2 Département Parole et Cognition,

Université Grenoble Alpes, Grenoble, France, 3 Mental Health Unit, St. Agustín Universitary Hospital, Linares, Spain

Dysfunction in motor skills can belinked to alterations in motor processing, such as the

anticipation of forthcoming graphomotor sequences. Weexpected that the difculties in

motor processing in schizophrenia would bereected in a decrease of motor anticipation.

In handwriting, motor anticipation concerns the ability to write a letter while processing

information on how to produce the following letters. It is essential for fast and smooth

handwriting, that is, for the automation of graphomotor gestures. In this study, weexamined

motor anticipation by comparing the kinematic characteristics of the rst l in the bigrams

ll and ln written on a digitiser. Previous studies indicated that the downstroke duration of

the rst l is modulated by the anticipation of the local constraints of the following letter.

Twenty-four adult individuals with diagnosis of schizophrenia and 24 healthy adults

participated in the study. The classic measures of duration (sec), trajectory (cm), and

dysuency (velocity peaks) were used for the kinematic analysis of the upstroke (US) and

downstroke (DS). In the control group, the duration of the downstroke of the l was longer

in ln than ll (US: ln = ll; DS: ln > ll) whereas no differences were found for the group with

schizophrenia. Likewise, the control group showed a longer DS trajectory for the l of ln

than ll in downstrokes, while the group of patients failed to show this effect. These results

suggest that the motor alterations in patients with schizophrenia could also affect their

ability for motor anticipation.

Keywords: handwriting, schizophrenia, motor anticipation, motor alterations, kinematical measurement

INTRODUCTION

Experimental and neuropsychological models consider that word writing results from a series

of central and peripheral processes that function according to a hierarchical manner (Van

Galen, 1991; Bonin et al., 2001; Miceli and Capasso, 2006; Damian and Stadthagen-Gonzalez,

2009; Kandel etal., 2011, 2017; Ellis, 2014). e central processes refer to linguistic processing,

such as gearing up the semantic system, syntactic construction, and orthographic retrieval.

e peripheral processes point to the motor-related aspects of letter production, where graphomotor

planning for handwriting takes place. ey are involved in the selection of allographs, where

motor program retrieval takes place, local parameter adjustments and muscular activation

leading to the production of letters (Bertram et al., 2015). e present research focuses on

Crespo Cobo et al. Handwriting in Schizophrenia

Frontiers in Psychology | www.frontiersin.org 2 April 2022 | Volume 13 | Article 807935

the latter, lower level peripheral aspects of writing. Of particular

interest for the purpose of the study is the phenomenon of

motor anticipation. Motor anticipation refers to the ability to

predict future behaviors, related to the perception of trajectories

and synchronization of movements (Oña et al., 1999). Motor

anticipation can be considered as a relevant motor-perceptual

process in most learned behaviors (Kandel et al., 2000), and

its dysfunction can be linked to some motor disorders as the

ones observed in schizophrenia (Finney, 2015).

Traditionally, research has employed diverse measures of

handwriting in the study of motor symptoms in psychotic

disorders (Caligiuri etal., 2015; Gawda, 2016). Motor symptoms

were rst studied in psychosis as side eects of the antipsychotic

treatment (Chengappa et al., 1994; Simpson and Lindenmayer,

1997). However, recently, the role of motor symptoms in

psychotic disorders has been revised (Jahn etal., 2002; Rogowska

et al., 2003) and they have been considered as a core feature

in the evaluation and the prognosis of the disorder. In this

regard, they have been detected in antipsychotic naïve patients

with a rst psychotic episode (Peralta et al., 2010), and

handwriting measures have revealed spontaneous motor

abnormalities even in individuals at high risk of psychosis

who have never been in pharmacological treatment (Dean

etal., 2013, 2014). e present study examined a novel measure

of handwriting, motor anticipation in handwriting, in order

to gain insight on motor dysfunctions in schizophrenic patients.

Planning and execution of complex sequences of movements

involve a signicant amount of look-ahead. In fact, units of

motor action being executed oen carry the imprint of yet-to-

be-executed units. More precisely, motor anticipation in ne

motor skills, such as writing, concerns the ability to write a

letter while processing information on how to produce the

following letters. It is essential for fast and smooth handwriting,

that is, for the automation of graphomotor gestures. In this

study, schizophrenic patients and healthy adults had to write

cursive letters on a digitizer. We compared the productions

of the two groups on their ability to anticipate forthcoming

motor sequences in cursive handwriting.

Previous studies carried out by Orliaguet and Boë (1990)

with healthy adults indicated that the anticipation of the local

production constraints of the following letter modulates the

spatio-temporal course of the current movement. ese authors

compared the kinematic characteristics of the rst l in bigrams

ll and ln written on a digitizer (Figure 1).

ey observed that the anticipatory processing of changes

in size and rotation direction of the n increased the downstroke

duration of the l with respect to the rst l of ll, where the

same motor program is reproduced (Orliaguet and Boë, 1990;

Boë etal., 1991). Furthermore, Kandel and Perret (2015) studied

motor anticipation in children at the period of writing automation.

Children of ages 8, 9, and 10 years wrote bigrams ll, le, ln in

cursive writing on a digitizer. ey analyzed the duration,

trajectory, and dysuency of the rst letter l, both on the

upstroke and downstroke. ey found that at all ages, the l’s

downstroke duration was shorter for ll than le and the latter

was in turn shorter than ln. is modulation of the l duration

reects that during this downstroke movement, the children

were processing in advance the following letter. e measures

of the length of the paths of the children’s productions further

revealed that the trajectories of the l of ll were shorter than

those of the l of bigrams le and ln. e dysuency data—

measured as the number of absolute velocity peaks in the

velocity prole for each stroke—indicated that at age 8, dysuency

values were equivalent for upstrokes and downstrokes, whereas

children of ages 9 and 10 years old showed more dysuency

on downstrokes than upstrokes. is experiment suggested that

learning to anticipate in handwriting production requires:

(a) rendering the movements to produce the upstroke constant;

and (b) modulating the downstroke as a function of the spatial

characteristics of the following letter. e pattern of movement

time data suggested that motor anticipation would start to

be adult-like at around age of 9. In other words, motor

anticipation is already present at age 8 and is a core component

of the automation process of handwriting production.

Motor anticipation has also been studied in Parkinson’s

patients (Bidet-Ildei etal., 2011). In this study, motor anticipation

of the control group was compared with the clinical sample

before and aer a treatment phase of dopaminergic medication

or bilateral deep brain stimulation. e results showed that

the control group participants exhibited signs of anticipation

whereas Parkinson’s patients did not. More specically, the

downstroke duration of the rst l of the healthy adults increased

as the constraints of the following letter increased, such that

(ll < le < ln). In the group of patients, there were no dierences

in the duration of the downstroke between dierent conditions.

However, aer treatment, the patients did exhibit a decrease

of the l downstroke duration when it was followed by another l.

As mentioned above, motor anticipation is a key process

for handwriting automation. e production of a movement

while anticipating the requirements of the following motor

sequence facilitates a smooth handwriting and, as motor demands

are reduced, more cognitive resources can bedevoted to higher

order processes, such as linguistic or conceptual processes.

erefore, a poor motor anticipation would result in poor

FIGURE1 | Bigrams ll and ln. The upstroke refers to the ascending stroke

and the downstroke to the descending stroke.

Crespo Cobo et al. Handwriting in Schizophrenia

Frontiers in Psychology | www.frontiersin.org 3 April 2022 | Volume 13 | Article 807935

execution of handwriting and other cognitive–motor tasks

(Lozano and Acosta, 2009; Blanchard et al., 2011; Fett et al.,

2011). is can be observed in some mental disorders, such

as in schizophrenia, which is characterized by a wide variety

of cognitive and motor decits. Besides this, schizophrenia is

associated with increased involuntary movements (Pappa and

Dazzan, 2009) and with neurological abnormalities in sensory

integration, motor coordination, sequencing complex motor

acts, and primitive reexes (Bombin et al., 2005). erefore,

motor-related decits associated with schizophrenia could aect

the processes involved in automation during handwriting. e

main objective of the present research was to examine motor

anticipation in handwriting in schizophrenia. Wehypothesized

that patients with schizophrenia would not beable to anticipate

the production of forthcoming movements. is would result

in an absence of modulation of the kinematic patterns of the

downstroke of the rst letter of the sequence.

As in previous experimental studies on motor anticipation,

we followed the methodology presented by Orliaguet and Boë

(1990). We measured motor anticipation by comparing the

kinematic characteristics of the rst l in the bigrams ll and

ln. e classic measures of duration (movement time in seconds),

trajectory (path in cm), and dysuency (number of velocity

peaks) were used for the kinematic analysis of the upstroke

(US) and the downstroke (DS; see Figure1). Motor anticipation

would bereected in a modulation of the kinematic parameters

of the DS of the rst letter, depending on the following letter

(l vs. n). As in the studies with adults (Orliaguet and Boë,

1990; Bidet-Ildei etal., 2011) and children (Kandel and Perret,

2015), the bigram was written in cursive handwriting on a

digitizing tablet. Wepredicted that the diculties in the cognitive

processes involved in ne motor production in schizophrenia

would bereected in a decrease or absence of this modulation.

MATERIALS AND METHODS

Participants

Twenty-four adult individuals attending the Mental Health Day

Unit at the University St. Agustin Hospital (Spain) participated

in the study. Inclusion criteria were ICD-10 diagnosis of

schizophrenia (F20), and age between 20 and 55 years old

(M = 37.29; SD = 9.58). Diagnosis of participants was made using

a semi-structured interview (SCID-I) according to ICD-10

criteria by the psychiatrist or clinical psychologist in charge

of the patient. Out of the 24 participants, 17 (70.8%) were

male. Twenty-two participants were right-handed whereas 2

were le-handed. eir mean illness duration was 15.36 years

(SD = 10.11). Due to the fact that in Spain there have been

dierent education regulations in the last years, wetransformed

the academic degree reported by the participants in the number

of years needed to obtain it. According to this criterion, the

average number of years in the formal education system in

this group was 10.79 (SD = 4.48 years). To try to better characterize

the educational level of the participants, we categorized the

number of years in the educational system into three other

categories: low educational level (from 0 to 6 years in the formal

educational system), medium educational level (from 7 to

12 years), and high educational level (more than 13 years).

According to these categories, the sample of patients included

12.5% with a low educational level, 62.5% with a medium

level, and 25% with a high level. ere were no patients who

suered from Tardive Dyskinesia: All patients had absent or

minimal symptomatology (a score of 0 or 1 in the items of

the AIMS). In order to compare doses of antipsychotic treatment,

weused Chlorpromazine equivalence (CPZE). CPZE is dened

as the dose of a drug which is equivalent to 100 mg of oral

dose of chlorpromazine (779.37, SD = 419.28).

e SAS rating scale was used for assessment of drug-

induced parkinsonism (Simpson and Angus, 1970). is scale

is used in both clinical practice and research settings, and it

is composed of 10 items: one item measures gait (hypokinesia),

six items measure rigidity, and three items measure glabella

tap, tremor, and salivation. For each item, the severity of the

symptoms was rated from 0 (none) to 4 (severe). A score of

1 in an item indicated the presence of motor symptoms in a

mild form. A mean global score of 3 or more in the full test

was used as a threshold to indicate the presence of the

extrapyramidal symptoms in a mild form (Ayehu etal., 2014).

e mean obtained in our sample was 3.21 (SD = 5.09).

To assess clinical symptoms of schizophrenia, we applied

the Spanish version (Peralta and Cuesta, 1994) of the Positive

and Negative Syndrome Scale (PANSS; Andreasen and Olsen,

1982; Kay etal., 1987). e PANSS is a 30-item rating instrument

comprising three subscales: the seven-item Positive Symptoms

subscale (PANSS-P), the seven-item Negative Symptoms subscale

(PANSS-N), and the 16-item General Psychopathology subscale

(PANSS-G). All 30 items are rated on a seven-point scale

(1 = absent to 7 = extreme). Obtained results were M = 16.04,

SD = 5.39 for PANSS-P, M = 20.22, SD = 6.96 for PANSS-N, and

M = 33.86, SD = 10.00 for PANSS-G.

In order to exclude patients with gross motor dysfunctions,

wemeasured nger and hand dexterity with the Purdue Pegboard

test (Tin and Asher, 1948; Tin, 1968). is board consists

of two parallel rows of 25 holes each. Pins (pegs) are located

at the extreme right-hand and le-hand cups at the top of

the board. Metal collars and washers occupy the two middle

cups. In the rst three subtests, the subject places as many

pins as possible in the holes, rst with the preferred hand

(dominant), then with the non-preferred hand (non-dominant),

and nally with both hands, within a 30-s time period. To

test the right hand, the subject must insert as many pins as

possible in the holes, starting at the top of the right-hand

row (M = 13.74, SD = 8.65). e le-hand test uses the le row

(M = 10.73, SD = 2.95). Both hands then are used together to

ll both rows top to bottom (M = 8.01, SD = 2.25). In the fourth

subtest, the subject uses both hands alternately to construct

“assemblies,” which consist of a pin, a washer, a collar, and

another washer. e subject must complete as many assemblies

as possible within 1 min (M = 28.59, SD = 9.23). We did not

exclude any participants based on their scores on this test.

For the control group, 24 adults were recruited from the

University of Jaén and an adult school of Jaén. e inclusion

criterion was that age was between 20 and 60 years (M = 36.83 years

Crespo Cobo et al. Handwriting in Schizophrenia

Frontiers in Psychology | www.frontiersin.org 4 April 2022 | Volume 13 | Article 807935

old; SD = 12.83 years old). Out of the 24 participants, 14 were

male. All of the participants were right-handed. Regarding

educational level, 3 participants had low level, 8 medium and

19 participants had high education level (M = 13.25, SD = 10).

Importantly, there were no signicant dierences between groups

on age (t = 1.73, p = 0.08), sex (χ2 = 0.82, p = 0.36) or educational

level considered either as the number of years in the educational

system (t = 1.1.59; p = 0.12) or categorized in low, medium or

high level (χ2 = 4.70, p = 0.09).

Exclusion criteria for both groups were: concurrent diagnosis

of neurological disorder, concurrent diagnosis of substance

abuse, history of developmental disability, inability to sign

informed consent or vision disorders (those vision disorders

which, although corrected by glasses or contact lenses, suppose

a loss of visual acuity, e.g., cataracts). In addition, an exclusion

criterion for the control group was the diagnosis of a mental

disorder (according to verbal reports from participants).

All participants gave their written informed consent according

to the Declaration of Helsinki and the Ethics Committee on

Human Research of the Hospital approved the study.

Procedure and Data Analysis

Participants were asked to perform an easy and brief handwriting

task. A A4 paper was axed to the surface of a WACOM

(Intuos pro small) digitizing tablet with dimensions of

269 × 170 × 8 mm (10.6 × 6.7 × 0.3 in), w ith an active area of

160 × 100 mm (6.3 × 3.9 in) and a resolution of 5,080 lpi. e

dierent bigrams (ll, ln) were presented randomly on a computer

screen, and participants were required to write the bigrams

using this paper, with a Wacom Pro Pen 2 (KP504E) digital

pen with 8192 levels of pressure sensitivity. Handwriting tasks

were carried out individually. e task had no time limit.

We measured three dependent variables: the time per stroke

(Duration, seg.), the path of the pen for each stroke (Trajectory,

cm), and the number of velocity peaks (Disuency).

For each of these dependent variables, we run separate

mixed models with Group, Direction and Bigram as independent

variables. Random intercepts were included for subjects. Analyses

were done in R [R Core Team (2020)] using the lmer() function

of the lme4 package (Bates et al., 2015). We utilized the

restricted maximum likelihood as the estimation procedure

and the Welch–Satterthwaite (Luke, 2017) approximation of

the degrees of freedom because of our relatively small sample

size (Gumedze and Dunne, 2011).

RESULTS

Duration

Figure 2 presents mean movement time per stroke across

trials, as a function of group (Schizophrenia-SCZ vs. Control-

CTRL), stroke direction (downstroke-DS vs. upstroke-US),

and type of bigram (ll vs. ln). e ANOVA on duration

revealed a signicant eect was found for Direction [F(1,

138) = 4.83, p = 0.029,

=.033], indicating a longer duration

for DS (M = 0.21, SD = 0.01) than US (M = 0.20, SD = 0.01).

A signicant eect was also found for Bigram [F(1, 138) = 11.69,

p < 0.01,

0.078], indicating a longer duration for ln

(M = 0.21, SD = 0.11) than for ll (M = 0.19, SD = 0.09). e

interaction Group by Direction by Bigram was also signicant

[F(1, 138) = 4.39, p = 0.037,

0.031]. No other eects

were signicant.

In order to analyze the Group by Direction by Bigram

interaction, weconducted pairwise comparisons using emmeans

function in R (Russell Lenth, 2020). In the control group,

we found no signicant dierences in US between LL and

LN (t < 1), but we found signicant longer duration for LN

than for LL (t = −3.32, p = 0.24) in DS. In the schizophrenia

group, wefound no signicant dierences were found between

LL and LN (t < 1) neither in US (t= −2.18, p = 0.36) nor in

DS (t = −1.20, p = 0.93).

Trajectory

Figure3 presents the mean Trajectory each stroke across trials,

as a function of group (Schizophrenia-SCZ vs. Control-CTRL),

stroke direction (downstroke-DS vs. upstroke-US), and type

of bigram (ll vs. ln). Error bars represent standard error. e

ANOVA showed a signicant eect of Group [F(1, 46) = 12.68,

p < 0.01,

0.216] indicating longer trajectory in the SCZ

group (M = 0.84, SD = 0.23) than the Control group (M = 0.65,

SD = 0.18). A marginally eect was also found for Type of

bigram [F(1, 138) = 3.64, p = 0.058], indicating a longer trajectory

for ln (M = 0.76, SD = 0.25) than for ll (M = 0.73, SD = 0.19).

e interaction Group by Direction was signicant [F(1,

138) = 5.87, p < 0.016,

2040

]. e interaction Group by

Direction by Bigram was also signicant [F(1, 138) = 3.91,

p < 0.049,

2027

]. No other eects were signicant.

In order to analyze the Group by Direction by Bigram interaction,

we conducted pairwise comparisons using emmeans function in

R (Russell Lenth, 2020). In the control group, wefound no signicant

dierences in DS between LL and LN (t = −1.98, p = 0.49), but

wefound signicant longer duration for LN than for LL (t= −3.32,

p= 0.24) in US. In the schizophrenia group, wefound no signicant

dierences were found between LL and LN (t < 1) neither in US

(t = −2.18, p = 0.36) nor in DS (t = −1.20, p = 0.93) .

Disuency

Figure 4 presents mean Dysuency values as a function of

group (Schizophrenia-SCZ vs. Control-CTRL), stroke direction

(downstroke-DS vs. upstroke-US), and type of bigram (ll vs.

ln). e results yielded a signicant Group eect [F(1, 48) = 4.34,

p = 0.042,

2082

] indicating more dysuency in the SCZ

group (M = 1.47, SD = 0.98) than in the Control group (M = 1.09,

SD = 0.24). A signicant eect was also found for Direction

[F(1, 144) = 6.31, p = 0.013,

2041=.

], indicating more velocity

peaks for the DS (M = 1.33, SD = 0.81) than US (M = 1.22,

SD = 0.61). No other eects were signicant.

Finally, correlations were carried out between the dierent

measures from motor evaluation scales, psychopathology scales,

and other characteristic variables of the disorder (illness duration,

educational level or pharmacological treatment doses), and the

kinematic measures of handwriting (see Figure 5). We found

Crespo Cobo et al. Handwriting in Schizophrenia

Frontiers in Psychology | www.frontiersin.org 5 April 2022 | Volume 13 | Article 807935

no signicant relationships between kinematic measures of

handwriting and the rest of variables, except for Trajectory,

which correlated with motor functioning values from the Purdue

test: patients with a worse motor function showed longer

trajectories in handwriting.

DISCUSSION

Handwriting becomes automatic with practice. Motor automation

is essential to free up cognitive and attentional resources for the

rest of the components of handwriting: conceptual planning,

FIGURE2 | Mean movement time per stroke (Duration, sec.) across trials, as a function of group (Schizophrenia-SCZ vs. Control-CTRL), stroke direction

(downstroke-DS vs. upstroke-US), and type of bigram (ll vs. ln). Error bars represent standard error.

Crespo Cobo et al. Handwriting in Schizophrenia

Frontiers in Psychology | www.frontiersin.org 6 April 2022 | Volume 13 | Article 807935

syntactical construction, lexical selection, etc. In adult cursive

handwriting, movements are smooth and continuous. Part of

this continuity is due to motor anticipation. It allows for the

programming of the graphomotor constraints of the following

stroke while executing the previous one. Since letters vary in

size and direction of the stroke when they are written in lowercase,

the anticipation of these variations requires a supplementary

cognitive load while preparing the production of the following

FIGURE3 | Mean trajectory (path in cm of the pen) for each stroke across trials, as a function of group (Schizophrenia-SCZ vs. Control-CTRL), stroke direction

(downstroke-DS vs. upstroke-US), and type of bigram (ll vs. ln). Error bars represent standard error.

Crespo Cobo et al. Handwriting in Schizophrenia

Frontiers in Psychology | www.frontiersin.org 7 April 2022 | Volume 13 | Article 807935

letter. is anticipation modulates the spatio-temporal course of

the production movement, which can be observed in some

kinematic parameters of handwriting. Based on this idea, and

following the experimental paradigm presented by Orliaguet and

Boë (1990), the present research evaluated motor anticipation

in patients with schizophrenia and controls. We measured the

kinematic variations in the upstroke (US) and downstroke (DS)

of the rst letter l of a bigram, as a function of the graphomotor

FIGURE4 | Mean Dysuency (number of velocity peaks) across trials, as a function of group (Schizophrenia-SCZ vs. Control-CTRL), stroke direction (downstroke-

DS vs. upstroke-US), and type of bigram (ll vs. ln). Error bars represent standard error.

Crespo Cobo et al. Handwriting in Schizophrenia

Frontiers in Psychology | www.frontiersin.org 8 April 2022 | Volume 13 | Article 807935

constraints of the following letter. Motor anticipation implies a

modulation of the duration of the descending stroke as a function

of size and rotation direction of the following letter (e.g., Kandel

and Perret, 2015). If the forthcoming letter requires a change

in size and direction from one letter to the other as in ln, the

cognitive load is usually reected in an increase in the duration

of the descending stroke compared to a bigram in which the

same motor program is repeated as in ll.

e results supported our main hypothesis: patients with

schizophrenia did not exhibit any sign of motor anticipation.

In the control group instead, changes in letter size and direction

increased the downstroke duration of the rst l, whereas for

the upstrokes it remained unaected (l US: ln = ll; l DS: ln > ll).

e fact that the duration of the downstroke increases as the

spatial parameters of the following letter change, is interpreted

as a sign of motor anticipation. ese results contrast with

those of the schizophrenia group: no dierences were found

in duration according to the type of bigram (ll = ln) or stroke

direction (US = DS); interaction type of bigram and direction

failed to reach signicance (US: ln = ll; DS: ln = ll).

Stroke duration seems to be the most sensitive kinematic

measure of motor anticipation according to previous research

(Orliaguet and Boë, 1990; Bidet-Ildei et al., 2011; Kandel and

Perret, 2015). ere are several brain structures involved in

handwriting that are disrupted in schizophrenia, but it is the

basal ganglia that have been mostly related to the deautomatization

of writing. is dysfunction of the basal ganglia can result in

a general impairment in motor planning and coordination. is

impairment, associated to a delay in corrective movements, could

cause a segmentation of sequential movements and disrupt motor

anticipation in handwriting (Pantelis et al., 1992; Lange et al.,

2006; Smiley-Oyen et al., 2007; Bidet-Ildei et al., 2011).

e trajectory results also reect this kind of impairment in

motor anticipation in the patients. e control group produced

FIGURE5 | Spearman coefcients for CPZ, educational level, illness years, PURDUE, SAS, PANSS, and the handwriting variables.

Crespo Cobo et al. Handwriting in Schizophrenia

Frontiers in Psychology | www.frontiersin.org 9 April 2022 | Volume 13 | Article 807935

longer trajectories for the DS of the l of ln than ll and no

dierences in US, while the group of patients failed to show this

eect. ese results can be interpreted as another sign of motor

anticipation in healthy people compared to patients with

schizophrenia. Although previous studies have not found this

trajectory modulation, our data clearly support that anticipation

can also be reected in the trajectory of the stroke. We also

found that the strokes in the schizophrenia group were in general

longer than the strokes in the control group. is result is consistent

with previous studies that report the presence of macrography

in schizophrenia (Gallucci etal., 1997; Caligiuri etal., 2015; Kömür

et al., 2015). Of particular interest are those studies that relate

a decrease in the size of handwriting with dopamine D2 receptor

occupancy aer risperidone treatment (Kuenstler et al., 1999;

Regenthal etal., 2005). Future research could deepen these results.

Finally, the uency measure (i.e., the number of velocity peaks)

was not sensitive to motor anticipation, although the patients

exhibited more dysuency than controls. is is in line with

previous studies revealing more dysuent movements in

handwriting in patients with schizophrenia and related disorders

(Lohr and Caligiuri, 2006; Caligiuri et al., 2015; Crespo et al.,

2019). In general, dysuent movements occur when the muscles

that coordinate the movements receive dysregulated signals from

the basal ganglia (Caligiuri et al., 2009, 2010). Whether

operationalized as absolute velocity peaks or acceleration changes

over time, dysuency in handwriting is always present in

schizophrenia. e fact that in our study the schizophrenia group

presented greater trajectory length and more dysuency is a sign

of a general motor impairment in this disorder.

It is also noteworthy that we did not observe any relation

between psychotic symptoms, pharmacological treatment, and

demographic variables and kinematic measurements. We can

therefore discard that the groups dierences in motor anticipation

could be due to psychomotor slowing related to the evolution

of the disorder, It is also noteworthy that wedid not observe

any relation between psychotic symptoms, pharmacological

treatment, and demographic variables and kinematic

measurements. We can therefore discard that the groups

dierences in motor anticipation could bedue to psychomotor

slowing related to the evolution of the disorder, but not to

the pharmacological treatment or another demographic variables.

However, we think that the absence of motor anticipation in

patients is not caused by pharmacological treatment. First,

drug induced parkinsonism consists of a number of motor

symptoms, such as rigidity, bradykinesia, and tremor, but, to

our knowledge, decits in motor anticipation have never been

considered as a symptoms of parkinsonism. Parkinsonism

could bereected in some characteristics of patients handwriting,

for example, velocity, trajectory or uency, but it would

be reected in a group eect, not in an interaction between

these characteristics and bigram or direction. at is,

parkinsonism could bereected in patients handwriting being

slower than controls handwriting, but, in our opinion, it has

no sense that a dierence between patients and controls only

in upstrokes but not downstrokes would reect parkinsonism.

us, we interpreted that, in our study, results in uency

reect parkinsonism (idiopathic or drug-induced), but results

in duration and trajectory (where we found and interaction

between group, direction, and bigram) reect a specic decit

in motor anticipation.

In summary, patients with schizophrenia fail to show the

typical motor anticipation eect in handwriting, evidenced by

a modulation of duration of the rst letter of the bigram as

a function of spatial constraints of the second letter. In a

broader sense, this research constitutes further evidence in

favor of the analysis of handwriting as a quantitative, objective,

and reliable tool to detect motor alterations in schizophrenia.

Traditionally, the assessment of motor alterations has been

carried out by means of observation scales, such as the Simpson-

Angus Scale (SAS) and the Abnormal Involuntary Movements

Scale (AIMS). However, some studies have highlighted the

insucient predictive value of these scales, a low specicity

(Blanchet et al., 2012) and an acceptable reliability only if the

evaluation is performed by trained evaluators (Lane etal., 1985;

Tonelli et al., 2003). e analysis of handwriting on digitizing

tablets allows us to extract a number of handwriting measures

that can reveal dierent cognitive and motor processes disrupted

by the disorder.

DATA AVAILABILITY STATEMENT

e datasets analysed during the current study are available

from the corresponding author on reasonable request.

ETHICS STATEMENT

e studies involving human participants were reviewed and

approved by Comité de ética de la investigación de Jaén, Junta

de Andalucía. e patients/participants provided their written

informed consent to participate in this study.

AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual

contribution to the work, and approved it for publication.

FUNDING

is research was funded by Junta de Andalucía (Biomedical

and Heath Science research project PI-0410-2014, PI-0386-2016

and AP-0033-2020-C1-F2) and PID2019-105145RB-I00/Agencia

Estatal de Investigación AEI/10.13039/501100011033. e funders

had no role in study design, data collection, and analysis,

decision to publish, or preparation of the manuscript.

ACKNOWLEDGMENTS

is paper would not have been possible without the exceptional

support of Diego Armando Acevedo, doctoral student of

University of Jaén, and Gipsa Laboratory of Grenoble, where

part of this investigation has been made.

Crespo Cobo et al. Handwriting in Schizophrenia

Frontiers in Psychology | www.frontiersin.org 10 April 2022 | Volume 13 | Article 807935

REFERENCES

Andreasen, N. C., and Olsen, S. (1982). Negative v positive schizophrenia.

Denition and validation. Arch. Gen. Psychiatry 39, 789–794. doi: 10.1001/

archpsyc.1982.04290070025006

Ayehu, M., Shibre, T., Milkias, B., and Fekadu, A. (2014). Movement disorders

in neuroleptic-naïve patients with schizophrenia spectrum disorders. BMC

Psychiatry 14:280. doi: 10.1186/s12888-014-0280-1

Bates, D., Mächler, M., Bolker, B. M., and Walker, S. C. (2015). Fitting linear

mixed-eects models using lme4. J. Stat. Sow. 67, 1–48. doi: 10.18637/jss.

v067.i01

Bertram, R., Tønnessen, F. E., Strömqvist, S., Hyönä, J., and Niemi, P. (2015).

Cascaded processing in written compound word production. Front. Hum.

Neurosci. 9, 1–10. doi: 10.3389/fnhum.2015.00207

Bidet-Ildei, C., Pollak, P., Kandel, S., Fraix, V., and Orliaguet, J. P. (2011).

Handwriting in patients with Parkinson disease: eect of l-dopa and stimulation

of the sub-thalamic nucleus on motor anticipation. Hum. Mov. Sci. 30,

783–791. doi: 10.1016/j.humov.2010.08.008

Blanchard, J. J., Kring, A. M., Horan, W. P., and Gur, R. (2011). Toward the

next generation of negative symptom assessments: the collaboration to advance

negative symptom assessment in schizophrenia. Schizophr. Bull. 37, 291–299.

doi: 10.1093/schbul/sbq104

Blanchet, P. J., Normandeau, L., and Rompré, P. H. (2012). Comparing three

screening tools for drug-induced parkinsonism in patients with advanced

schizophrenia: a pilot study. Schizophr. Res. 137, 230–233. doi: 10.1016/j.

schres.2012.01.013

Boë, L. J., Orliaguet, J. P., and Belhaj, R. (1991). “Eet de contexte inter-lettre

sur le déroulement temporel des mouvements d’écriture: Similarités avec la

parole [Inter-letter context eect on the time course of written movements:

Similarities with speech], Proceedings of the XIIth International Congress of

Phonetic Sciences. Aix-En-Provence, France, Vol. 5, pp. 22–25.

Bombin, I., Arango, C., and Buchanan, R. W. (2005). Signicance and meaning

of neurological signs in schizophrenia: two decades later. Schizophr. Bull.

31, 962–977. doi: 10.1093/schbul/sbi028

Bonin, P., Chalard, M., Meot, A., and Fayol, M. (2001). Age-ofacquisition and

word frequency in the lexical decision task: Further evidence form the

French language. Current Psychology of Cognition 20, 401–443.

Caligiuri, M. P., Teulings, H.-L., Dean, C. E., and Lohr, J. B. (2015). A

quantitative measure of handwriting dysfluency for assessing tardive

dyskinesia. J. Clin. Psychopharmacol. 35, 168–174. doi: 10.1097/JCP.

0000000000000277

Caligiuri, M. P., Teulings, H.-L., Dean, C. E., Niculescu, A. B., and Lohr, J.

(2009). Handwriting movement analyses for monitoring drug-induced motor

side eects in schizophrenia patients treated with risperidone. Hum. Mov.

Sci. 28, 633–642. doi: 10.1016/j.humov.2009.07.007

Caligiuri, M. P., Teulings, H.-L., Dean, C. E., Niculescu, A. B., and Lohr, J. B.

(2010). Handwriting movement kinematics for quantifying extrapyramidal

side eects in patients treated with atypical antipsychotics. Psychiatry Res.

177, 77–83. doi: 10.1016/j.psychres.2009.07.005

Chengappa, K. N., Shelton, M. D., Baker, R. W., Schooler, N. R., Baird, J.,

and Delaney, J. (1994). e prevalence of akathisia in patients receiving

stable doses of clozapine. J. Clin. Psychiatry 55, 142–145.

Crespo, Y., Ibañez, A., Soriano, M. F., Iglesias, S., and Aznarte, J. I. (2019).

Handwriting movements for assessment of motor symptoms in schizophrenia

spectrum disorders and bipolar disorder. PLoS One 14:e0213657. doi: 10.1371/

journal.pone.0213657

Damian, M. F., and Stadthagen-Gonzalez, H. (2009). Advance planning of form

properties in the written production of single and multiple words. Lang.

Cognit. Processes 24, 555–579. doi: 10.1080/01690960802346500

Dean, D. J., Teulings, H.-L., Caligiuri, M., and Mittal, V. A. (2013). Handwriting

analysis indicates spontaneous dyskinesias in neuroleptic naïve adolescents

at high risk for psychosis. J. Visualized Exp. 81:e50852. doi: 10.3791/50852

Dean, D. J., Bernard, J. A., Orr, J. M., Pelletier-Baldelli, A., Gupta, T., Carol, E. E.,

et al. (2014). Cerebellar Morphology and Procedural Learning Impairment

in Neuroleptic-Naive Youth at Ultrahigh Risk of Psychosis. Clinical Psychological

Science, 2, 152–164. doi: 10.1177/2167702613500039

Ellis, (2014). “Modelling the writing process,” in Perspectives on Cognitive

Neuropsychology. eds. G. Denes, C. Semenza and P. Bisiacchi.

Fett, A. K. J., Viechtbauer, W., Domingue, M.-d.-G., Penn, D. L., van Os, J.,

and Krabbendam, L. (2011). e relationship between neurocognition and

social cognition with functional outcomes in schizophrenia: a meta-analysis.

Neurosci. Biobehav. Rev. 35, 573–588. doi: 10.1016/j.neubiorev.2010.07.001

Finney, G. R. (2015). Perceptual-motor dysfunction. Continuum 21, 678–689.

doi: 10.1212/01.CON.0000466660.82284.69

Gallucci, R. M., Phillips, J. G., Bradshaw, J. L., Vaddadi, K. S., and Pantelis, C.

(1997). Kinematic analysis of handwriting movements in schizophrenic

patients. Biol. Psychiatry 41, 830–833. doi: 10.1016/S0006-3223(96)00544-6

Gawda, B. (2016). Dysuent handwriting in schizophrenic outpatients. Perceptual

Motor Skills 122, 560–577. doi: 10.1177/0031512516637019

Gumedze, F. N., and Dunne, T. T. (2011). Parameter estimation and inference

in the linear mixed model. Linear Algebra Appl. 435, 1920–1944. doi: 10.1016/j.

laa.2011.04.015

Jahn, T., Hubmann, W., Karr, M., Mohr, F., Schlenker, R., Heidenreich, T.,

et al. (2002). Motoric neurological so signs and psychopathological symptoms

in schizophrenic psychoses. Psychiatry Res. 142, 191–199. doi: 10.1016/j.

psychres.2002.12.003

Kandel, S., Orliaguet, J.-P., and Viviani, P. (2000). Perceptual anticipation in

handwriting: the role of implicit motor competence. Percept. Psychophys.

62, 706–716. doi: 10.3758/BF03206917

Kandel, S., Peereman, R., Grosjacques, G., and Fayol, M. (2011). For a

psycholinguistic model of handwriting production: testing the syllable-bigram

controversy. J. Exp. Psychol. Hum. Percept. Perform. 37, 1310–1322. doi:

10.1037/a0023094

Kandel, S., and Perret, C. (2015). How do movements to produce letters become

automatic during writing acquisition? Investigating the development of motor

anticipation. Int. J. Behav. Dev. 39, 113–120. doi: 10.1177/0165025414557532

Kandel, S., Peereman, R., Ghimenton, A., and Perret, C. (2017). Letter coding

aects movement production in word writing: An English-Italian cross-

linguistic study. Reading & Writing: An Interdisciplinary Journal. 34, 219–251.

doi: 10.1007/s11145-017-9756-y

Kay, S. R., Fiszbein, A., and Opler, L. A. (1987). e positive and negative

syndrome scale (PANSS) for schizophrenia. Schizophr. Bull. 13, 261–276.

doi: 10.1093/schbul/13.2.261

Kömür, I., Gürler, A. S., Başpinar, B., Şahin, E., Kantarci, M. N., Emül, M.,

et al. (2015). Dierences in handwritings of schizophrenia patients and

examination of the change aer treatment. J. Forensic Sci. 60, 1613–1619.

doi: 10.1111/1556-4029.12858

Kuenstler, U., Juhnhold, U., Knapp, W. H., and Gertz, H. J. (1999). Positive

correlation between reduction of handwriting area and D2 dopamine receptor

occupancy during treatment with neuroleptic drugs. Psychiatry Res. 90,

31–39. doi: 10.1016/S0925-4927(98)00054-7

Lane, R. D., Glazer, W. M., Hansen, T. E., Berman, W. H., and Kramer, S. I.

(1985). Assessment of tardive dyskinesia using the abnormal involuntary

movement scale. J. Nerv. Ment. Dis. 173, 353–357. doi: 10.1097/00005053-

198506000-00005

Lange, K. W., Mecklinger, L., Walitza, S., Becker, G., Gerlach, M., Naumann, M.,

et al. (2006). Brain dopamine and kinematics of graphomotor functions.

Hum. Mov. Sci. 25, 492–509. doi: 10.1016/j.humov.2006.05.006

Lohr, J. B., and Caligiuri, M. P. (2006). Abnormalities in motor physiology in

bipolar disorder. J. Neuropsychiatr. 18, 342–349. doi: 10.1176/appi.

neuropsych.18.3.342

Lozano, L. M., and Acosta, R. (2009). Alteraciones cognitivas en la esquizofrenia.

Revista Med. 17-1, 87–94.

Luke, S. G. (2017). Evaluating signicance in linear mixed-eects models in

R. Behav. Res. Methods 49, 1494–1502. doi: 10.3758/s13428-016-0809-y

Miceli, G., and Capasso, R. (2006). Spelling and dysgraphia. Cognitive

Neuropsychology 23, 110–134. doi: 10.1080/02643290500202730

Oña, A., Martínez, M., Moreno, F., and Ruiz, L. (1999). CONTROL Y APRENDIZAJE

MOTOR|MANUEL MARTINEZ MARIN|Casa del Libro. Available at: https://

www.casadellibro.com/libro-control-y-aprendizajemotor/9788477386872/668115

(Accessed February 5, 2021).

Orliaguet, J.-P., and Boë, L.-J. (1990). “Régulation temporelle des mouvements

d’écriture en fRégulation temporelle des mouvements d’écriture en fonction

des contraintes spatiales [Temporal regulation of handwriting movements

as a function of spatial constraints],” in eds. V. Nougier and J. P. Blanchi

(Grenoble, France: Collection Grenoble Sciences), 163–177.

Crespo Cobo et al. Handwriting in Schizophrenia

Frontiers in Psychology | www.frontiersin.org 11 April 2022 | Volume 13 | Article 807935

Pantelis, C., Barnes, T. R. E., and Nelson, H. E. (1992). Is the concept of

frontal-subcortical dementia relevant to schizophrenia? Br. J. Psychiatry 160,

442–460. doi: 10.1192/bjp.160.4.442

Pappa, S., and Dazzan, P. (2009). Spontaneous movement disorders in antipsychotic-

naive patients with rst-episode psychoses: a systematic review. Psychol.

Med. 39, 1065–1076. doi: 10.1017/S0033291708004716

Peralta, V., Campos, M. S., de Jalon, E. G., and Cuesta, M. J. (2010). DSM-IV

catatonia signs and criteria in rst-episode, drug-naive, psychotic patients:

psychometric validity and response to antipsychotic medication. Schizophr.

Res. 118, 168–175. doi: 10.1016/j.schres.2009.12.023

Peralta, V., and Cuesta, M. J. (1994). Validation of positive and negative symptom

scale (PANSS) in a sample of Spanish schizophrenic patients. Actas Luso-

Espanolas de Neurologia, Psiquiatria y Ciencias Anes 22, 171–177.

R Core Team (2020). European Environment Agency. (n.d.). Available at: https://

www.eea.europa.eu/data-and-maps/indicators/oxygen-consuming-substances-

in-rivers/r-development-core-team-2006 (Accessed May 18, 2021).

Regenthal, R., Kunstler, U., Hesse, S., Sabri, O., and Preiss, R. (2005). D2

dopamine receptor occupancy, risperidone plasma level and extrapyramidal

motor symptoms in previously drug-free schizophrenic patients. Int. J. Clin.

Pharmacol. er. 43, 370–378. doi: 10.5414/CPP43370

Rogowska, J., Gruber, S., and Yurgelun-Todd, D. (2003). Functional magnetic

resonance imaging in schizophrenia: cortical response to motor stimulation.

Psychiatry Res. 130, 227–243. doi: 10.1016/j.pscychresns.2003.12.004

Russell Lenth (2020). emmeans: Estimated Marginal Means, aka Least-Squares

Means. R package version 1.5.1. Available at: https://CRAN.R-project.org/

package=emmeans

Simpson, G. M., and Angus, J. W. (1970). A rating scale for extrapyramidal

side eects. Acta Psychiatr. Scand. Suppl. 212, 11–19. doi: 10.1111/j.1600-0447.

1970.tb02066.x

Simpson, G. M., and Lindenmayer, J. P. (1997). Extrapyramidal symptoms in

patients treated with risperidone. J. Clin. Psychopharmacol. 17, 194–201.

doi: 10.1097/00004714-199706000-00010

Smiley-Oyen, A. L., Lowry, K. A., and Kerr, J. P. (2007). Planning and control

of sequential rapid aiming in adults with Parkinson’s disease. J. Mot. Behav.

39, 103–114. doi: 10.3200/JMBR.39.2.103-114

Tin, J. (1968). Purdue Pegboard examiner manual. Available at: https://www.

worldcat.org/title/purdue-pegboard-examiner-manual/oclc/642586232

(Accessed May 18, 2021).

Tin, J., and Asher, E. J. (1948). e Purdue pegboard: norms and studies

of reliability and validity. J. Appl. Psychol. 32, 234–247. doi: 10.1037/h0061266

Tonelli, H., Tonelli, D., Poiani, G. R., Vital, M., and Andreatini, R. (2003). Reliability

and clinical utility of a Portuguese version of the Abnormal Involuntary Movements

Scale (AIMS) for tardive dyskinesia in Brazilian patients. Braz. J. Med. Biol.

Res. 36, 511–514. doi: 10.1590/S0100-879X2003000400014

Van Galen, G. P. (1991). Handwriting: issues for a psychomotor theory. Hum.

Mov. Sci. 10, 165–191. doi: 10.1016/0167-9457(91)90003-G

Conict of Interest: e authors declare that the research was conducted in

the absence of any commercial or nancial relationships that could beconstrued

as a potential conict of interest.

Publisher’s Note: All claims expressed in this article are solely those of the

authors and do not necessarily represent those of their aliated organizations,

or those of the publisher, the editors and the reviewers. Any product that may

be evaluated in this article, or claim that may be made by its manufacturer, is

not guaranteed or endorsed by the publisher.

open-access article distributed under the terms of the Creative Commons Attribution

License (CC BY). e use, distribution or reproduction in other forums is permitted,

provided the original author(s) and the copyright owner(s) are credited and that

the original publication in this journal is cited, in accordance with accepted academic

practice. No use, distribution or reproduction is permitted which does not comply

with these terms.

ORIGINAL RESEARCH

published: 18 May 2022

doi: 10.3389/fpubh.2022.890960

Frontiers in Public Health | www.frontiersin.org 1May 2022 | Volume 10 | Article 890960

Edited by:

Carmen María Galvez-Sánchez,

University of Jaén, Spain

Reviewed by:

M. Rudi Irwansyah,

Ganesha University of

Education, Indonesia

Alexander Kuroyedov,

Pirogov Russian National Research

Medical University, Russia

Nileswar Das,

All India Institute of Medical

Sciences, India

*Correspondence:

Tianjun Liu

liutj2911@126.com

Specialty section:

This article was submitted to

Public Mental Health,

a section of the journal

Frontiers in Public Health

Received: 07 March 2022

Accepted: 19 April 2022

Published: 18 May 2022

Citation:

Tao Y, Chen Y, Zhou W, Lai L and Liu T

(2022) The Effectiveness of the

Moving to Emptiness Technique on

Clients Who Need Help During the

COVID-19 Pandemic: A Real-World

Study.

Front. Public Health 10:890960.

doi: 10.3389/fpubh.2022.890960

The Effectiveness of the Moving to

Emptiness Technique on Clients Who

Need Help During the COVID-19

Pandemic: A Real-World Study

Yanqiang Tao 1, Yi Chen 2, Wen Zhou 3, Lihui Lai 4and Tianjun Liu 5

1Beijing Key Laboratory of Applied Experimental Psychology, School of Psychology, Beijing Normal University, Beijing, China,

2School of Psychology, Nanjing Normal University, Nanjing, China, 3Yikong Skill Research Institute, Nanjing, China, 4School

of Continuing Education, Guangzhou University, Guangzhou, China, 5School of Acupuncture-Moxibustion and Tuina, Beijing

University of Chinese Medicine, Beijing, China

With Western therapeutic techniques prevailing in Chinese therapies, some techniques

that include Chinese traditional cultural features are required since some cultural factors

are not considered in the Western method. Our study introduced a new technique, the

moving to emptiness technique (MET), which combines Western structural progress

and core factors of Chinese culture. Seventeen therapists treated 107 clients with

the MET. Clients reported their target symptoms initially, and therapists helped them

transfer invisible symptoms to perceivable stuff and remove their jarring stuff using the

psychological emptiness area. At the end of the consultations, we found that MET

could eliminate symptoms immediately. By grouping target symptoms according to their

frequency, the results showed that clients in the high-frequency symptom group had

higher rehabilitation rates than those in the low-frequency symptom group. Additionally,

the results of the bereavement group were better than those of the non-bereavement

group, indicating that the MET can signiﬁcantly alleviate clients’ target symptoms. In

future studies, the replication and stability of the MET can be assessed by integrating

questionnaires, experimental designs, and neurological equipment.

Keywords: moving to emptiness, COVID-19, psychological consultation, clients, Chinese traditional cultural

“There is no Bodhi tree, mirror, or stand. Originally, there was nothing around us. Therefore, no dust

will fall.” – Hui Neng, the founder of Zen Buddhism.

INTRODUCTION

There are many psychosomatic symptoms in the community as a result of the COVID-19

pandemic, such as panic, tension, melancholy, and helplessness. Therefore, China has paid

increasing attention to providing psychological therapy for those with mental health problems.

However, most therapists in China still use Western therapies, which leads to some cultural

maladjustment. Because of this, some experts in China are now researching the application

of Chinese traditional philosophy and culture in Western structural therapies. The moving to

emptiness technique (MET) has become a choice for therapists in this context.

Tao et al. Moving to Emptiness Technique Clients

THE THEORETICAL FOUNDATION OF THE

MET

In one branch of Chinese philosophy, nothing exists around

us. Anything that disturbs us comes from our inner world.

Therefore, if we put any of our problems into a more extensive

context, the so-called emptiness state, our mental troubles will

no longer exist. Clinical psychology is regarded as a valuable ﬁeld

for all human beings in China, even if the research method is

still new. Moreover, China has a long history of psychological

thinking, which can be traced back to various traditional

philosophical and medical works (1,2). Confucianism, Taoism,

and Buddhism are three philosophical branches that impact

Chinese civilization on a large scale. As mentioned earlier,

poems are common in modern China, in which principles

lay a solid foundation for Chinese people’s faith and actions.

Additionally, these philosophical ideas have inﬂuenced the

Chinese medical system since ancient times, especially in treating

mental disorders and illnesses and cultivating a healthy life. The

MET is one therapeutic skill that applies the essence of traditional

Chinese philosophical thinking and cultivates medical systems in

clinical therapies (3).

Structural therapeutic techniques, such as cognitive behavioral

therapy, [CBT, (4)] and non-structural therapeutic techniques,

such as psychoanalysis, are two main methods that are

popular among therapists. CBT is a type of therapy in which

therapists focus on how clients’ dysfunctional beliefs aﬀect

their current behaviors and functions (5). CBT helps clients

explore, challenge, and modify their dysfunctional beliefs,

which is called cognitive restructuring, and can transfer the

distorted way they interpret reality into a more adaptive

direction. CBT oﬀers an operational structure for therapists

to lead clients in identifying their beliefs and core values

and then better revise them by considering new possibilities

(6).

Therapists who apply CBT assign homework outside of

therapeutic sessions for clients to experience the value of the

proposed changes that were developed through the collaboration

between therapists and clients in therapeutic sessions. Instead

of revising and restructuring the cognitive system or locus

of attention, the MET aims to eliminate negative feelings

represented by target symptoms. Taking the structure of CBT

into account, the MET is a psychosomatic treatment that includes

the academic concept of Chinese medicine in which the priority

in treating a mental disorder is to heal the mind. In summary,

the MET integrates traditional Chinese culture while preserving

somatic relaxation and the operational process of CBT.

In 2019 and 2021, the MET operational manual was published

in Chinese (7) and German, respectively. The MET goes

beyond structural therapy abilities and includes attempts to

discover clients’ target symptoms to reduce the symptoms in

a broader psychological context. Before therapeutic sessions,

therapists help clients relax and lead them in being mindful

of their emotions. Following this, the therapists guide the

clients to identify and express their target symptoms and

place them in appropriate “containers”. With guidance, the

clients can move these containers back and forth before their

psychological symptoms occur, and then put them even further

away until they disappear in a psychological emptiness area.

Using ten operational steps (see Section Measures Consultation

Step), the MET, a psychosomatic therapy, can reduce or

eliminate symptoms.

Compared to Western psychotherapies that focus on

“existence,” Chinese therapeutic skills involve both “emptiness”

and “existence.” “Emptiness” is a status in which only

consciousness exists. At the same time, the existence

mentioned here refers to transforming one mood into

another or surmounting one without removing anything;

for example, transforming a negative mood into a positive

mood or overcoming sickness through wellness. In comparison,

“emptiness” here relates to psychological emptiness. When a

client in a bad mood enters the consultant room, a therapist

who uses CBT will guide him or her to identify and restructure

the maladaptive cognitive system. Therefore, an unchanged

bad mood is covered by a good mood. Once a therapist applies

the MET in counseling, the client will be guided to a state of

psychological emptiness where the bad mood will disappear. If

we describe the so-called state of emptiness in English, it is a

neutral, non-positive, and non-negative state of existence instead

of a state of action.

The psychological emptiness area can be regarded as a purely

mental and emotional condition without any troubles. What

must be mentioned here is that clients use their psychological

emptiness area to solve their problems, which does not depend

on their defensive mechanisms, such as denial, repression,

projection, avoidance, transference, replacement, or sublimation.

Their troubles are directly absorbed, accepted, and processed

rather than being rejected or disguised.

The psychological emptiness area can provide cures because

it is an inﬁnite psychological space without any trouble to which

therapists can guide clients. Therefore, the problems are put in

psychological emptiness, a broader background, which vanish

automatically using this skill. For example, if one spoon of

salt dissolves in a cup of water, the water will be brackish.

Nevertheless, the taste will not change much when the salt is

put into one water tank. Moreover, if it is poured into one lake,

nothing will change. Theoretically, once a mental disorder can

be put into a person’s massive psychological emptiness area, it

can disappear automatically. In traditional Chinese medicine,

the psychological emptiness area is widely proposed for solving

psychological troubles (3). The MET provides clear, fast, and

practical guidance to enter the psychological emptiness area.

It is not only an innovative and non-antagonistic idea but

also a particular way for problem solving. Moreover, the entire

methodology includes Chinese traditional ideology and wisdom

by summarizing the core of the ancient Chinese medical system.

GOALS OF TREATMENT WITH THE MET

As mentioned above, the main diﬀerence between the MET

and CBT is that the MET directly targets symptom reduction,

Frontiers in Public Health | www.frontiersin.org 2May 2022 | Volume 10 | Article 890960

Tao et al. Moving to Emptiness Technique Clients

while CBT focuses more on restructuring the cognitive or

emotional systems.

Thus, to test the eﬀectiveness of the MET in practical

counseling applications, we sampled 107 subjects to assess

whether it could work eﬀectively for the entire group. The index

of inﬂuence was used as the criterion. We ﬁrst analyzed the

immediate and long-term counseling eﬀects of the MET using a

paired-samples t-test. To specify the eﬀectiveness of the MET for

diﬀerent symptoms, we classiﬁed all target symptoms reported

into two categories based on a high and low frequency of the top

three body parts with the highest presence of symptoms. Since

we conducted our research during the COVID-19 pandemic,

with clients losing beloved family members, we obtained the

clients’ familial information before treatment. Based on their

backgrounds, we divided them into two groups, the bereavement

group or the non-bereavement group, to determine whether the

MET has diﬀerent eﬀects.

METHODS

Participants and Consultations

Participants

We started our study during the COVID-19 pandemic and

recruited 107 participants and a total of 17 psychotherapists

in mainland China. Notably, some detailed procedures for

participants recruitment need to be introduced. Firstly, the

present study was an open program to those who require trauma

healing treatments due to the COVID-19 pandemic on mainland

China and oﬀers them psychological counseling services without

requiring a clinical diagnosis at the time of enrollment. However,

certain criteria must still be met: experiencing a distressed state

of mind during the epidemic, including physical pain such as

insomnia, headaches, and chest tightness, as well as negative

emotions such as fear, anxiety, guilt, self-blame, irritability,

loneliness, and sadness, as well as a desire to improve the

psychosomatic condition. Clients in a psychoactive phase or who

were unable to complete the three relaxation steps of MET and

required crisis assistance were excluded.

We publicized recruitment advertisements on one online

platform, and all participants joined voluntarily. The ethics

committee of the corresponding author’s university approved this

research (Reference Number: H20006, ChiCTR2000034164).

Consultation Steps

The Trio (Body, Mind, and Breath) Relaxation Exercise

Regulate the body: Shake and relax the body, sit in the ﬁrst 1/3 of

the chair in a comfortable position with the neck and shoulders

relaxed. Straighten the waist and back and then rest both hands

on the thighs and close the eyes.

Regulate the breath: Breathe deeply, slowly, and naturally.

There is no need to ﬁll or empty the lungs to avoid blood

pressure ﬂuctuations.

Regulate the mind: Focus only on exhaling without special

attention while inhaling. Empty the mind while exhaling. Practice

these steps for ∼3 min with the eyes closed and open the eyes

when the mind is clear. The relaxation of the body, breath, and

mind is the pre-consultation phase and a prerequisite of therapy

with the MET. If a client is unable to relax, he or she cannot

proceed to the subsequent steps.

Select a Symptom That Causes Trouble for the Client

as the Target Symptom

It could be a negative emotion such as fear, anxiety, anger, or

a negative physical sensation such as tightness of the chest,

shortness of breath, or pain of the body. In each session, only one

symptom is treated. If there is more than one physical or mental

problem, clients will be asked to choose the one that is most

problematic or urgent for this session by evaluating its inﬂuence

on a scale from 0 to 10. Usually, when people seek help, their

target symptoms have scores of 7 or above.

Visualize and Locate the Target Symptom

There are two ways to determine the symbolic object of the

target symptom. One way is to ask the client how they embody

their target symptom. For example, the therapist could ask: what

makes you feel bad at that body part? Asking such questions could

encompass the client’s physical sensations and feelings from the

target symptom. The other way is to locate the emotion in one

somatic part. If the client feels stressed, he or she may report

that a pile of cotton is blocking his or her chest to the therapist.

After a symbolic object and a certain somatic part are determined

and located, the client should elaborate and highlight various

dimensions of the object and embody the object. The therapist

can help by asking the client to describe the size, shape, weight,

sound, texture, and smell related to the symbolic object so it

becomes vivid.

Visualize a Symbolic Container

A symbolic container is the device that holds that object

mentioned in Step 3. It signiﬁes a client’s internal resources and

energy. Clients are encouraged to create a container with rich and

vivid perceptual features that are similar to the symbolic object.

Moving the Symbolic Object Into the Psychological

Emptiness Area

Guide the client to:

i) put the symbolic object into the container in the mind;

ii) moving the container farther and farther away

psychologically and ﬁnally into the psychological emptiness

area. First, move the container 3 meters away and back again.

Repeat this step 2 to 3 times. Then, move the container farther

away so that that it will look like a small dot, then move it

back. Repeat this process 10 times. Finally, move the container

far enough away so the client cannot see or feel it. Clients get

to the “emptiness area” where nothing exists in this stage.

iii) When the container with the symbolic object is moved to the

psychological emptiness area, clients can feel comfortable and

relaxed with the target symptom leaving. The client is asked to

experience the sensation of emptiness by staying there as long

as they can with their eyes closed.

Assessment of Changes After the Intervention

Ask the client to score the inﬂuence of their symptoms again after

the intervention. If the initial score is reduced by 50% or more,

Frontiers in Public Health | www.frontiersin.org 3May 2022 | Volume 10 | Article 890960

Tao et al. Moving to Emptiness Technique Clients

FIGURE 1 | Consort diagram for the present study.

the treatment is considered to be highly eﬀective; if the score is

reduced by 1/3 or more but by <50%, the treatment is considered

to be adequately eﬀective; and if the score is reduced by <1/3, the

treatment is considered to be ineﬀective.

Measures

Consulting Assessment

A visual analog scale (VAS) is a Likert psychometric scale that is

used to evaluate subjective characteristics or attitudes (8). These

scales have previously been used to diagnose various disorders in

market research and social science assessments.

First, we assessed the inﬂuence of the clients’ target symptoms

and scored it from 1 to 10. At the end of the MET treatment

session, the client re-evaluated the impacts of their symptoms

when the container was far away and invisible in their minds.

Follow-Up Feedback

The clients were asked how much the previous target symptoms

had inﬂuenced them in a follow-up survey after 1 week.

Procedure

In this study, a dedicated reservationist establishes a consultation

time based on the client’s registration information. When the

client enters the consultation room, the psychotherapists will

spend about 1–3 min gathering basic information and identifying

the client’s primary symptoms that need to be addressed and a

1–10 rating (i.e., the inﬂuence of the clients’ target symptoms).

The formal consultation session is then conducted, and after

the consultation, the 1–10 scale is administered once more. An

average single session of ∼50 min for each client.

The appointment maker will contact the visitor within about

a week to inquire about the consultation’s outcome and conduct

a 1–10 rating. Due to the principle of respecting client’s wishes,

data on the eﬀectiveness of counseling are unavailable for

some visitors, as shown in Figure 1. In total, 297 sessions

were collected. A total of 276 sessions were collected for the

ﬁnal process with outlier deletion (i.e., duplicate and incorrect

data rows).

Statistical Analysis

In the present study, all data were analyzed by R. We used the

package of compareGroups (9) to collect the essential information

for the participants and therapists. Then, the WordCloud package

(10) was used to visualize the target symptoms of the participants.

The afex package (11) was used to perform the mixed ANOVA.

RESULTS

Descriptive Demographic Analysis

As shown in Tbale 1, we recruited 107 participants, with the

majority being female (n=93; 86.9%). More than half of the

participants had undergraduate and junior college education

degrees (n=69; 64.5%). More than half of them were employed

(n=62; 57.9%) and were not students (n=93; 86.9%). More

than 60% of them were married (n=69; 64.5%). As reported

in the clients’ information, most of them did not have mental

health issues (n=94; 87.9%) or took medicine (n=90, 84.1%;

see Table 1).

A total of 17 psychotherapists worked for this study. Most of

them were female (n=15; 88.2%) and aged older than 40 years

(n=15; 88.24%). Most of them had an undergraduate education

level (n=16; 94.12%). More than half of them had more than 5

years of working experience (n=11; 64.71%). More than half of

them had worked in counseling between 1 and 3 years (n=11;

64.7%) and had supervision time below 50 h (n=11; 64.71%; see

Table 2).

Frontiers in Public Health | www.frontiersin.org 4May 2022 | Volume 10 | Article 890960

Tao et al. Moving to Emptiness Technique Clients

TABLE 1 | Summary descriptives table for clients (n=107).

Variable N(Proportion/SD)

Gender Male 14 (13.1%)

Female 93 (86.9%)

Age 40.0 (10.95)

Education Below high school 3 (2.80%)

High school and polytechnic school 3 (2.80%)

Undergraduate and junior college 69 (64.5%)

Master and doctor 30 (28.0%)

Others 2 (1.87%)

Occupation Worker 4 (3.74%)

Cadre 10 (9.35%)

Technician 16 (15.0%)

Teacher 18 (16.8%)

Profession 14 (13.1%)

Others 45 (42.1%)

Job status Student 11 (10.3%)

On the job 62 (57.9%)

Unemployed 13 (12.1%)

Retire 8 (7.48%)

Others 13 (12.1%)

Marriage Unmarried 31 (29.0%)

Married 69 (64.5%)

Divorce 4 (3.74%)

Separation 2 (1.87%)

Widowed 1 (0.93%)

Students Yes 14 (13.1%)

No 93 (86.9%)

Mental health Yes 13 (12.1%)

No 94 (87.9%)

Medicine Yes 17 (15.9%)

No 90 (84.1%)

Descriptive Analysis of the Number of MET

Consultations

A single individual conducted one initial interview and three

consultations in the standardized process. In particular, only

a few clients required several counseling sessions to achieve

the goal. Speciﬁcally, a total of 107 people were interviewed

and treated ﬁrst. Sixty-nine people had two sessions, and 51

people consulted three times. As shown in Figure 2, sessions

decreased continuously with fewer clients joining subsequent

counseling sessions. Three persons took nine sessions, which was

the maximum number of sessions for a single person.

Overall Intervention Effect of MET

Subsequently, this study screened 276 sessions of 107 people who

interviewed and consulted for the ﬁrst time. Only the sessions

with consultation participation were retained, and the interview

data were deleted if there was no score of the inﬂuence index,

which included 210 sessions. First, we tested the eﬀectiveness of

the MET intervention for participants using one-way repeated-

measures ANOVA (Time: Pretest vs. Posttest vs. Follow-up). The

TABLE 2 | Summary descriptives table for psychological consultant (n=17).

Variable N(proportion)

Gender Female 15 (88.2%)

Male 2 (11.8%)

Age 20–30 1 (5.88%)

30–40 1 (5.88%)

40–50 10 (58.8%)

50–60 4 (23.5%)

Above 60 1 (5.88%)

Education Below undergraduate 1 (5.88%)

Master 7 (41.2%)

Undergraduate 9 (52.9%)

Working time 1–3 years 3 (17.6%)

3–5 years 3 (17.6%)

5–10 years 6 (35.3%)

Above 10 years 5 (29.4%)

Treatment time 1–3 years 11 (64.7%)

Above 5 years 2 (11.8%)

<1 year 4 (23.5%)

Supervision hours <20 h 7 (41.2%)

20–50 h 4 (23.5%)

Above 50 h 6 (35.3%)

results indicated that the main eﬀect of time was signiﬁcant

[F(2,209) =651.7, p=0.000]. The posttest revealed that the

individual inﬂuence decreased signiﬁcantly after the counseling

intervention (Mean =2.162) compared with before consultation

(Mean =7.861, t=36.08, p=0.000). It should be noted that

over time, the inﬂuence after follow-up (Mean =2.858) was

signiﬁcantly stronger than that after the intervention (t=4.215,

p=0.000), but it was still signiﬁcantly lower than the inﬂuence

score before the consultation (t=26.07, p=0.000; see Figure 3).

Word Cloud Analysis to Categorize

Symptoms

Then, we deleted the missing target symptom locations, and 156

sessions were reserved for the word cloud analysis. First, we

coded the target symptom locations into two parts: the body

surface and internally. The results indicated that 122 symptoms

were located inside the body and 34 symptoms were located

outside the body (see Figure 4A). Considering the speciﬁc

location, we coded the target symptoms into 15 parts: head,

eyes, throat, neck, shoulders, back, bosom, heart, lungs, waist,

stomach, abdomen, upper limbs, lower limbs, and others. The

results indicated 30 symptoms for the head, 26 symptoms for the

bosom, 26 symptoms for the heart, 14 symptoms for the throat,

and 14 symptoms for the abdomen (see Figure 4B).

ANOVA Analysis for Target Symptoms

Here, it should be pointed out that this study grouped speciﬁc

locations of target symptoms. The top 3 symptoms (i.e., the

three parts of the body where the symptoms appeared the

Frontiers in Public Health | www.frontiersin.org 5May 2022 | Volume 10 | Article 890960

Tao et al. Moving to Emptiness Technique Clients

FIGURE 2 | Times of psychological consultation.

FIGURE 3 | The repeated measurement analysis of inﬂuence index.

***p< .001.

most as the high-frequency group) were deﬁned as the high-

frequency group, and the rest were deﬁned as the low-frequency

group to explore whether there were diﬀerences in the index of

inﬂuence between the two groups. Furthermore, to investigate

the diﬀerence in the inﬂuence index between the high-frequency

and low-frequency groups under three measurements, a mixed

ANOVA was conducted: 2 (Group: high-frequency vs. Low-

frequency) ×3 (Time: Pretest vs. Posttest vs. Follow up).

The results showed that the main eﬀect of the high-frequency

and low-frequency groups was insigniﬁcant (F=0.14, p=0.709,

η2=0.001). However, the main eﬀect of the time point was

signiﬁcant (F=461.14, p=0.000, η2=0.609). The interaction

between the high-frequency and low-frequency groups and the

time point was signiﬁcant (F=5.76, p=0.005, η2=0.019).

According to the comparison results, in the low-frequency group,

the inﬂuence after consultation (Mean =2.20, SE =0.204) was

signiﬁcantly lower than those before consultation (Mean =7.41,

SE =0.186, t= −19.841, p=0.000) and after follow-up

(Mean =3.34, SE =0.283, t= −12.440, p=0.000). The

inﬂuence after follow-up was signiﬁcantly higher than that after

consultation (t=4.403, p=0.002).

In the high-frequency group, the degree of inﬂuence after

the consultation (Mean =2.16, SE =0.194) was signiﬁcantly

lower than those before consultation (Mean =7.97, SE =0.177,

t= −23.303, p=0.000) and after follow-up (Mean =2.57,

SE =0.269, t=17.384, p=0.000). The degree of inﬂuence after

follow-up did not signiﬁcantly diﬀer from that after consultation

(t=1.687, p=0.213). The results are shown in Figure 5A.

ANOVA Analysis for Bereavement

At the same time, after deleting the missing data, 156 sessions

were ﬁnally obtained. Then, the participants were grouped

according to whether they lost relatives during the study time.

Furthermore, whether the degree of inﬂuence under bereavement

was diﬀerent under the three measurement time points was

investigated. That is, we conducted a mixed ANOVA of 2

(Bereavement vs. No Bereavement) X 3 (Pretest vs. Posttest

vs. Follow-up). The results showed that the main eﬀect of

bereavement was not signiﬁcant (F=0. 83, p=0.365), and

the main eﬀect of the time point was signiﬁcant (F=252.82,

p=0.000), but the interaction between bereavement and the

time point was insigniﬁcant (F=1.60, p=0.206). According

to the comparison results, in the bereavement group, the degree

of inﬂuence after consultation (Mean =2.39, SE =0.34) was

signiﬁcantly lower than those before consultation (Mean =7.39,

SE =0.31, t= −11.456, p=0.000) and follow-up (Mean =2.35,

SE =0.47, t= −9.056, p=0.000). However, there was

no signiﬁcant diﬀerence between the degree of inﬂuence after

follow-up and consultation (t=0.086, p=0.996).

In the non-bereaved group, the degree of inﬂuence after

consultation (Mean =2.13, SE =0.15) was signiﬁcantly lower

than those before consultation (Mean =7.77, SE =0.14,

t= −28.261, p=0.000) and after follow-up (Mean =3.01,

SE =0.22, t= −18.719, p=0.000). The degree of inﬂuence after

follow-up was signiﬁcantly lower than that after consultation

(t=4.446, p=0.000). The results are shown in Figure 5B.

DISCUSSION

In summary, the MET preserves somatic relaxation and the CBT

operational process while integrating traditional Chinese culture.

The MET is highly eﬀective during the COVID-19 pandemic for

removing negative emotions and terrible physical feelings. As a

safe, fast-acting therapeutic technique, although there is some

deterioration after a week, the eﬀects of the MET are mainly

maintained, which is essential for bereaved people. Hence, some

points are worthy of discussion here.

We recognize that most current counseling techniques in

China are derived from the West (12). This provides us with

numerous beneﬁts, both essential to those with psychological

needs and critical to developing the psychological discipline

in China. For diﬀerent types of clients, divergent therapeutic

techniques should be applied. Hence, many people in Eastern

Frontiers in Public Health | www.frontiersin.org 6May 2022 | Volume 10 | Article 890960

Tao et al. Moving to Emptiness Technique Clients

FIGURE 4 | Word cloud analysis of target symptoms. (A) indicated the frequency of approximate locations about target symptoms. (B) indicated the frequency of

speciﬁc locations about target symptoms.

FIGURE 5 | The results of the mixed ANOVA analysis. (A) indicated the results of different degrees of frequency. (B) showed that whether bereavement or not could

have a difference in inﬂuence.

cultures can be better served by the MET, which combines

the fundamental logic of Western counseling techniques with

Eastern philosophical principles.

In the present study, we found that the target symptoms

identiﬁed by the clients decreased signiﬁcantly after the

intervention. What should not be overlooked is the rebound in

the follow-up survey. This also highlights that our consultation

outcomes were delivered promptly. According to the follow-up

data, the inﬂuence of a client’s symptoms returns, although it

is still signiﬁcantly below the initial reported levels. It is clear

that counseling is pretty successful in improving an individual’s

current state and that more counseling should be considered

progressively over time to strengthen the treatment’s beneﬁts.

We separated the top three symptoms according to target

symptoms into a high-frequency group and the remainder into

a low-frequency group to verify the eﬃcacy of the MET from

multiple perspectives. Compared with those in the low-frequency

group, there was no signiﬁcant diﬀerence in client eﬀect scores

among the high-frequency group at post-intervention and

follow-up, which indicates that the eﬀectiveness is apparent

Frontiers in Public Health | www.frontiersin.org 7May 2022 | Volume 10 | Article 890960

Tao et al. Moving to Emptiness Technique Clients

when the MET is evaluated from a randomized perspective. The

structure of the MET is similar to that of CBT. Nevertheless,

the goals are diversiﬁed. CBT assists clients in identifying

illogical ideas and attempting to adjust their behavior patterns

by modifying their irrational or illusory perceptions. The idea

of CBT is the circulation among mood, behavior, and cognition.

Compared to identifying maladaptive beliefs (13), the MET oﬀers

therapists a fresh perspective; it regards people as a whole.

The therapeutic target is to remove those severe problems in

personalities. CBT is rational, while MET is emotional. In other

words, they have diﬀerent focuses and diﬀerent treatment goals.

The treatment goal of MET is to bring the client to a place where

there is no problem. Compared to CBT, MET has short and

quick characteristics. Based on the traditional Chinese concept

of emptiness, the MET leads clients to dissolve their internal

problems into inﬁnite emptiness by identifying and embodying

their symptoms.

Bereavement is described as the situation of having lost a

signiﬁcant loved one due to death (14). Everyone will experience

bereavement, which is a highly stressful event, during their

lifetime. Individuals seldom experience bereavement in early

childhood (∼3.4%) (15). Then, as people grow, their risks of

experiencing bereavement grow, with ∼45% of women and 15%

of men in elderly populations experiencing bereavement (16). In

the present study, participants were grouped according to who

had been bereaved and who had not been bereaved. There was no

signiﬁcant diﬀerence in the client eﬀect scores among the group

with bereavement experiences at post-intervention and follow-

up, which indicates that the eﬀectiveness is apparent when the

MET is re-evaluated from a randomized perspective.

Bereavement is linked to a higher risk of mortality for

a multitude of reasons, including suicide (17). Furthermore,

mental and physical illness is severe and persistent in a

small proportion of the population. Notably, depending on the

individual and their culture, recovery might take months or even

years. As a result, bereavement is both a preventative and clinical

concern. However, child grief therapies do not appear to produce

good results like other professional psychotherapy interventions

(18). As children’s early intimate relationships come to an abrupt

end, early memories fade with aging. Children’s bereavement

experiences can be reshaped by later social attention and the

formation of new personal attachments (19). However, due to the

general stability of the object-subject relationship in adulthood

(20), adults’ memories of painful bereavement experiences are

diﬃcult to erase in a short period. They can even remain with

them throughout their lives, inﬂuencing their daily emotional

(21,22) and life functions (23,24).

Some researchers have employed a variety of psychological

counseling therapies to aid persons experiencing grief who

have been bereaved (25–27). Studies investigating counseling

techniques refer to the dual process model and the meaning

reconstruction model. However, due to cultural diﬀerences, the

mainstream models in mainland China are currently cognitive

behavioral therapy (CBT) and other counseling techniques.

To treat and intervene Chinese people who have experienced

bereavement, a counseling technique with a high degree of

cultural-ecological validity is needed. The moving to the

emptiness technique (MET) may be a good choice.

Limitations

However, this study has potential limitations. To begin, we

presently have data on only 107 clients, although these clients

reveal great demographic heterogeneity. Given that a large

proportion of clients to this study received multiple counseling

sessions (see Figure 3), we could not integrate demographic

characteristics as covariates in ANOVA analysis. However, in

future studies, MET should explore examining the impact

of interventions on speciﬁc populations and controlling for

demographic heterogeneity, such as people with depression.

Second, the MET is a new counseling technique that stems

from ancient Chinese philosophical thinking and Chinese

medical theory, mixed with the logic of Western counseling

techniques. Compared with CBT and other techniques, it is

still in development. Third, although the therapeutic beneﬁts of

the MET were investigated in this study using a randomized

design, more analytical approaches, such as questionnaire

measurements, fMRI, fNIRS, and other similar methodologies,

should be used to assess the eﬃcacy of the MET. Furthermore,

the beneﬁts of the theoretical framework of the MET can be

used in future studies concerning more clinical patients with

speciﬁc medical issues, such as neurological headaches and

frozen shoulder. The eﬃcacy of the MET may be evaluated

by attempting to relieve persons’ somatic diseases from a

psychological counseling perspective.

DATA AVAILABILITY STATEMENT

The raw data supporting the conclusions of this article will be

made available by the authors, without undue reservation.

ETHICS STATEMENT

The studies involving human participants were reviewed and

approved by Beijing University of Chinese Medicine. The

patients/participants provided their written informed consent to

participate in this study.

AUTHOR CONTRIBUTIONS

TL and YC: study design and critical revision of the manuscript.

WZ and LL: data collection. YT and YC: analysis and

interpretation. YT: drafting of the manuscript. All authors

contributed to the article and approved the submitted version.

FUNDING

This study was supported by the Scientiﬁc Foundation

of Institute of Psychology, Chinese Academy of Sciences,

No. EOCX331008.

ACKNOWLEDGMENTS

The authors thank the counselors who participated in the

research, Dan Chen, an undergraduate student at the School

of Psychology, Nanjing Normal University, and Wenxin Hou

of the Faculty of Psychology, Beijing Normal University, for

their contributions.

Frontiers in Public Health | www.frontiersin.org 8May 2022 | Volume 10 | Article 890960

Tao et al. Moving to Emptiness Technique Clients

REFERENCES

1. Higgins LT, Zheng M. An introduction to Chinese psychology—

Its historical roots until the present day. J Psychol. (2002)

136:225–39. doi: 10.1080/00223980209604152

2. Jing Q. Development of psychology in China. Int J Psychol. (1994) 29:667–

75. doi: 10.1080/00207599408246557

3. Xia Y, Wu X, and Liu T. (2013). Analysis of Yikong Skills and Its Future

Research from the Perspective of Local Psychotherapy. Chinese Journal of

Clinical Psychology. 21:852–5.

4. Beck AT. Cognitive Therapy of Depression. New York, NY: Guilford

Press (1979).

5. Cuijpers P, Van Straten A, Andersson G, Van Oppen P. Psychotherapy for

depression in adults: a meta-analysis of comparative outcome studies. J

Consult Clin Psychol. (2008) 76:909–22. doi: 10.1037/a0013075

6. Cuijpers P, Berking M, Andersson G, Quigley L, Kleiboer A, Dobson KS. A

meta-analysis of cognitive-behavioural therapy for adult depression, alone

and in comparison with other treatments. Can J Psychiatry. (2013) 58:376–

85. doi: 10.1177/070674371305800702

7. Liu. (2019). The Moving to Emptiness Technique Manual: A localized mind-

body therapy technique. China Press of Traditional Chinese Medicine.

8. Crichton N. Visual analogue scale (VAS). J Clin Nurs. (2001) 10:697–

706. doi: 10.1046/j.1365-2702.2001.00525.x

9. Subirana I, Sanz H, Vila J. Building bivariate tables: the compareGroups

package for R. J Stat Softw. (2014) 57:1–16. doi: 10.18637/jss.v057.i12

10. Fellows I. wordcloud: Word Clouds. R package version 2.6 (2018).

11. Singmann H, Bolker B, Westfal J, Aust F, Ben-Shachar MS. afex: Analysis of

Factorial Experiments. R package version 1.0–1 (2021).

12. Higgins LT, Davey G, Gao X, Zheng R, Ni Z, Lang L. Counselling

in China: past, present and future. Psychol Dev Soc J. (2008) 20:99–

109. doi: 10.1177/097133360702000105

13. Hofmann SG, Asnaani A, Vonk IJ, Sawyer AT, Fang A. The eﬃcacy of

cognitive behavioral therapy: a review of meta-analyses. Cognit Ther Res.

(2012) 36:427–40. doi: 10.1007/s10608-012-9476-1

14. Stroebe MS, Hansson RO, Stroebe WE, Schut HE. Handbook of Bereavement

Research: Consequences, Coping, and Care. Washington, DC: American

Psychological Association (2001).

15. Christ GH, Siegel K, Christ AE. Adolescent grief: It never really

hit me... until it actually happened. JAMA. (2002) 288:1269–

78. doi: 10.1001/jama.288.10.1269

16. Roona MR, Streke AV, Marshall DG. Substances, adolescence (metaanalysis).

In: Gulotta TP, Bloom M, editors. Encyclopedia of Primary Prevention and

Health Promotion. New York, NY: Kluwer Academic/Plenum Publishers

(2003). p. 1065–79

17. Pitman A, Osborn D, King M, Erlangsen A. Eﬀects of suicide bereavement

on mental health and suicide risk. Lancet Psychiatry. (2014) 1:86–

94. doi: 10.1016/S2215-0366(14)70224-X

18. Currier JM, Holland JM, Neimeyer RA. The eﬀectiveness of

bereavement interventions with children: a meta-analytic review of

controlled outcome research. J Clin Child Adolesc Psychol. (2007)

36:253–9. doi: 10.1080/15374410701279669

19. Holland J. Understanding Children’s Experiences of Parental Bereavement.

London: Jessica Kingsley Publishers (2001).

20. Troll LE, Smith J. Attachment through the life span: Some

questions about dyadic bonds among adults. Hum Dev. (1976)

19:156–70. doi: 10.1159/000271526

21. Jacobson NC, Lord KA, Newman MG. Perceived emotional social support in

bereaved spouses mediates the relationship between anxiety and depression. J

Aﬀect Disord. (2017) 211:83–91. doi: 10.1016/j.jad.2017.01.011

22. Marshall B, Davies B. Bereavement in children and adults following the death

of a sibling. In: Neimeyer R, Harris D, WinokuerH, Thornton G, editors. Grief

and Bereavement in Contemporary Society. Routledge (2021). p. 107–116.

23. Parkes CM, Prigerson HG. Bereavement: Studies of Grief in Adult Life. London:

Routledge (2013).

24. Reitz AK, Weidmann R, Wünsche J, Buehler JL, Burriss R, Grob A.

In Good Times and in Bad: A Longitudinal Analysis of the Impact

of Bereavement on Self-Esteem and Life Satisfaction in Couples.

Eur J Pers. (2021) 08902070211054896. doi: 10.1177/089020702110

54896

25. Garti D, Bat Or M. Subjective experience of art therapists

in the treatment of bereaved clients. Art Therapy. (2019)

36:68–76. doi: 10.1080/07421656.2019.1609329

26. Golsworthy R, Coyle A. Practitioners’ accounts of religious and spiritual

dimensions in bereavement therapy. Couns Psychol Q. (2001) 14:183–

202. doi: 10.1080/09515070110037993

27. Larsen LH, Guldberg A, Kring V. A group therapy program

for parentally bereaved young adults with grief complications:

rationale, method and case examples. OMEGA J Death Dying. (2021)

0030222821997705. doi: 10.1177/0030222821997705

Conﬂict of Interest: The authors declare that the research was conducted in the

absence of any commercial or ﬁnancial relationships that could be construed as a

potential conﬂict of interest.

Publisher’s Note: All claims expressedin this article are solely those of the authors

and do not necessarily represent those of their aﬃliated organizations, or those of

the publisher, the editors and the reviewers. Any product that may be evaluated in

this article, or claim that may be made by its manufacturer, is not guaranteed or

endorsed by the publisher.

distributed under the terms of the Creative Commons Attribution License (CC BY).

The use, distribution or reproduction in other forums is permitted, provided the

original author(s) and the copyright owner(s) are credited and that the original

publication in this journal is cited, in accordance with accepted academic practice.

No use, distribution or reproduction is permitted which does not comply with these

terms.

Frontiers in Public Health | www.frontiersin.org 9May 2022 | Volume 10 | Article 890960

ORIGINAL RESEARCH

published: 06 June 2022

doi: 10.3389/fpubh.2022.919608

Frontiers in Public Health | www.frontiersin.org 1June 2022 | Volume 10 | Article 919608

Edited by:

Carmen María Galvez-Sánchez,

University of Jaén, Spain

Reviewed by:

Xiaohe Wang,

Hangzhou Normal University, China

Catalina Sau Man Ng,

The Education University of Hong

Kong, Hong Kong SAR, China

Roslina Othman,

International Islamic University

Malaysia, Malaysia

*Correspondence:

Xiaoting Zhang

zxtbing@hznu.edu.cn

†These authors share ﬁrst authorship

Specialty section:

This article was submitted to

Public Mental Health,

a section of the journal

Frontiers in Public Health

Received: 13 April 2022

Accepted: 13 May 2022

Published: 06 June 2022

Citation:

Huang X, Gao Y, Chen H, Zhang H

and Zhang X (2022) Hospital Culture

and Healthcare Workers’ Provision of

Patient-Centered Care: A Moderated

Mediation Analysis.

Front. Public Health 10:919608.

doi: 10.3389/fpubh.2022.919608

Hospital Culture and Healthcare

Workers’ Provision of

Patient-Centered Care: A Moderated

Mediation Analysis

Xianhong Huang 1†, Yuan Gao 1† , Hanlin Chen 1, Hao Zhang 1and Xiaoting Zhang2

1Department of Health Policy and Management, School of Public Health, Hangzhou Normal University, Hangzhou, China,

2Department of Administration, School of Public Administration, Hangzhou Normal University, Hangzhou, China

Background: Patient-centered care (PCC) is globally recognized as a high-quality and

high-value healthcare service. It emphasizes the broad participation of patients and

families in health-related decision-making and the provision of healthcare services that

cater to patients’ needs, preferences, and values. However, the mechanisms driving

healthcare workers’ provision of PCC are yet to be fully uncovered.

Methods: Using stratiﬁed random sampling, we recruited 1,612 healthcare workers from

different levels of public hospitals in Hangzhou. We conducted survey interviews using

questionnaires based on psychometrically sound scales. Structural equation modeling

was used to analyze the effects of hospital culture, self-efﬁcacy, and achievement

motivation on the perceived provision of PCC by healthcare workers and to explore the

mechanisms underlying their relationships.

Results: Self-efﬁcacy had a positive mediating effect in the relationship between

hospital culture and healthcare workers’ perceived provision of PCC (β=0.424, p<

0.001). Furthermore, the pursuit of success positively moderated the mediating role

of self-efﬁcacy (β=0.128, p<0.001), whereas, the avoidance of failure negatively

moderated the mediating role of self-efﬁcacy (β=-0.017, p<0.001).

Conclusion: The ﬁndings suggest that hospitals should foster patient-centered

and innovative cultures and develop strategies focusing on both internal motivation

(self-efﬁcacy and achievement motivation) and external environments (hospital culture)

to help and encourage healthcare workers to implement PCC. For example, hospitals

could further communication skills training, enhance leadership, build team spirit, and

promote collaboration among healthcare workers.

Keywords: hospital culture, healthcare worker, patient-centeredness, self-efﬁcacy, achievement motivation

INTRODUCTION

The notion of patient-centered care (PCC) was ﬁrst proposed by Balint in 1995 to express the

belief that healthcare workers—who are involved in the process of treatment—should be familiar

with patients’ living conditions, social environments, and disease progression; that is, healthcare

workers must deliver services that recognize and cater to the preferences, needs, and values of

Huang et al. Healthcare Workers’ Patient-Centered Care

patients (1). PCC is a high-value healthcare service that is crucial

to improving the quality of care and building harmonious doctor-

patient relationships (2,3). However, PCC calls for substantial

competence among healthcare workers as it is complex and can

be characterized as integrated medicine that is multi-leveled and

comprehensive, covering the entire life cycle (4,5). Therefore,

clarifying the driving mechanism for the provision of healthcare

is necessary to eﬀectively intervene in the behaviors of healthcare

workers. Furthermore, it is important to gain thorough and

systematic insights into the factors driving PCC from diﬀerent

aspects, such as outer contexts and intrinsic motivation.

In recent years, many scholars have tried to elucidate the

concept of PCC and explore the factors inﬂuencing behaviors

of healthcare professionals pertaining to the provision of

PCC; for example, in 2000, Ma studied the development

and improvement of the hospital service system based on

insights from the patient-centered approach, proposing a basic

framework and solutions for the establishment of a patient-

centered hospital service system (6). In 2002, Wang and

Liu (7) argued that “patient-centeredness” involves improving

doctor-patient communication, which can facilitate doctors’

understanding of their patients and help doctors and patients

reach a consensus on medical decisions. In 2019, Liang

(8) summarized Western studies on patient-centered medical

services and administration at the theoretical level and outlined

practice guidelines for the implementation of PCC in China.

Gender, grade, empathy, and communication skills were found

to have statistically signiﬁcant eﬀects on dental students’ attitudes

regarding patient-centered services, which could be improved via

a focus on enhancing empathy, emphasizing positive attitudes

toward learning communication skills, and conducting patient-

centered learning seminars (9). Furthermore, Kanat et al. (10)

found that the doctor-patient relationship and communication,

doctors’ characteristics, and patients’ engagement were important

determinants of PCC. Paiva et al. (11) investigated the factors

facilitating and inhibiting healthcare workers’ implementation of

PCC; they contended that the creation of an atmosphere that is

conducive to communication, engagement of patients in medical

decisions, and enhancement of medical personnel’s ability to

communicate eﬀectively might foster the provision of PCC by

medical workers.

Studies have linked PCC with various positive patient

outcomes, including empowerment and engagement (10,12)

favorable health outcomes, diminished socioeconomic, and racial

disparities, shorter hospitalization periods and earlier discharge,

and lower treatment costs (13,14). However, the existing

literature on PCC has the following drawbacks. First, while many

studies have identiﬁed various external (contextual) and internal

(personal) factors associated with PCC provision, comparatively

fewer studies have considered both contextual and personal

factors holistically and identiﬁed the pathways linking them.

It is extremely important to better understand how these

factors interact to facilitate (or undermine) healthcare workers’

provisions of PCC. Second, among the studies focusing on the

factors inﬂuencing PCC, few have focused on organizational

culture, which is an important factor that facilitates service ability

by valuing people, stimulating new thoughts, fostering team

spirit, and adopting systems that are recognized by employees.

Third, studies focusing on the driving mechanisms of healthcare

workers’ provision of PCC have not investigated the impact

of their intrinsic motivation. Thus, several studies so far have

revealed that self-eﬃcacy and achievement motivation have

joint eﬀects on personal behaviors (15,16). However, few have

been conducted in the ﬁeld of hospital administration, and

the mechanisms underlying the synergy between self-eﬃcacy

and achievement motivation are yet to be fully uncovered; for

example, it is unclear whether hospital culture has diﬀerent

eﬀects on the self-eﬃcacy of and provision of care by

healthcare professionals based on their level of achievement

motivation. Therefore, the pathways of eﬀects between hospital

culture and healthcare workers’ implementation of PCC are

worth investigating.

Motives—both physiological and social—have been identiﬁed

as major internal driving forces of human behavior in diverse

domains (17). Achievement motivation refers to the perceived

motivation that drives individuals to undertake challenging and

meaningful work tasks and/or activities and surpass others to

attain satisfactory outcomes (18). Thus, achievement motivation

is an important internal variable that is positively associated

with quality and initiative at work (19,20). With regard to

individual consciousness, achievement motivation is embodied

in two opposing psychological tendencies: the pursuit of success

and avoidance of failure (21). Schone (22) contended that strong

achievement motivation is a positive predictor of employees’ job

engagement, work performance, and organizational behaviors,

among others. Wang (20) explored the correlation between

doctors’ achievement motivation and sense of responsibility and

found that achievement motivation is positively associated with

service initiative. Feng (19) found that achievement motivation

is one of the factors driving community practitioners to deliver

ﬁrst-contact services. Song et al. (23) showed that doctors with

high achievement motivation maintain a positive outlook toward

work, are friendlier with patients, and can overcome emotional

exhaustion. According to McClelland’s (24) theory of needs (also

known as the theory of motivation), healthcare workers with a

high (vs. low) level of achievement motivation are more devoted

to work and are more focused, proactive, and persistent in

delivering PCC owing to positive feedback (25).

In light of the aforementioned ﬁndings, we advocated the

importance of considering the eﬀects of achievement motivation

while assessing the inﬂuence of hospital culture on patient-

centered practice among medical staﬀ. In addition, we included

self-eﬃcacy as an essential factor in the proposed model as this—

as another intrinsic variable—could reﬂect healthcare personnel’s

level of conﬁdence in the provision of PCC (26). Self-eﬃcacy

is an individual’s belief in their capacity to set and achieve

certain goals, that is, conﬁdence in one’s abilities (10). It is

important to note that self-eﬃcacy aﬀects people’s mindsets,

responsiveness to emotions, and choice of behaviors (27).

From a behavioral science perspective, self-eﬃcacy might be a

predictive factor for the adoption of and changes in health-

related behaviors (28). For example, Ham and Tak (29) suggested

that low self-eﬃcacy could lead to avoidance behaviors and

ineﬀective communication, resulting in poor clinical outcomes.

Frontiers in Public Health | www.frontiersin.org 2June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

Additionally, Afsar and Masood (30) found that enhancements

in self-eﬃcacy can facilitate innovative work-related behaviors

such as PCC. Furthermore, Zhang et al. (31) demonstrated

that self-eﬃcacy is positively associated with health protection

behaviors among pharmacists during the coronavirus disease

2019 pandemic.

Self-eﬃcacy can be inﬂuenced by several contextual factors

associated with hospital culture. For example, Meurling et al. (32)

suggested that teamwork (or collaboration) and communication

can contribute to the enhancement of self-eﬃcacy among

healthcare workers. Zhao et al. (33) proposed that the provision

of external support to healthcare workers can enhance their self-

eﬃcacy and reduce job burnout. Additionally, the engagement

and behaviors of leaders can inﬂuence the vitality, caliber,

performance, and cohesion of their team (34). Xue et al.

(35) found that leaders who favor and encourage changes can

set a positive example for their coworkers and employees,

fostering unity, and initiative in the organization; these factors

are conducive to the organization’s future. They also suggested

that work atmosphere is positively correlated with initiative

among employees. Furthermore, Liang and Gu (36) found that

innovative organizational culture has positive eﬀects on creativity

and motivation (to perform) among employees. Therefore, an

eﬃcient hospital culture serves several functions—including

guidance, bonding, stimulation, constraint, regulation, and

security—which can contribute to the enhancement of self-

eﬃcacy among healthcare workers.

“Hospital culture” refers to the sum of collective

consciousness, values, ethics, and norms in medical practice

held by hospital employees under certain socioeconomic

conditions (37). Studies have shown that hospital cultures

based on internal communication, cross-team collaboration,

innovation, and charismatic leadership (38–43) can facilitate

the implementation of PCC by enhancing patient-centered

consciousness among healthcare workers. Thus, the following

hypothesis was formulated.

Hypothesis 1 (H1): Hospital culture has a positive eﬀect on

healthcare workers’ implementation of PCC. In sum, on the

one hand, hospital culture has a direct positive eﬀect on the

provision of PCC by healthcare professionals; on the other

hand, self-eﬃcacy mediates the eﬀect of hospital culture on

healthcare workers’ implementation of PCC. Therefore, the

following hypotheses were formulated.

H2: Hospital culture positively aﬀects self-eﬃcacy among

healthcare workers.

H3: Self-eﬃcacy among healthcare workers has a positive eﬀect

on the delivery of PCC.

H4: Self-eﬃcacy mediates the relationship between hospital

culture and healthcare workers’ implementation of PCC.

Furthermore, achievement motivation moderates the

relationship between hospital culture and healthcare workers’

self-eﬃcacy for the provision of PCC. Gong and Xue (44)

explored the mechanism underlying the eﬀect of empowering

leadership on creativity among employees; they found

that achievement motivation moderated the relationship

between empowering leadership and self-eﬃcacy. In addition,

Sommaruga et al. (45) suggested that patient-centered practice

among medical personnel can be enhanced by emotional

intelligence. In sum, a good hospital culture has a relatively

weaker positive eﬀect on self-eﬃcacy among healthcare workers

with low achievement motivation. On the contrary, a good

hospital culture has a stronger positive eﬀect on the self-eﬃcacy

of healthcare professionals with high achievement motivation.

Therefore, the following hypotheses were formulated.

H5: Achievement motivation moderates the relationship

between hospital culture and self-eﬃcacy among

healthcare workers.

H6: Achievement motivation moderates the relationship

between self-eﬃcacy among healthcare workers and their

implementation of PCC.

Thus, we surveyed healthcare workers from 27 public hospitals

(of diﬀerent levels) in Hangzhou to explore the pathways linking

inner communication, cross-team collaboration, innovative

organizational culture, charismatic leaders, self-eﬃcacy, and

achievement motivation to healthcare personnel’s provision of

PCC, considering intrinsic motivation and hospital culture. In

light of theories of motivation, we proposed a theoretical model

with hospital culture as the independent variable, healthcare

professionals’ implementation of PCC as the dependent

variable, self-eﬃcacy as the mediating variable, and achievement

motivation as the moderating variable (Figure 1).

MATERIALS AND METHODS

Sample Characteristics

Healthcare workers—including doctors, nurses, and medical

technicians from diﬀerent levels of public hospitals in Hangzhou

city—voluntarily participated in this study. Hangzhou is a

provincial capital located in coastal southeast China. It is

economically well-developed and had a per capita GDP of

134,900 yuan in 2021. It is known for its quality of healthcare

and high-ranked hospital administration, rendering it an apt

setting for this study. The inclusion criteria were as follows: (1)

providing informed consent; (2) being employed in the target

hospital; and (3) having work experience of more than 6 months.

The exclusion criteria were as follows: (1) being oﬀ-duty during

the survey period owing to reasons like maternity leave, personal

aﬀairs, sick leave, learning, holiday, and/or business trips; and (2)

questionnaires from interns or trainees from other organizations.

Stratiﬁed random sampling was used to select healthcare

institutions based on three hospital levels; six tertiary and six

secondary hospitals and 15 community health centers were

randomly selected from among the medical organizations in

Hangzhou. Convenience sampling was then used to select

healthcare workers from each institution; 200 healthcare workers

were selected from each tertiary hospital, 50 from each

secondary hospital, and 20 from each community health center.

Accordingly, 1,800 questionnaires were sent to these institutions

through a face-to-face survey approach, of which 1,612 qualifying

questionnaires were retrieved. The criteria for considering

questionnaires invalid were: (1) incomplete responses; (2) same

responses for more than 50% of the completed questions;

Frontiers in Public Health | www.frontiersin.org 3June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

FIGURE 1 | sResearch model diagram of healthcare workers’ patient-centered care.

and (3) obvious contradictions in responses across questions.

If a questionnaire met even one of these criteria, it was

deemed invalid.

Measures

General Information

The form for general information comprised items pertaining

to gender, marital status, age, level of education, title, post,

department, years of experience, type of employment, daily

working hours, monthly income, hospital level, teaching status

of the hospital, communication skills training, and familiarity

with PCC.

Provider-Patient Relationship Questionnaire

To evaluate the healthcare personnel’s perceptions of their

provision of PCC, we used the Provider-Patient Relationship

Questionnaire (PPRQ), developed by Gremigni et al. (46). The

PPRQ includes four dimensions: eﬀective communication (four

items), interest in the patient’s agenda (four items), empathy (four

items), and patient involvement in care (four items). Items are

scored on a ﬁve-point Likert scale, ranging from one (never)

to ﬁve (always); higher scores reﬂect better patient-centered

services. The PPRQ has good reliability and validity (Table 1).

Hospital Culture

A self-developed hospital culture questionnaire (38–43) was

used to evaluate hospital culture based on four dimensions:

internal communication (three items), cross-team collaboration

(three items), innovative organizational culture (three items), and

charismatic leadership (three items). Items are rated on a ﬁve-

point Likert scale, ranging from one (strongly disagree) to ﬁve

(strongly agree); higher scores represent better hospital culture.

The scale had good reliability and validity (Table 1).

General Self-Efﬁcacy Scale

Self-eﬃcacy among healthcare workers was evaluated using the

General Self-Eﬃcacy Scale (GSES), developed by Schwarzer and

Jerusalem (47). The GSES has two components: conﬁdence about

interpersonal communication (ﬁve items) and conﬁdence about

eﬀective communication (ﬁve items). Items are rated on a ﬁve-

point Likert scale, ranging from one (strongly disagree) to ﬁve

(strongly agree), and higher scores represent higher self-eﬃcacy.

The scale had good reliability and validity (Table 1).

Achievement Motivation Measurement Scale

Healthcare workers’ achievement motivation was evaluated using

the Achievement Motivation Measurement Scale, developed by

Huang (21). This scale includes two sub-dimensions: pursuit of

success (four items) and avoidance of failure (four items). Items

are rated on a ﬁve-point Likert scale, ranging from one (“strongly

disagree”) to ﬁve (“strongly agree”); items in the avoidance of

failure subscale are reverse scored. Higher scores reﬂect higher

achievement motivation. The scale had good reliability and

validity (Table 1).

Quality Control

Prior to data collection, we conducted a pilot study. We gathered

and analyzed the problems that were identiﬁed in the preliminary

study. We then discussed the results of the preliminary study,

revised the questionnaire, established a speciﬁc study plan,

and ﬁnalized the questionnaire. Questionnaire distribution

was performed by postgraduates with adequate experience in

conducting personal surveys. The survey was conducted between

July 1 and September 30, 2021. Furthermore, concentrated

training sessions were conducted prior to the commencement

of the oﬃcial survey to ensure that the investigators had a

clear understanding of the project, questionnaire, and key points

during investigation; this process also ensured that they followed

Frontiers in Public Health | www.frontiersin.org 4June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

TABLE 1 | Description, factor analysis, and reliability coefﬁcients of construct measures.

Construct Dimension Items LoadaCronbach’s

Correlation

coefﬁcient

Cumulative

variance

contribution rate

(in %)

Overall

αvalue

Hospital culture Internal

communication

Group meetings are frequently organized in the

department to discuss patients’ conditions and

treatment plans.

0.763 0.834 0.719** 75.201 0.944

Resources and information can be shared within the

hospital.

0.786 0.754**

Employees can freely point out when incidents with

adverse consequences are likely to occur.

0.735 0.688**

Cross-team

collaboration

The hospital encourages transdisciplinary cooperation. 0.862 0.902 0.791** 83.661

Members in the department have mutual understanding

and acceptance.

0.86 0.783**

Different departments cooperate effectively and

efﬁciently to solve patients’ problems.

0.866 0.813**

Innovative

organizational

culture

Different departments in the hospital often collaborate

with each other to develop innovative ways of providing

health services.

0.822 0.936 0.830** 88.7

The hospital encourages constant innovation in health

information technologies.

0.892 0.829**

The hospital encourages constant innovation in

administrative skills and know-hows.

0.894 0.826**

Charismatic

leadership

The leaders strive to set a good example for employees. 0.891 0.912 0.826** 85.148

The leaders emphasize that employees should show

respect and concern toward patients and protect their

rights.

0.845 0.833**

The leaders encourage employees to participate in

discussions and decision-making processes.

0.851 0.802**

Achievement

motivation

Pursuit of

success

I would be very happy to get recognized by patients. 0.859 0.92 0.588** 71.998 0.86

I endeavor to provide personalized care for patients. 0.892 0.528**

I try to satisfy reasonable needs of patients and solve

their problems.

0.915 0.538**

I prefer completing the work assigned to me as quickly

as possible.

0.862 0.619**

Avoidance of

failure

I feel uneasy when treatment, examination, and/or

nursing don’t show clear effectiveness.

0.821 0.851 0.586** 74.796

I would feel anxious if I am unable to reach a consensus

with patients on decisions about treatment, examination,

and/or nursing.

0.876 0.566**

I dislike dealing with incidents involving malpractices

and/or disputes.

0.709 0.576**

I would feel anxious if I am unable to immediately

understand questions asked by patients.

0.859 0.616**

Self-efﬁcacy Conﬁdence

about

interpersonal

communication

I have a strong ability to build mutual trust with patients. 0.712 0.893 0.834** 69.321 0.942

I have a strong ability to discern patients’ emotions. 0.66 0.853**

I can solve patients’ emotional issues and physical

problems.

0.522 0.824**

Generally, I can be friendly with patients. 0.879 0.750**

I understand when patients do not approve of the

treatment, examination, and/or nursing plan suggested

by me.

0.643 0.745**

Conﬁdence

about effective

communication

My patients would honestly tell me their medical history if

I asked them.

0.586 0.911 0.834** 81.537

I have a strong ability to detect non-verbal hints or

actions of patients.

0.816 0.853**

(Continued)

Frontiers in Public Health | www.frontiersin.org 5June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

TABLE 1 | Continued

Construct Dimension Items LoadaCronbach’s

Correlation

coefﬁcient

Cumulative

variance

contribution rate

(in %)

Overall

αvalue

I possess the ability to ask pertinent questions at

appropriate times.

0.802 0.824**

I can explain medical terms in simple language. 0.727 0.750**

Patients would be willing to communicate with me if I

asked them about sensitive or private issues.

0.83 0.745**

Patient-centered

care

Patient

engagement

In most cases, I provide detailed information regarding

treatment, examination, and nursing to patients.

0.685 0.948 0.789** 78.435 0.967

In most cases, I allot ample time for patients to consult

with me.

0.72 0.780**

In most cases, I speak gently with patients. 0.558 0.818**

Empathy In most cases, I patiently listen to patients. 0.632 0.933 0.866** 79.26

In most cases, I care about the extent of patients’

understanding about their disease status and prognosis.

0.786 0.824**

In most cases, I inquire about patients’ preferences and

needs.

0.533 0.866**

Interest in

patients’

agendas

In most cases, I attach importance to the protection of

patients’ privacy.

0.802 0.905 0.795** 79.498

In most cases, I care about patients’ expectations about

the outcomes of care.

0.753 0.805**

In most cases, I can understand patients’ negative

emotions.

0.594 0.865**

In most cases, I can see things from the patients’

perspective.

0.612 0.878**

In most cases, I can bring conﬁdence and provide a

sense of security to patients.

0.588 0.878**

Effective

communication

In most cases, I can reach a consensus and resolve

conﬂicts together with patients.

0.695 0.857 0.892** 83.36

In most cases, I encourage patients to get involved in

discussions and decision-making about treatment,

examination, and/or nursing.

0.78 0.825**

In most cases, I can resolve patients’ concerns about

their diseases and offer timely help.

0.774 0.877**

In most cases, I allow patients to ask questions and

express their ideas when I ask about their symptoms.

0.764 0.876**

In most cases, I try to determine why a patient is

reluctant to receive care.

0.766 0.821**

aAll load values are signiﬁcant at the 0.001 level *p<0.05, **p<0.01, and ***p<0.001.

uniﬁed standards and methods. The survey was conducted

via one-on-one interviews after obtaining informed consent

from the participants. After the questionnaires were ﬁlled out,

the investigators inspected them and checked with participants

regarding the questionnaires that did not meet the study

requirements. Subsequently, the questionnaires were coded and

data were double-entered.

Statistical Analyses

Preliminary Analyses

Data were coded and entered in the database. The demographic

data were evaluated and compared by descriptive analysis, one-

way analysis of variance, and independent t-test. In addition,

SPSS version 22 (IBM Corp., Armonk, NY, United States) was

used to test the Pearson’s correlations between hospital culture,

self-eﬃcacy, achievement motivation, and PCC. Statistical

signiﬁcance was set at p<0.01.

Mediation and Moderation Analyses

Structural equation modeling was used to analyze the mediating

eﬀect of self-eﬃcacy among medical staﬀ and the moderating

eﬀect of achievement motivation. Hayes’ Process Macro Model

4 was employed to conduct mediation analysis, and Model

58 was employed for moderated mediation analysis (48). The

aforementioned program is suitable for a variety of mediation,

moderation, and moderated mediation models. Hypothesis

testing for the regression coeﬃcients was conducted using the

bias-corrected percentile bootstrap method with 5,000 replicates

Frontiers in Public Health | www.frontiersin.org 6June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

TABLE 2 | Comparison of healthcare workers’ mean scores on the

provider-patient relationship questionnaire based on different demographic

variables.

Variable Categorization (n) Total score

Gender Men (419) 66.74 ±9.68

Women (1,193) 66.68 ±9.08

t(p) 0.12 (0.90)

Marital status Married (1,154) 66.97 ±9.18

Single (426) 66.05 ±9.48

Divorced (21) 66.24 ±7.58

Other (11) 64.09 ±6.70

F (p) 1.34 (0.26)

Age ≤25 years (229) 65.37 ±10.12

26–35 (712) 66.10 ±9.09

36–45 (489) 67.74 ±8.83

≥46 years (182) 67.91 ±9.49

F (p) 5.77 (<0.01)

Level of education Junior college or below (221) 66.93 ±9.10

Graduate (1,142) 66.8 ±10.64

Postgraduate or above (249) 66.44 ±9.10

F (p) 1.37 (0.25)

Title None (177) 66.01 ±9.41

Junior (602) 65.57 ±9.46

Middle (556) 67.18 ±8.88

Sub-senior (211) 68.53 ±8.93

Senior (66) 68.92 ±9.10

F (p) 5.99 (<0.01)

Department Internal medicine (501) 66.20 ±9.13

Surgery (125) 67.35 ±8.55

Gynecology and obstetrics (96) 67.88 ±7.37

Pediatrics (64) 68.53 ±8.56

Medical technologies (230) 65.34 ±11.25

Emergency room (70) 65.50 ±8.71

Ophthalmology and

otorhinolaryngology (27)

65.26 ±10.14

Psychiatry (105) 67.38 ±8.32

Other (394) 67.46 ±8.93

F (p) 2.24 (0.05)

Daily working hours <8 (277) 65.99 ±9.54

8–10 (1,175) 66.73 ±9.21

10–12 (128) 67.30 ±9.16

>12 (32) 69.31 ±6.99

F (p) 1.59 (0.19)

Post Doctor (591) 67.98 ±8.55

Nurse (744) 65.90 ±9.02

Medical technician (256) 66.08 ±10.75

Other (21) 66.62 ±11.88

F (p) 6.10 (<0.01)

Level of hospital Tertiary (1,179) 67.77 ±9.87

Secondary (204) 65.15 ±10.34

Community health center 66.86 ±8.75

F (p) 3.31 (0.05)

(Continued)

TABLE 2 | Continued

Variable Categorization (n) Total score

Teaching status of

the hospital

Teaching hospital (1,259) 66.83 ±9.19

Not a teaching hospital (353) 66.23 ±9.37

t(p) 1.07 (0.28)

Level of familiarity

with

patient-centered

care

Very unfamiliar (56) 60.32 ±10.84

Quite unfamiliar (132) 63.47 ±10.17

Fairly familiar (639) 64.42 ±9.09

Quite familiar (653) 67.97 ±8.09

Very familiar (132) 73.10 ±8.78

F (p) 38.07 (<0.01)

Work experience

(years)

<1 (183) 66.19 ±9.11

1–5 (372) 65.97 ±9.53

5–10 (402) 66.35 ±9.25

10–15 (287) 66.78 ±8.91

15–20 (137) 68.21 ±8.83

>20 (231) 67.88 ±9.32

F (p) 2.20 (0.05)

at the 95% conﬁdence interval (CI). If it did not include 0, the

diﬀerence in eﬀect was considered statistically signiﬁcant.

Ethical Considerations

The study was approved by the Institutional Review Board

of Hangzhou Normal University. All participants provided

informed consent, and the study was performed in accordance

with the ethical standards as laid down in the 1964 Declaration of

Helsinki and its later amendments.

RESULTS

Demographic Characteristics

Of the 1,612 participants included in the study, 419 (54.3%)

were male, 1,154 (71.6%) were married, and 182 (11.3%)

were aged 46 or older. Additionally, 1,142 participants (70.8%)

had a bachelor’s degree, 602 (37.3%) had a junior title, 591

(36.5%) were doctors, and 501 (31.1%) were placed in the

internal medicine unit. Moreover, 231 participants (14.3%) had

work experience of 20 years or more, 1,199 (74.4%) held an

oﬃcially budgeted post, and 1,175 (72.9%) worked for 8–10 h

per day. Furthermore, 1,179 participants (73.1%) worked in

tertiary hospitals and 1,259 (78.1%) in teaching hospitals; 581

participants (36.0%) had a monthly income of 5,001–7,000 yuan.

Lastly, 958 participants (59.4%) had received training in patient-

centered communication skills, 285 participants (17.7%) took 5–

10 min on average to see one patient, and 132 participants (8.2%)

were “very familiar” with PCC.

The healthcare workers’ perceptions of their provision of

PCC were evaluated based on their scores on the PPRQ (M=

66.70, standard deviation (SD) =9.23). There were signiﬁcant

diﬀerences in participants’ PPRQ scores based on age (F=5.77,

p<0.01), title (F=5.99, p<0.01), post (F=6.10, p<0.01),

Frontiers in Public Health | www.frontiersin.org 7June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

TABLE 3 | Means, standard deviations, and correlation coefﬁcients of the variables.

Variable Mean SD Hospital

culture

Self-efﬁcacy Pursuit of

success

Avoidance of

failure

Patient-centered

care

Hospital culture 48.561 7.584 1

Self-efﬁcacy 38.891 6.228 0.549*** 1

Pursuit of success 16.899 2.647 0.630*** 0.638*** 1

Avoidance of failure 9.627 2.989 −0.270*** −0.326*** −0.400*** 1

Patient-centered care 66.478 9.387 0.662*** 0.559*** 0.628*** −0.265*** 1

*p<0.05, **p<0.01, and ***p<0.001. SD, standard deviation.

TABLE 4 | Testing the mediation model of self-efﬁcacy.

Variable Equation 1 (PCC) Equation 2 (Self-efﬁcacy) Equation 3 (PCC)

βSE tβSE tβSE t

Constant 23.609 1.339 17.635*** 13.877 0.983 14.113*** 17.721 1.344 13.186***

Age 0.628 0.280 2.241* 0.200 0.206 0.974 0.543 0.266 2.038*

Title 0.765 0.248 2.813* 0.765 0.182 4.197*** 0.373 0.237 1.576

Post −0.993 0.229 −4.342*** −0.627 0.168 −3.733*** −0.727 0.218 −3.331***

Familiarity with PCC 0.520 0.191 2.730* 0.599 0.140 4.282*** 0.266 0.182 1.461

Hospital culture 0.815 0.022 36.681*** 0.446 0.016 14.113*** 0.626 0.025 24.764***

Self-efﬁcacy 0.424 0.031 13.612***

R²0.466 0.345 0.517

F (5,1724) =300.384*** F (5,1724) =181.710*** F (6,1723) =307.960***

*p<0.05, **p<0.01, and ***p<0.001. PCC, patient-centered care.

and level of familiarity with PCC (F=38.07, p<0.01). Senior

healthcare workers (M =68.92, SD =9.10) obtained higher

scores than those with relatively junior titles (M=67.18, SD =

8.88). Doctors (M=67.98, SD =8.55) obtained higher scores

than nurses (M=65.90, SD =9.02). Furthermore, participants

who were “very familiar” with PCC obtained the highest scores

(M=73.10, SD =8.78), and those who were “very unfamiliar”

with PCC obtained the lowest scores (M=60.32, SD =10.84) on

the PPRQ. The PPRQ scores based on participants’ demographic

characteristics have been presented in Table 2.

The means, SDs, and correlation coeﬃcients of the variables

under study have been presented in Table 3. Hospital culture was

positively associated with self-eﬃcacy (r=0.549, p<0.001) and

healthcare workers’ perceived provision of PCC (r=0.662, p

<0.001). Similarly, there was a signiﬁcant positive association

between self-eﬃcacy and healthcare workers’ perceived provision

of PCC (r=0.559, p<0.001). Additionally, signiﬁcant positive

correlations were observed between the variables (hospital

culture, self-eﬃcacy, pursuit of success, avoidance of failure,

and PCC).

Analysis of the Mediating Effect of

Self-Efﬁcacy

First, a mediation model was constructed with hospital culture

as the independent variable and the perceived provision of

PCC as the dependent variable after controlling for age, title,

post, and level of familiarity with PCC (Table 4). The results

of Equation 1 showed that hospital culture had a signiﬁcant

positive eﬀect on healthcare workers’ perceived provision of

PCC (β=0.815, p<0.001). Bootstrapping was performed

using 5,000 bootstrap replicates (as parameter estimation). The

95% CI was 0.772–0.859 and did not include 0; thus, the

results supported H1, suggesting that hospital culture had

a positive eﬀect on healthcare workers’ perceived provision

of PCC.

Second, self-eﬃcacy was entered into the model as the

mediating variable. The results for the mediating eﬀects of self-

eﬃcacy have been depicted in Table 4 (Equations 2, 3). Hospital

culture had a signiﬁcant positive eﬀect on self-eﬃcacy (β=

0.446, p<0.001) with the 95% CI within the 0.414–0.478 range;

thus, the results supported H2, validating that hospital culture

positively inﬂuenced self-eﬃcacy among healthcare workers. As

shown in Equation 3, self-eﬃcacy had a signiﬁcant positive eﬀect

on healthcare workers’ perceived provision of PCC (β=0.424,

p<0.001); the estimated 95% CI was within the 0.363–0.485

range, and did not include zero. Thus, the results supported

H3, reﬂecting that self-eﬃcacy among healthcare workers had a

positive eﬀect on the delivery of PCC. Self-eﬃcacy had a partial

mediating eﬀect in the relationship between hospital culture

and PCC implementation. The size of the mediating eﬀect has

been presented in Table 5. Additionally, 76.81% of the total

eﬀects of hospital culture on the perceived provision of PCC

were direct eﬀects and 23.19% were indirect eﬀects, mediated

by self-eﬃcacy. These results supported H4, suggesting that self-

eﬃcacy mediated the relationship between hospital culture and

healthcare workers’ PCC implementation.

Frontiers in Public Health | www.frontiersin.org 8June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

TABLE 5 | Total, direct, and mediating effects of self-efﬁcacy in the relationship between hospital culture and healthcare workers’ perceived provision of patient-centered

care.

Effect size Boot SE Lower end of

boot CI

Upper end of

boot CI

Percentage

Total effect 0.815 0.022 0.772 0.859

Direct effect 0.626 0.025 0.577 0.676 76.81%

Mediating effect of self-efﬁcacy 0.189 0.024 0.143 0.238 23.19%

Boot, bias-corrected percentile bootstrap method; SE, standard error; CI, 95% conﬁdence interval. All values have been rounded up to three decimal places.

TABLE 6 | Analysis of the moderated mediation effect.

Antecedent Consequent

M (self-efﬁcacy) Y (PCC)

βSE tβSE t

Constant 35.896 5.724 52.967*** 49.26 1.425 34.567

Age 0.119 0.202 0.589 0.493 0.264 1.864

Title 0.74 0.178 4.148*** 0.364 0.235 1.549

Post −0.603 0.164 −3.664*** −0.747 0.217 −3.449***

Familiarity with PCC 0.62 0.137 4.510*** 0.26 0.181 1.432

Hospital culture 0.405 0.017 24.047*** 0.593 0.026 22.997***

Self-efﬁcacy 0.403 0.031 12.938***

Pursuit of success 0.211 0.041 5.171*** 1.064 0.252 4.217***

Avoidance of failure −0.493 0.223 −2.209*** 0.574 0.218 2.631**

Hospital culture ×Pursuit of success 0.022 0.043 3.479***

Hospital culture ×Avoidance of failure 0.074 0.014 −7.979***

Self-efﬁcacy ×Pursuit of success 0.128 0.115 4.385***

Self-efﬁcacy ×Avoidance of failure −0.017 0.005 −3.327***

R²=0.377 R²=0.522

F (7, 1722) =149.017*** F (8, 1721) =234.921***

All values have been rounded up to three decimal places. *p<0.05, **p<0.01, and ***p<0.001. Boot, bias-corrected percentile bootstrap method; SE, standard error; CI, 95%

conﬁdence interval; PCC, patient-centered care.

Analysis of the Moderating Effect of

Achievement Motivation

The moderating eﬀect of healthcare workers’ achievement

motivation was examined using the mediation model. The two

dimensions of achievement motivation were analyzed separately

as the pursuit of success was positively correlated with the

perceived provision of PCC, whereas the avoidance of failure

was negatively correlated with the perceived provision of PCC.

A moderated mediation model was constructed after decentering

the pursuit of success, avoidance of failure, and hospital

culture and including the interaction terms. The results of the

moderated mediation model have been displayed in Table 6.

The interactions between hospital culture and the pursuit of

happiness and the avoidance of failure had signiﬁcant eﬀects on

self-eﬃcacy (β=0.022, p<0.001; β= −0.493, p<0.001).

Simple slope tests (Figure 2A) further revealed that hospital

culture had a relatively weaker eﬀect on healthcare workers’ self-

eﬃcacy when the pursuit of success was low whereas hospital

culture had a relatively stronger eﬀect on healthcare workers’

self-eﬃcacy when the pursuit of success was high. Furthermore,

Figure 2B shows that hospital culture had a relatively stronger

eﬀect on healthcare workers’ self-eﬃcacy when the avoidance

of failure was low, whereas hospital culture had a relatively

weaker eﬀect on healthcare workers’ self-eﬃcacy when the

avoidance of failure was high. Thus, these results supported

H5, indicating that achievement motivation moderated the

relationship between hospital culture and healthcare workers’

self-eﬃcacy. Additionally, the pursuit of success had a positive

eﬀect in the relationship between the aforementioned variables,

whereas the avoidance of failure had a negative eﬀect.

Moreover, Table 6 demonstrates that the interaction eﬀects

between self-eﬃcacy and the pursuit of success and avoidance of

failure had signiﬁcant moderating eﬀects on PCC (β= −0.128,

p<0.001; β= −0.017, p<0.001). These results indicated that

the mediating eﬀect of self-eﬃcacy in the relationship between

hospital culture and the provision of PCC was moderated by

achievement motivation (comprising the pursuit of success and

avoidance of failure). To further evaluate the moderating eﬀect of

achievement motivation on the mediating eﬀect of self-eﬃcacy,

we performed bootstrap testing and explored the indirect eﬀect

Frontiers in Public Health | www.frontiersin.org 9June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

FIGURE 2 | (A) Moderating effect of the pursuit of success in the relationship between hospital culture and healthcare workers’ self-efﬁcacy. (B) Moderating effect of

the avoidance of failure in the relationship between hospital culture and healthcare workers’ self-efﬁcacy.

Frontiers in Public Health | www.frontiersin.org 10 June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

TABLE 7 | Bootstrap testing for the moderated mediation effect.

Moderating variable Boot SE Effect Bootstrap 95% CI

Lower end of

Boot CI

Higher end of

Boot CI

Pursuit of success

−1SD (14.253) 0.036 0.190 0.12 0.259

M (16.899) 0.033 0.265 0.199 0.331

+1SD (19.546) 0.037 0.340 0.268 0.413

Avoidance of failure

−1SD (6.637) 0.023 0.190 0.148 0.239

M (9.627) 0.02 0.172 0.135 0.215

+1SD (12.616) 0.028 0.153 0.105 0.217

Boot, bias-corrected percentile bootstrap method; SE, standard error; CI, 95%

conﬁdence interval. All values have been rounded up to three decimal places.

of the pursuit of success and avoidance of failure on three levels

of self-eﬃcacy. As shown in Table 7, the results demonstrate that

self-eﬃcacy had a stronger mediating eﬀect on the perceived

provision of PCC among healthcare workers with a high need

for pursuing success (ρ=0.340, 95% CI not including 0) as

compared to healthcare workers with a low need for pursuing

success (ρ=0.190, bootstrap 95% CI not including 0). On the

contrary, self-eﬃcacy had the weakest mediating eﬀect on the

perceived provision of PCC among healthcare workers with a

high need for avoidance of failure; that is, the indirect eﬀect of

self-eﬃcacy was lowest when the avoidance of failure was 1SD

above the mean (ρ=0.153, 95% CI not including 0).

DISCUSSION

Effects of Hospital Culture on PCC

The present study found that hospital culture played an

important role in healthcare workers’ care delivery. As a

manifestation of healthcare personnel’s collective values and

norms, hospital culture can deeply inﬂuence their mindset

and behaviors. Consistent with Zhang’s study (13), innovative

organizational culture was found to have a positive eﬀect on

healthcare workers’ provision of PCC. This could be because

an open atmosphere stimulates positivity and creativity among

employees, creating a favorable environment for them to work on

new projects and develop new skills, and especially to implement

innovative patient-focused treatment plans. Additionally, the

present ﬁndings are corroborated by those of Zhu’s study

(14) as charismatic leadership facilitated empathy for patients

among healthcare workers; participants reported that they

could think from the patients’ perspective, and listen to and

communicate eﬀectively with them. Leaders eﬀectively enhanced

their employees’ awareness of PCC and the importance of

empathy by urging them to respect and care about patients and

protect patients’ rights.

Healthcare workers who had received training to develop

their communication skills scored higher than those who had

not received such training. This ﬁnding is consistent with Jeong

and Park’s study (49), suggesting that a healthy atmosphere

involving communication is conducive to healthcare workers’

eﬃciency in the provision of healthcare services as it boosts

understanding and togetherness, fostering tolerance and unity.

Additionally, in line with the conclusions drawn by Fralicx

(50), cross-team collaboration was associated with eﬀective

communication among healthcare workers. Thus, hospitals

can set up multidisciplinary teams and construct streamlined

communication or feedback mechanisms to enhance cooperation

among diﬀerent departments and facilitate the implementation

of PCC.

Mediating Effects of Self-Efﬁcacy

While exploring the mechanisms for the adoption of PCC, it is

beneﬁcial to examine the mediating eﬀects of self-eﬃcacy in the

relationship between hospital culture and healthcare personnel’s

provision of PCC. Consistent with Yang et al. (51), the current

study revealed that hospital culture aﬀected healthcare workers’

perceived provision of PCC, which was mediated by their self-

eﬃcacy. Self-eﬃcacy can have direct eﬀects on an individual’s

choices, goals, mindset, and attribution style, among others.

Good hospital culture can foster healthcare professionals’ self-

eﬃcacy, further enabling them to implement PCC.

The positive inﬂuence of hospital culture on self-eﬃcacy can

be explained in two ways. First, a supportive environment within

a healthy hospital culture could increase healthcare workers’

self-eﬃcacy (52). When leaders highlight and champion the

use of PCC in addition to advances in medical technologies

and services that empower cross-team collaboration, healthcare

workers might experience high levels of meaningfulness and

conﬁdence in completing their work; this could accelerate their

ability to deliver healthcare services. Second, communication

is an important factor in patient-centered medicine and calls

for healthcare personnel’s transition from traditional attitudes

to attitudes of respect and care for patients’ preferences, needs,

and values (53). Open-minded leaders, innovative culture,

and multidisciplinary cooperation can boost medical workers’

conﬁdence in eﬀective communication, and thus, increase their

willingness to implement PCC.

Meanwhile, self-eﬃcacy had a positive eﬀect on PCC;

this is consistent with Sommaruga et al.’s ﬁndings (45),

suggesting that healthcare providers with high self-eﬃcacy

had good interpersonal relationships and professional skills.

Welsh (54) suggested that the enhancement of self-eﬃcacy

among doctors was eﬀective in improving their communication

skills; for example, doctors with high self-eﬃcacy—consistent

with the requirements of PCC—provided more disease-related

information and medical knowledge to patients; encouraged

them to communicate; and were responsive to their questions,

suggestions, needs, and worries.

Moderating Effects of Achievement

Motivation

According to McClelland’s (45) achievement need theory,

achievement motivation among social members stems from

speciﬁc social and cultural environments. In this study, we

constructed a moderated mediation model based on achievement

theories. Achievement motivation exerted signiﬁcant moderating

Frontiers in Public Health | www.frontiersin.org 11 June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

eﬀects in the relationship between hospital culture and self-

eﬃcacy. In the same culture, healthcare workers with high

achievement motivation and self-eﬃcacy were highly willing to

communicate with patients. This could be because they cared

about the patients’ feelings and were mindful about responding

to them and meeting patients’ preferences and needs. This is in

line with Wang’s study (20), suggesting that healthcare workers’

self-eﬃcacy, and hence, achievement motivation, are reinforced

when they overcome problems and diﬃculties in services. On the

contrary, when issues keep sustaining, achievement motivation is

triggered among healthcare workers with high self-eﬃcacy owing

to their conﬁdence, whereas healthcare workers with low self-

eﬃcacy might adopt negative attitudes concerning the provision

of PCC to avoid failure.

In line with Khongsamai et al.’s ﬁndings (55), in the current

study, an innovative organizational culture contributed to an

increase in eﬀective communication. This could be because an

open and innovative work environment facilitates initiatives

among healthcare workers, improves their consciousness about

communication, fosters innovation in communication styles

and skills, and thus, promotes doctor-patient communication.

On the contrary, achievement motivation moderated the

relationship between self-eﬃcacy and patient-centered practice

among healthcare workers. Similar to Yim and Lee’s ﬁndings (56),

we found that healthcare workers who pursued success tended to

have strong social responsibility, introspected about their actions

based on patients’ feedback, showed progress in ethical values and

skills, and encouraged patients to participate in discussions and

express their feelings.

The avoidance of failure was found to negatively moderate the

relationship between self-eﬃcacy and the provision of PCC; this

has repeatedly been demonstrated in the ﬁelds of psychology and

education (57,58). In the current study, this could be because

medical personnel with a strong need to avoid failure might have

anticipated job burnout owing to the patient-centered services

that might require several emotional resources in addition to

concentration; they might also have been worried about the

status of rewards and outcomes of the long-term input. However,

the negative eﬀect of avoidance of failure was not stronger

than the positive eﬀect of pursuit of success, implying that the

healthcare workers’ need to pursue success primarily inﬂuenced

their achievement motivation.

Implications and Limitations

We believe that the present results have theoretical and

practical signiﬁcance. To begin with, the ﬁndings have strong

implications for the development of strategies for stimulating

healthcare workers’ provision of PCC, which might be eﬀective

in transforming hospital administration methods and enhancing

the eﬃciency of healthcare services. For example, to strengthen

patient services, hospitals should establish an innovative

culture, constantly work on creativity, encourage healthcare

workers to innovate, and stimulate healthcare workers’ internal

achievement motivation by challenging them. Second, hospitals

should establish a charismatic leadership culture. Hospital

leaders should set an example and fully mobilize healthcare

workers’ consciousness and enthusiasm to serve patients.

Moreover, hospital leaders should pay attention to their

own quality improvement, respect healthcare workers, and

be sensitive to their needs. Third, hospitals should establish

a multidisciplinary diagnosis and treatment team to foster a

culture of cooperation. By facilitating information exchange

and providing a sharing platform for each department, a good

communication and feedback mechanism is formed within

the hospital, promoting internal cooperation. A good internal

communication atmosphere is helpful for healthcare workers to

understand each other, cultivate team spirit, and improve patient

service eﬃciency. Our study found that the external hospital

culture drives healthcare workers’ attitudes and behaviors. It is

not combined with the viewpoint of motivation theory, which

adds further value to the literature on motivation theories.

The study also has some limitations. First, the recruited

healthcare workers were from hospitals in Hangzhou; this could

limit the representativeness of the sample and generalizability

of our ﬁndings. Thus, further research should be conducted

in other regions of China. Second, this study adopted a cross-

sectional design; consequently, we could not capture the causal

eﬀect of changes over time. Therefore, future studies must

adopt longitudinal, experimental, or cross-sequential designs and

employ hierarchical linear models; studies must also test for the

confounding and mediating eﬀects of diﬀerent variables. Third,

this study explored healthcare workers’ perceptions of their

provision of PCC; future studies must focus on the perspectives

of both healthcare providers and patients. Fourth, all the variables

were assessed through self-report, and although surveys were

answered anonymously, social desirability bias may still have

inﬂuenced the responses to some extent.

CONCLUSIONS

The present study indicated that hospital culture can aﬀect

healthcare workers’ implementation of PCC. Accordingly,

hospitals could organize activities for healthcare workers

to discuss hospital culture; this would strengthen their

understanding of the importance of hospital culture.

Additionally, hospital culture can boost the provision of

PCC via the enhancement of self-eﬃcacy and achievement

motivation among healthcare workers. Therefore, hospital

administrators should pay attention to the psychological status

of their staﬀ and develop their conﬁdence in interpersonal

networking and eﬀective communication to help them build

their self-eﬃcacy for PCC and motivation for success.

DATA AVAILABILITY STATEMENT

The original contributions presented in the study are included

in the article/supplementary material, further inquiries can be

directed to the corresponding author/s.

ETHICS STATEMENT

The study was approved by the Institutional Review Board of

Hangzhou Normal University. All study participants provided

Frontiers in Public Health | www.frontiersin.org 12 June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

informed consent, and the study was performed in accordance

with the Ethical Standards as laid down in the 1964 Declaration

of Helsinki and its later amendments. The patients/participants

provided their written informed consent to participate in

this study.

AUTHOR CONTRIBUTIONS

XH conceptualized the study, drafted the methodology, and

operated the software. YG performed the statistical analysis and

prepared the ﬁrst draft of the manuscript. HC helped with

visualization and investigation. HZ operated the software and

validated the ﬁndings. XZ helped in writing, reviewing, and

editing the manuscript. All authors approved the submitted

version of the manuscript.

FUNDING

This study was supported by the National Natural Science

Foundation of China Project (Grant No. 72004051) and the Soft

Science Research Program of Zhejiang Provincial Science and

Technology Plan (Grant No. 2021C35012).

ACKNOWLEDGMENTS

We would like to express our deepest gratitude to the

Health Commission of Zhejiang Province for their support in

conducting the research. We would like to thank the participants

for their valuable time and eﬀort in completing the questionnaire.

We are also grateful to Jixiang Lai who helped us improve

the manuscript.

REFERENCES

1. Sowerby P. Balint reassessed: the doctor, his patient, and the illness: a

reappraisal. J R Coll Gen Pract. (1977) 27:583–9.

2. Benson T. Principles of Health Interoperability HL7 and SNOMED. London:

Springer (2012). p. 802–3. doi: 10.1007/978-1-4471-2801-4

3. Berghout M, van Exel J, Leensvaart L, Cramm JM. Health care professionals’

views on patient-centered care in hospitals. BMC Health Serv Res. (2015)

15:e385. doi: 10.1186/s12913-015-1049-z

4. Jeske L, Kolmer V, Muth M, Cerns S, Moldenhaur S, Hook ML.

Partnering with patients and families in designing visual cues to

prevent falls in hospitalized elders. J Nurs Care Qual. (2006) 21:236–41.

doi: 10.1097/00001786-200607000-00008

5. Mead N, Bower P. Patient-centredness: a conceptual framework and

review of the empirical literature. Soc Sci Med. (2000) 51:1087–110.

doi: 10.1016/s0277-9536(00)00098-8

6. Ma XS. Construction of patient-centered care system in

hospital. Med Soc. (2000) 1:57–9. Available online at: https://

/kns.cnki.net/kcms/detail/detail.aspx?dbcode=CJFD&dbname=CJFD2000&ﬁl

ename=YXSH200001024&uniplatform=NZKPT&v=yOVmzoHFGx-MnvFrCa

5mnsbwHzTso0Kwh6P6acyEsKZPVAjNIspfgLvZ-Km88Suf

7. Wang XW, Liu LH. Development and conception of medical care model of

patient centered medicine. Med Philos. (2002) 3:24–7. Available online at:

https://kns.cnki.net/kcms/detail/detail.aspx?dbcode=CJFD&dbname=CJFD20

02&ﬁlename=YXZX200203007&uniplatform=NZKPT&v=1hiib4dgXEG66FN

U2IKxsJvSBetoIWIqljjxtzkO3Fg1Jtr-jHJqVkigB1pDqZEy/33.nsdetail.aspx?db

code=CJFD&dbname=CJFDLAST2019&ﬁlename=YYGL201911021&uniplatf

orm=NZKPT&v= qPq4aSA9xIVd28eNhFH6GN6plCtxR43D1JYlVh_FmYDl

qO5hNpHugGUMorm7cYSm

8. Liang HL. Patient-Centered Care and Management. Beijing: Chemical

Industry Press (2019).

9. Lee M, Ihm J. Empathy and attitude toward communication skill

learning as a predictor of patient-centered attitude: a cross-sectional

study of dental students in Korea. BMC Med Educ. (2021) 21:e225.

doi: 10.1186/s12909-021-02674-z

10. Kanat M, Schaefer J, Kivelitz L, Dirmaier J, Voigt-Radloﬀ S, Heimbach B, et al.

Patient-centeredness in the multimorbid elderly: a focus group study. BMC

Geriatr. (2021) 21:e567. doi: 10.1186/s12877-021-02448-8

11. Paiva D, Abreu L, Azevedo A, Silva S. Patient-centered communication in

type 2 diabetes: the facilitating and constraining factors in clinical encounters.

Health Serv Res. (2019) 54:623–35. doi: 10.1111/1475-6773.13126

12. Yan YH, Yang CW, Fang SC. Patient-centered e-health supports value

cocreation and quality of medical care in Taiwan. Health Info Libr J. (2022)

39:68–78. doi: 10.1111/hir.12372

13. Epstein RM, Fiscella K, Lesser CS, Stange KC. Why the nation needs a

policy push on patient-centered health care. Health Aﬀ. (2010) 29:1489–95.

doi: 10.1377/hlthaﬀ.2009.0888

14. Tomaselli G, Buttigieg SC, Rosano A, Cassar M, Grima G. Person-centered

care from a relational ethics perspective for the delivery of high quality

and safe healthcare: a scoping review. Front Public Health. (2020) 8:e44.

doi: 10.3389/fpubh.2020.00044

15. Zhang R. The Inﬂuence of Patient-Centered Hospital Culture on the

Professionalism of Doctors in Public Hospitals. Dissertation, Huazhong

University of Science and Technology,Wuhan (2017).

16. Zhu Y. Practice of empathy in staﬀ management of public hospitals. China

Health Ind. (2018) 339:199–200. doi: 10.16659/j.cnki.1672-5654.2018.04.193

17. Ryan RM. Motivation and the organization of human behavior: three reasons

for the reemergence of a ﬁeld. In: Ryan RM, editor. The Oxford Handbook

of Human Motivation. Oxford: Oxford University Press (2012). p. 3–10.

doi: 10.1093/oxfordhb/9780195399820.013.0001

18. Peng DL. General Psychology. Beijing: Beijing Normal University Press (2004).

p. 320–9.

19. Feng W. Study on the formative mechanism and incentives of the community

doctor ﬁrst contact service motivation (dissertation). Hangzhou Normal

University, Hangzhou (2017).

20. Wang H. Research on the relationship of doctors’ achievement motivation

and sense of responsibility in the city of Qingdao (dissertation). Southwest

University, Chongqing (2009).

21. Huang YZ. Study on professional commitment and its relationship with

achievement motivation for male undergraduates toward nursing in Xuzhou

(dissertation). Xuzhou Normal University, Xuzhou (2011).

22. Schoen JL. Eﬀects of implicit achievement motivation, expected evaluations,

and domain knowledge on creative performance. J Organ Behav. (2015)

36:319–38. doi: 10.1002/job.1982

23. Song H, Xu YX, Song J. Study on the relationship between achievement

motivation and job burnout of standardized training residents.

China Contin Educ. (2017) 9:1–3. doi: 10.3969/j.issn.1674-9308.2017.

12.001

24. McClelland DC, Koestner R. The achievement motive. In: Smith CP,

editor. Motivation and Personality: Handbook of Thematic Content

Analysis. Cambridge: Cambridge University Press (1992). p. 143–52.

doi: 10.1017/CBO9780511527937.010

25. Guo B, Qiang B, Zhou J, Yang X, Qiu X, Qiao Z, et al. The relationship between

achievement motivation and job performance among Chinese physicians:

a conditional process analysis. BioMed Res Int. (2021) 2021:6646980.

doi: 10.1155/2021/6646980

26. Guo BY, Jiang FY. Self-eﬃcacy Theory and its Application. Shanghai: Shanghai

Education Press (2008). p. 263–7.

27. Bandura A. Social Foundations of Thought and Action: A Social Cognitive

Theory. Hoboken NJ: Prentice Hall (1986). p. 10–1.

28. Wang CL. Conceptual model of information framing eﬀect

in diﬀerent stages of health behavior change. Libr Inf

Serv. (2019) 63:23–31. doi: 10.13266/j.issn.0252-3116.2019.

05.003

Frontiers in Public Health | www.frontiersin.org 13 June 2022 | Volume 10 | Article 919608

Huang et al. Healthcare Workers’ Patient-Centered Care

29. Ham HJ, Tak SH. Factors inﬂuencing transcultural self-eﬃcacy among nurses

with foreign patient care experience. J Transcult Nurs. (2022) 33:87–95.

doi: 10.1177/10436596211016514

30. Afsar B, Masood M. Transformational leadership, creative self-eﬃcacy, trust

in supervisor, uncertainty avoidance, and innovative work behavior of nurses.

J Appl Behav Sci. (2018) 54:36–61. doi: 10.1177/0021886317711891

31. Zhang N, Yang H, Hong D, Huang X, Wang L. Risk perception, self-eﬃcacy,

lay theories of health, and engagement in health-protective behaviors among

hospital pharmacists during the COVID-19 pandemic. Int J Behav Med.

(2021) 23:1–6. doi: 10.1007/s12529-021-10004-2

32. Meurling L, Hedman L, Sandahl C, Felländer-Tsai L, Wallin CJ. Systematic

simulation-based team training in a Swedish intensive care unit: a diverse

response among critical care professions. BMJ Qual Saf. (2013) 22:485–94.

doi: 10.1136/bmjqs-2012-000994

33. Zhao YG, Li XX, Cui Y, Qiu YQ. Analysis of the impact of social support on job

burnout among doctors and nurses: based on mediation eﬀect of self-eﬃcacy.

Chin Hosp Manag. (2019) 39:48–50.

34. Liu YA. Research on the relationship between authentic leadership and team

innovation from the perspective of positive cultural atmosphere. Leadersh Sci.

(2018) 11:24–6. doi: 10.19572/j.cnki.ldkx.2018.11.006

35. Xue X, Xue W, Yao C. Study on relationship between transformational

leadership and individual initiative—achievement motivation as mediating

variable. J Zhejiang Sci Tech Univ. (2015) 34:385–9. Available online at:

https://kns.cnki.net/kcms/detail/detail.aspx?dbcode=CJFD&dbname=CJFDLA

ST2015&ﬁlename=ZJSG201510004&uniplatform=NZKPT&v=2oQqchrh0gN

I5EYGLm1sekqeiQoPFhsKuryUPcvQaV6UyTiyymRxSHgnHTz8y0hZ

36. Liang BQ, Gu QX. Inﬂuence of innovative cultural awareness

on the intrinsic motivation and creativity from the performance-

oriented perspective. Shanghai Manag Sci. (2016) 38:89–96.

doi: 10.3969/j.issn.1005-9679.2016.03.018

37. Zhang LL. Hospital Administration. USA: People’s Medical Publishing

House (2014).

38. Burton D, Blundell N, Jones M, Fraser A. Elwyn G. Shared decision-making in

cardiology: do patients want it and do doctors provide it? Patient Educ Couns.

(2010) 80:173–9. doi: 10.1016/j.pec.2009.10.013

39. Peek ME, Wilson SC, Gorawara-Bhat R, Odoms-Young A, Quinn MT,

Chin MH. Barriers and facilitators to shared decision-making among

African-Americans with diabetes. J Gen Intern Med. (2009) 24:1135–9.

doi: 10.1007/s11606-009-1047-0

40. Hower KI, Vennedey V, Hillen HA, Kuntz L, Stock S, Pfaﬀ H, et al.

Implementation of patient-centered care: which organizational determinants

matter from decision maker’sperspective? Results from a qualitative interview

study across various health and social care organizations. BMJ Open. (2019)

9:e027591. doi: 10.1136/bmjopen-2018-027591

41. Jakimowicz S, Perry L, Lewis J. An integrative review of supports, facilitators

and barriers to patient-centered nursing in the intensive care unit. J Clin Nurs.

(2017) 4:25–6. doi: 10.1111/jocn.13957

42. Bokhour BG, Fix GM, Mueller NM, Barker AM, Lavela SL, Hill JN, et al. How

can healthcare organizations implement patient-centered care? Examining

a large-scale cultural transformation. BMC Health Serv Res. (2018) 18:e168.

doi: 10.1186/s12913-018-2949-5

43. Alharbi TSJ, Ekman I, Olsson LE, Dudas K, Carlström E. Organizational

culture and the Implementation of person centered care: results from a

change process in Swedish hospital care. Health Policy. (2012) 108:294–301.

doi: 10.1016/j.healthpol.2012.09.003

44. Gong W, Xue J. The impact of empowering leadership on employee creativity:

roles of personal learning and achievement motivation. J Technol Econ.

(2020) 39:21–9. Available online at: https:/ /kns.cnki.net/kcms/detail/detail.

aspx?dbcode=CJFD&dbname=CJFDLAST2020&ﬁlename=JSJI202002004&u

niplatform=NZKPT&

45. Sommaruga M, Casu G, Giaquinto F, Gremigni P. Self-perceived provision

of patient centered care by healthcare professionals: the role of emotional

intelligence and general self-eﬃcacy. Patient Educ Couns. (2017) 100:974–80.

doi: 10.1016/j.pec.2016

46. Gremigni P, Casu G, Sommaruga M. Dealing with patients in healthcare:

a self-assessment tool. Patient Educ Couns patient ed. (2016) 99:1046–53.

doi: 10.1016/j.pec.2016.01.015

47. Schwarzer R, Jerusalem M. (1995). Generalized self-eﬃcacy scale. In:

Weinman J, Wright S, Johnston M, editors. Measures in Health Psychology: A

User’s Portfolio. Causal and Control Beliefs. Windsor: NFER-NELSON (1995).

p. 35–7.

48. Hayes AF. Introduction to mediation, moderation, and conditional process

analysis. J Educ Meas. (2013) 51:335–7. doi: 10.1111/jedm.12050

49. Jeong H, Park M, A. predictive model on patient-centered care of

hospital nurses in Korea. J Korean Acad Nurs. (2019) 49:191–202.

doi: 10.4040/jkan.2019.49.2.191

50. Fralicx R. Strange bed(side) fellows physician-ﬁnance collaboration. Healthc

Financ Manage. (2012) 66:90–4, 96.

51. Yang MJ, Ge M, Yang XQ. Study on the correlation

between self-leadership and self-eﬃcacy of nursing

students. Nurs Res. (2018) 32:1753–7. Available online at:

https://kns.cnki.net/kcms/detail/detail.aspx?dbcode=CJFD&dbname=CJFDL

AST2018&ﬁlename=SXHZ201811025&uniplatform=NZKPT&v=goEPf2dkH

s-Fkbw0wxAVQ1my7XSLQuZ8HKGH2TK51JB6v9XONB111wDT1tY5HeAJ

52. Rahmer BM. Integrative exploration of the primary care interaction: tensions

and ideals in patient-centered care (dissertation). University of Delaware, Ann

Arbor (MI) (2014).

53. Liu JH. Building a new doctor-patient relationship empowers the

construction of a healthy China. Peoples Tribune. (2021) 724:86–8.

doi: 10.3969/j.issn.1004-3381.2021.33.020

54. Welsh JA, Nelson JM, Walsh S, Sealer H, Palmer W, Vos MB. Brief

training in patient-centered counseling for healthy weight management

increases counseling self-eﬃcacy and goal setting among pediatric primary

care providers: results of a pilot program. Clin Pediatr. (2015) 54:425–9.

doi: 10.1177/0009922814553432

55. Khongsamai T, Intarakamhang U. Causal model of cultural competence

and behavior among Thai nurses in international hospitals. J Behav Aci.

(2020) 15:1–5. Available online at: https://www.researchgate.net/publication/

346410346

56. Yim KH, Lee I. The eﬀect of achievement motivation on learning agility

of nursing students: The mediating eﬀect of self-leadership. The eﬀect of

achievement motivation on learning agility of nursing students: the mediating

eﬀect of self-leadership. J Korean Acad Soc Nurs Educ. (2021) 27:80–90.

doi: 10.5977/jkasne.2021.27.1.80

57. Capa RL, Audiﬀren M, Ragot S. The interactive eﬀect of

achievement motivation and task diﬃculty on mental eﬀort. Int

J Psychophysiol. (2008) 70:144–50. doi: 10.1016/j.ijpsycho.2008.

06.007

58. Jiang FJ. A study of inﬂuencing factors of college students’ behavioral

intention to learn in blended teaching environment. Electron

Educ Res. (2021) 42:105–12, 28. doi: 10.13811/j.cnki.eer.2021.

06.015

Conﬂict of Interest: The authors declare that the research was conducted in the

absence of any commercial or ﬁnancial relationships that could be construed as a

potential conﬂict of interest.

The reviewer XW declared a shared aﬃliation with all authors at the time

of review.

Publisher’s Note: All claims expressedin this article are solely those of the authors

and do not necessarily represent those of their aﬃliated organizations, or those of

the publisher, the editors and the reviewers. Any product that may be evaluated in

this article, or claim that may be made by its manufacturer, is not guaranteed or

endorsed by the publisher.

article distributed under the terms of the Creative Commons Attribution License (CC

BY). The use, distribution or reproduction in other forums is permitted, provided

the original author(s) and the copyright owner(s) are credited and that the original

publication in this journal is cited, in accordance with accepted academic practice.

No use, distribution or reproduction is permitted which does not comply with these

terms.

Frontiers in Public Health | www.frontiersin.org 14 June 2022 | Volume 10 | Article 919608

OPINION

published: 11 July 2022

doi: 10.3389/fpsyg.2022.905811

Frontiers in Psychology | www.frontiersin.org 1July 2022 | Volume 13 | Article 905811

Edited by:

Tindara Caprì,

National Research Council of Italy

(CNR), Italy

Reviewed by:

Usree Bhattacharya,

University of Georgia, United States

*Correspondence:

Fabrizio Stasolla

f.stasolla@unifortunato.eu

Specialty section:

This article was submitted to

Psychology for Clinical Settings,

a section of the journal

Frontiers in Psychology

Received: 27 March 2022

Accepted: 22 June 2022

Published: 11 July 2022

Citation:

Stasolla F, Vinci LA and Cusano M

(2022) The Integration of Assistive

Technology and Virtual Reality for

Assessment and Recovery of

Post-coma Patients With Disorders of

Consciousness: A New Hypothesis.

Front. Psychol. 13:905811.

doi: 10.3389/fpsyg.2022.905811

The Integration of Assistive

Technology and Virtual Reality for

Assessment and Recovery of

Post-coma Patients With Disorders

of Consciousness: A New Hypothesis

Fabrizio Stasolla*, Leonarda Anna Vinci and Maria Cusano

Giustino Fortunato University of Benevento, Benevento, Italy

Keywords: post-coma, disorders of consciousness, assessment, rehabilitation, new technologies, acquired brain

injuries

INTRODUCTION

Multiple disabilities due to an outcome of coma combined with severe to profound disorders of

consciousness may pose serious challenges to daily medical centers and rehabilitative settings.

Beside speciﬁc pharmacological treatments delivered by specialized professionals, they may need

diagnostic tools and rehabilitative interventions enabling those patients with an active role,

constructive engagement, positive participation, independence, and self-determination (Pistoia

et al., 2008; Lancioni et al., 2014b; Formisano et al., 2018; Kulyk, 2019). Thus, two basic objectives

may be targeted within this framework, namely (a) the assessment and (b) the recovery of cognitive,

motor, and communicative functioning (Lancioni et al., 2009a, 2011; Kirsch et al., 2017; de

Tommaso et al., 2020).

Many clinical and research eﬀorts have recently been devoted to the aforementioned critical

features (i.e., evaluation and rehabilitation). With regard to the assessment, two main viewpoints

may be emphasized. First of all, the existing literature is focused on determining the patient’s

state of functioning. That is, it is useful to identify whether patients are in a vegetative state or

a more favorable diagnosis of minimally conscious state could be made (Lancioni et al., 2008a;

Formisano et al., 2011; Pistoia and Sarà, 2012). Secondly, the dichotomy between the two above

clinical conditions (i.e., vegetative state or and minimally conscious state) is critically discussed

and any speciﬁc need to clarify the borderline between those two states requires to rectify more

straightful strategies (Kim et al., 2012).

With regard to the rehabilitation, diﬀerent approaches may be acknowledged. For instance,

one may envisage environmental stimulation (Lancioni et al., 2014a, 2015). Otherwise, deep

brain stimulation may be adopted (Lancioni et al., 2010b). Additionally, brain computer interface

strategies may be implemented (Stasolla and De Pace, 2014). Those strategies rely on diﬀerent

theoretical background which may have clinical and practical implications on the role of the

assessment and the role of the patient. The decision on whether the person is in a vegetative state or

in a minimally conscious state should be considered crucial prior to the intervention and the setup

for the intervention should be highly individualized to ensure the participant with a successful

learning process (Lancioni et al., 2017).

The purpose of this paper is to argue on both assessment and rehabilitative strategies, to

introduce the use of the technology as crucial means for the evaluation and the recovery of post-

coma patients and disorders of consciousness, either due to a stroke or a traumatic brain injury,

and to propose a new hypothesis of integration between assistive technology-based devices and

Stasolla et al. AT and VR in Post-coma

virtual reality setups to improve clinical conditions of post-coma

patients diagnosed with disorders of consciousness.

ASSESSMENT STRATEGIES

Basic assessment tools for identifying with certitude whether

a patient is in a vegetative state or in a minimally conscious

state commonly include behavioral scales, neuropsychological

evaluation, neuroimaging techniques, and behavioral data based

on learning setups (Ponsford et al., 2014; Kim et al., 2022;

Ngadimon et al., 2022). Frequently, more than one tool

and/or strategy are used for the evaluation (Lancioni et al.,

2017). Behavioral scales probably represent the most adopted

approach (Pistoia et al., 2013). An illustrative example is

constituted by the JFK Coma Recovery Scale-Revised (Giacino

et al., 2004) to determine the patient’s responsiveness on

communicative, sensorial, orientation, and motor levels of

functioning. Procedural diﬃculties may arise whenever the

patient does not have head/hand control in his/her behavioral

repertoire consistent with the scale’ s requests or is unable to

understand verbal instructions (Bosco et al., 2010).

Neuropsychological procedures including event-related

potentials such as P300 and/or mismatch negativity are also used

to assess the responsiveness of patients with severe disorders of

consciousness (Lancioni et al., 2009b, 2011). Thus, empirical

evidences of P300 or mismatch negativity are usually viewed

as basic signs of awareness or consciousness with meaningful

implications for the recovery process (Estraneo et al., 2022;

Pruvost-Robieux et al., 2022). Recent reviews (Pan et al., 2021;

Aubinet et al., 2022) critically discussed the use of event-related

potentials and emphasized the strengths to use multiple measures

to enhance the signiﬁcance of the ﬁndings (Calabrò et al., 2021).

Neuroimaging techniques (e.g., functional magnetic

resonance imaging, fMRI) may be a reliable tool to identify

potential capacities or skills even in cases of minimal or

apparently absent responsiveness (Drayson, 2014:Kirsch et al.,

2017; Corsi et al., 2020). In fact, the use of those techniques can

strongly help researchers to catch relevant diagnostic outcomes

(Kirsch et al., 2017). Nevertheless, their use is still diﬃcult and

the application may pose serious methodological challenges

in a wide part of medical or rehabilitative settings due to the

assessment of speciﬁc stimuli and/or the comprehension of

verbal instructions (Schwarzbauer and Schafer, 2011). A valid

alternative to the fMRI is represented by the positron emission

tomography (Briand et al., 2020).

Behavioral data based on learning principles may refer to

two diﬀerent approaches. On one hand, based on classical

paradigm they include the capacity of the patient to positively

associate pairs of stimuli (Ricchi et al., 2022). On the other hand,

based on the operant paradigm, they consist on the capacity

of the patient to correctly match a behavioral response with

an environmental consequence (Lee et al., 2021). Empirical

evidences of learning between both approaches (i.e., a correct

association between events or its positive achievement) may

be considered as a non-reﬂective response and suggest a

diagnosis of minimally conscious state (Lancioni et al., 2008b).

Those strategies may be acknowledged as clinically relevant for

those patients who have a minimal behavioral repertoire (e.g.,

eyelid or lip movements) and may pose practical problems

with the JFK Coma Recovery Scale accordingly. Furthermore,

learning principles based on the operant paradigm may be very

useful to introduce a technological-aided program focused on

promoting the participant’s active role, constructive engagement,

and positive participation (Stasolla et al., 2015).

REHABILITATIVE STRATEGIES

Intervention solutions for persons with disorders of

consciousness may include diﬀerent forms of environmental

stimulation, deep brain stimulation, transcranial magnetic

stimulation, brain computer interfaces, and learning-based

programs or technological-mediated options (Lancioni

et al., 2011, 2014a; Kim et al., 2012; Pistoia et al., 2013). An

environmental stimulation program is typically delivered by

the therapist/professional in charge of the patient. In its basic

form it includes the presentation of daily stimulus events such as

familiar music and/or verbal inputs during speciﬁc intervals of

time. In a more sophisticated form, it may involve speciﬁc daily

sessions with an intensive multi-sensorial intervention combined

with a verbal and a physical guide of relevant events provided by

the therapist (Lancioni et al., 2010a, 2014b). Although the basic

form is less eﬀective with regard to the improvement on the

patient’s level of alert and positive involvement in the context,

the intensive form of environmental stimulation is more likely to

have beneﬁcial eﬀects on the participant’s level of attention and

participation (Pape et al., 2015).

Transcranial magnetic stimulation and deep brain stimulation

are considered common approaches within this speciﬁc

framework whose implementation does not require any speciﬁc

participation by the patient exposed to such strategy. Thus, while

evidence-based support is available in the literature for that

approach, studies suggest caution for its implementation because

the eﬀects on the patient’s awareness and/or consciousness are

mixed with regard to the amplitude and clinical signiﬁcance

(Kulyk, 2019).

Brain computer interfaces are systems devoted to measure

brain activity and convert such activity into artiﬁcial outputs

that restore, replace, enhance or support natural outputs of the

Central Nervous System. Accordingly, such strategy is expected

to modify the ongoing interactions between the Central Nervous

System and its external or internal environment. Diﬀerent

techniques may be included to measure brain activity for brain

computer interfaces. The most frequent method is represented

by electrical signals detected through electrodes ﬁxed invasively

or non-invasively on the surface of the cortex or the scalp.

Additionally, a metabolic measure may be recorded through

fMRI (Lancioni et al., 2015).

Learning-based strategy are widely diﬀerent from the

aforementioned detailed strategies. In fact, that approach

emphasizes the participant’s active role, constructive engagement,

and social interactions, mediated by the technology. That is,

the strategy is largely designed to monitor the participant’s

Frontiers in Psychology | www.frontiersin.org 2July 2022 | Volume 13 | Article 905811

Stasolla et al. AT and VR in Post-coma

behavioral repertoire and modify it through the manipulation

of environmental consequences to ensure the participant with

an independent access to positive stimulation (Stasolla et al.,

2022). Those programs are aimed at fostering the participant’s

self-determination and reducing either caregivers or families’

burden accordingly (Savoia et al., 2021). Recently, the Covid-

19 pandemic outlined the development of the new technologies

with an emphasis on virtual reality setups and telerehabilitation

strategies to supervise patients remotely (Caprì et al., 2021;

Momsen et al., 2022).

VIRTUAL REALITY

Virtual reality (VR), and augmented reality (AR) setups, have

currently been adopted as crucial means of new technological-

aided programs in diﬀerent area of public health, namely

(a) assessment, (b) diagnosis, (c) recovery, rehabilitation, and

wellbeing. With regard to rehabilitative programs, VR has been

largely adopted to positively overcome neurological impairments

including neurodevelopmental disorders, and neurodegenerative

diseases (Stasolla, 2021; Stasolla et al., 2021). VR ensures

persons with neurological impairments with sensory experiences,

computer-mediated in artiﬁcial environments, enhancing virtual

interactions similarly to the real life. AR, as part of VR,

emphasizes an interaction in a physical condition, diﬀerently

from the artiﬁcial context provided by VR. That is, VR usually

requires the use of speciﬁc headsets, which may not be easily

wearable for individuals with neurological disorders. Conversely,

AR may be viewed as easier to use because it refers to I-PAD,

tablets, and smartphones, which are more suitable to the real

world (Bekkers et al., 2020; Held et al., 2020; Levin and Demers,

2021). Although widely used in patients with disorders of

consciousness (Hinze et al., 2021; Kwok et al., 2021), to the best

of our knowledge it has never been used in patients with acquired

brain injuries, history of coma and post-coma outcomes, except

for the contribution of Maggio et al. (2020). Although it

may be considered as ethically controversial and questionable,

such hypothesis undoubtedly merits to be empirically tested,

eventually integrated with an assistive technology device, for both

assessment and recovery goals.

DISCUSSION

Two ending conclusions may be putted forward on assessment

strategies. First, using suitable corrections or supplements,

behavioral scales may be considered as a practically signiﬁcant

solution to surpass the limitations of the scale and improve

diagnostic accuracy accordingly. The aforementioned learning

setups may be viewed as suitable issues in this regard

(Lancioni et al., 2007a,b). Second, repeating a combined

assessment between two or more strategies (i.e., behavioral

scales, neuropsychological approach, and/or behavioral data),

one may argue that the risk of individual’s ﬂuctuations and/or

misdiagnosis might be proﬁtably prevented (Pistoia and Sarà,

2012). Among new technologies, Hyun et al. (2021) proposed a

virtual reality technology-based quantitative assessment method

combined with an eye-tracking system to minimize misdiagnosis

of a patient’s eye movements, such as visual startle, visual ﬁxation,

and visual pursuit. Twenty healthy patients and ﬁve chronic

patients in a vegetative state were systematically compared. Three

stimuli were presented and visual responses data were recorded

to identify valid and accurate responses to each stimulus. The

system deﬁned three of the chronic patients as showing visual

ﬁxation, undetectable through clinical assessment beforehand.

Lech et al. (2021) proposed the term “Cyber-Eye” to include

the emerging cognitive applications of eye-tracking interfaces for

neuroscience research, clinical practice, and biomedical industry.

The perspective paper suggested a brain computer interface to

become less invasive, less dependent on brain activities, and more

applicable as the Cyber-Eye technologies continue to develop.

Two ending considerations may be formulated on

rehabilitative strategies. First, additional data are mandatory to

accurately identify the impact of the diﬀerent strategies and their

reliability over the time and across patients. Second, systematic

comparisons between procedures may be fundamental to

determine the eﬀects on a number of dimensions such as

active role and positive participation (Lancioni et al., 2017).

Kujawa et al. (2022) recently investigated the outcomes of an

oculomotor training course aimed at the therapy of visual-spatial

functions. Five patients with brain damage who were unable to

communicate verbally or motorically, diagnosed between the

vegetative state and the emergence from the minimally conscious

state were enrolled. Over a 6-week period, the participants

underwent to solved tasks associated with recognizing objects,

size perception, color perception, perception of objects structure

such as letters, detecting diﬀerences between images and

assembling image components into the complete image with

the use of an eye tracker. Findings evidenced the eﬀectiveness

of the oculomotor training based on a longer duration of

the work with the eye-tracker to improve visual-spatial

functions. Sanz et al. (2021) demonstrated clinical relevance

and translational potential in both diagnosis and prognosis of

post-coma patients with disorders of consciousness. Magnetic

resonance imaging and high-density electroencephalography

provided measurements of brain connectivity between functional

networks, assessment of language functions, detection of covert

consciousness, and prognostic markers of recovery. Positrons

emission tomography could identify patients with preserved

brain metabolism despite clinical unresponsiveness and could

measure glucose consumption rates in targeted brain regions.

Such techniques were considered encouraging and promising for

both assessment and recovery purposes in clinical settings.

Finally, our new hypothesis of the integration between

assistive technology-based devices and virtual reality setups may

be interesting practical implications and may be adopted for both

assessment and recovery purposes. For example, it may enable

post-coma patients with an independent access to immersive

virtual environments similar to real life. In this regard it may

be viewed as a basic option of scaﬀolding (Dicé et al., 2018).

Otherwise, one may argue that it may constitute a further form

of constructive engagement and favorable occupation (Stasolla

et al., 2014a) and/or of psychological wellbeing (Freda et al.,

2019). Its implementation may be helpful for communicative

Frontiers in Psychology | www.frontiersin.org 3July 2022 | Volume 13 | Article 905811

Stasolla et al. AT and VR in Post-coma

purposes and/or challenging behaviors and the clinical relevance

may be evaluated through social validation procedures (Lancioni

et al., 2009a; Chiapparino et al., 2011; Stasolla et al., 2014b,

2017). For example, one may envisage the implementation

in clinical settings of a virtual reality setup activated by an

assistive technology-based device. Strategies of telerehabilitation

(Zucchella et al., 2018; Raso et al., 2021) might be implemented.

Moreover, cognitive and motor rehabilitation solutions may be

embedded (Maggio et al., 2020; Daibert-Nido et al., 2021).

Future research perspectives within this framework should

deal with the following topics: (a) an extension to new

technological solutions combined to virtual reality-based setups

to investigate the assessment and rehabilitation purposes of post-

coma patients with disorders of consciousness, (b) diﬀerentiate

between traumatic brain injuries, stroke, and viral causes of the

coma, (c) integrate a multi-componential approach which should

include behavioral scales with neuropsychological strategies,

electrophysiological measures (e.g., event-related potentials), and

behavioral data with the mediation of assistive technology-

based devices and virtual reality setups to enhance cognitive,

communicative, emotional, and motor skills of post-coma

individuals with disorders of consciousness in both clinical and

home-based settings.

Furthermore, the sustainability of such approach with

regard to (1) its costs, (2) human resources, and (3)

technological solutions available (e.g., mobile devices, wearable

devices, computer-based options) should be investigated.

Additionally, the inclusion in medical or rehabilitative

centers should be exhaustively addressed. For instance,

Bhattacharya and Pradana (in press) evaluated the literacy

process in a three-year old child with Rett syndrome

and signiﬁcant disabilities. Two diﬀerent modalities were

considered, namely (a) corporal, and (b) oral. It would

probably be interesting to transfer such approach to post-

coma individuals with extensive motor disabilities and lack

of speech.

AUTHOR CONTRIBUTIONS

FS conceived and wrote the paper. LV and MC edited and revised

the manuscript. All authors made a substantial contribution to

the article.

REFERENCES

Aubinet, C., Chatelle, C., Gosseries, O., Carrière, M., Laureys, S., and

Majerus, S. (2022). Residual implicit and explicit language abilities

in patients with disorders of consciousness: A systematic review.

Neurosci. Biobehav. Rev. 132, 391–409. doi: 10.1016/j.neubiorev.2021.1

2.001

Bekkers, E. M. J., Mirelman, A., Alcock, L., Rochester, L., Nieuwhof, F.,

Bloem, B. R., et al. (2020). Do patients with Parkinson’s disease with

freezing of gait respond diﬀerently than those without to treadmill training

augmented by virtual reality? Neurorehabil. Neural Repair 34, 440–449.

doi: 10.1177/1545968320912756

Bhattacharya, U., and Pradana, W. A. (in press). Exploring literacy

engagement in a signiﬁcant disability context. J. Early Childhood Literacy.

doi: 10.1177/14687984221100129

Bosco, A., Lancioni, G. E., Belardinelli, M. O., Singh, N. N., O’Reilly, M. F., and

Sigafoos, J. (2010). Vegetative state: Eﬀorts to curb misdiagnosis. Cogn. Process.

11, 87–90. doi: 10.1007/s10339-009-0355-y

Briand, M. -., Gosseries, O., Staumont, B., Laureys, S., and Thibaut, A.

(2020). Transcutaneous auricular vagal nerve stimulation and disorders of

consciousness: A hypothesis for mechanisms of action. Front. Neurol. 11, 933.

doi: 10.3389/fneur.2020.00933

Calabrò, R. S., Pignolo, L., Müller-Eising, C., and Naro, A. (2021). Pain

perception in disorder of consciousness: A scoping review on current

knowledge, clinical applications, and future perspective. Brain Sci. 11:50665.

doi: 10.3390/brainsci11050665

Caprì, T., Nucita, A., Iannizzotto, G., Stasolla, F., Romano, A., Semino, M., et al.

(2021). Telerehabilitation for improving adaptive skills of children and young

adults with multiple disabilities: A systematic review. Rev. J. Autism Dev.

Disord. 8, 244–252. doi: 10.1007/s40489-020-00214-x

Chiapparino, C., Stasolla, F., de Pace, C., and Lancioni, G. E. (2011). A touch

pad and a scanning keyboard emulator to facilitate writing by a woman with

extensive motor disability. Life Span Disability 14, 45–54.

Corsi, F. M., Gerace, C., and Ricci, M. (2020). Rotigotine eﬀect in prolonged

disturbance of consciousness. Brief report of two cases. Acta Biomed.

91, 132–133. doi: 10.23750/abm.v91i1.7659

Daibert-Nido, M., Pyatova, Y., Cheung, K. G., Reginald, A., Garcia-Giler, E.,

Bouﬀet, E., et al. (2021). An audiovisual 3d-immersive stimulation program

in hemianopia using a connected device. Am. J. Case Rep. 22:e931079.

doi: 10.12659/AJCR.931079

de Tommaso, M., Betti, V., Bocci, T., Bolognini, N., Di Russo, F., Fattapposta, F.,

et al. (2020). Pearls and pitfalls in brain functional analysis by event-related

potentials: A narrative review by the italian psychophysiology and cognitive

neuroscience society on methodological limits and clinical reliability—part I.

Neurol. Sci. 41, 2711–2735. doi: 10.1007/s10072-020-04420-7

Dicé, F., Auricchio, M., Boursier, V., De Luca Picione, R., Santamaria, F., Salerno,

M., et al. (2018). Psychological scaﬀolding for taking charge of Intersex/DSD

conditions. The joint listening settings. [Lo Scaﬀolding psicologico per la presa

in carico delle condizioni Intersex/DSD. I Setting di Ascolto Congiunto]. Psicol.

Della Salute 2018, 129–145. doi: 10.3280/PDS2018-001008

Drayson, Z. (2014). Intentional action and the post-coma patient. Topoi 33, 23–31.

doi: 10.1007/s11245-013-9185-8

Estraneo, A., Magliacano, A., Fiorenza, S., Formisano, R., Grippo, A., Angelakis,

E., et al. (2022). Risk factors for 2-year mortality in patients with prolonged

disorders of consciousness: An international multicentre study. Eur. J. Neurol.

29, 390–399. doi: 10.1111/ene.15143

Formisano, R., Aloisi, M., Iosa, M., Contrada, M., Rizza, F., and Sattin, D. (2018).

A new tool to assess responsiveness in disorders of consciousness (DoC):

A preliminary study on the brief post-coma scale (BPCS). Neurol. Sci. 39,

1651–1656. doi: 10.1007/s10072-018-3466-3

Formisano, R., D’Ippolito, M., Risetti, M., Riccio, A., Caravasso, C. F., Catani, S.,

et al. (2011). Vegetative state, minimally conscious state, akinetic mutism and

parkinsonism as a continuum of recovery from disorders of consciousness: An

exploratory and preliminary study. Funct. Neurol. 26, 15–24.

Freda, M. F., Savarese, L., Dolce, P., and Picione, R. L. (2019). Caregivers’

sensemaking of children’s hereditary angioedema: A semiotic narrative

analysis of the sense of grip on the disease. Front. Psychol. 10:e02609.

doi: 10.3389/fpsyg.2019.02609

Giacino, J. T., Kalmar, K., and Whyte, J. (2004). The JFK coma recovery scale-

revised: Measurement characteristics and diagnostic utility. Arch. Phys. Med.

Rehabil. 85, 2020–2029. doi: 10.1016/j.apmr.2004.02.033

Held, J. P. O., Yu, K., Pyles, C., Veerbeek, J. M., Bork, F., Heining, S. -., et al. (2020).

Augmented reality-based rehabilitation of gait impairments: Case report. JMIR

mHealth uHealth 8:e17804. doi: 10.2196/17804

Hinze, J., Röder, A., Menzie, N., Müller, U., Domschke, K., Riemenschneider,

M., et al. (2021). Spider phobia: Neural networks informing diagnosis and

(Virtual/Augmented reality-based) cognitive behavioral Psychotherapy—A

narrative review. Front. Psychiatry 12:e704174. doi: 10.3389/fpsyt.2021.704174

Hyun, C., Kim, K., Lee, S., Lee, H. H., and Lee, J. (2021). Quantitative evaluation of

the consciousness level of patients in a vegetative state using virtual reality and

Frontiers in Psychology | www.frontiersin.org 4July 2022 | Volume 13 | Article 905811

Stasolla et al. AT and VR in Post-coma

an eye-tracking system: A single-case experimental design study. Neuropsychol.

Rehabil. 6, 1–8. doi: 10.1080/09602011.2021.1974496

Kim, E. J., Park, J. M., Kim, W. H., Lee, K. L., Kim, H. N., Lee, K. E., et al. (2012).

A learning set up for detecting minimally conscious state (MCS). Ann. Rehabil.

Med. 36, 428–431. doi: 10.5535/arm.2012.36.3.428

Kim, J., Comuzzi, M., Dumas, M., Maggi, F. M., and Teinemaa, I. (2022). Encoding

resource experience for predictive process monitoring. Decis. Support Syst.

153:e113669. doi: 10.1016/j.dss.2021.113669

Kirsch, M., Guldenmund, P., Ali Bahri, M., Demertzi, A., Baquero, K., Heine,

L., et al. (2017). Sedation of patients with disorders of consciousness during

neuroimaging: Eﬀects on resting state functional brain connectivity. Anesth.

Analg. 124, 588–598. doi: 10.1213/ANE.0000000000001721

Kujawa, K., Zurek, A., Goraczko, A., Olejniczak, R., and Zurek, G. (2022).

Implementing new technologies to improve Visual–Spatial functions in

patients with impaired consciousness. Int. J. Environ. Res. Public Health

19:e3081. doi: 10.3390/ijerph19053081

Kulyk, A. V. (2019). The system of diﬀerentiated medical neurorehabilitation in

the traumatic post-coma conditions of long-term disorders of consciousness.

Acta Neuropsychol. 17, 9–24. doi: 10.5604/01.3001.0013.1405

Kwok, S. C., Xu, X., Duan, W., Wang, X., Tang, Y., All,é, M. C., et al. (2021).

Autobiographical and episodic memory deﬁcits in schizophrenia: A narrative

review and proposed agenda for research. Clin. Psychol. Rev. 83:e101956.

doi: 10.1016/j.cpr.2020.101956

Lancioni, G., O’Reilly, M., Singh, N., Buonocunto, F., Sacco, V., Colonna, F.,

et al. (2009b). Evaluation of technology-assisted learning setups for undertaking

assessment and providing intervention to persons with a diagnosis of vegetative

state. Dev. Neurorehabil. 12, 411–420. doi: 10.3109/17518420903200581

Lancioni, G., Singh, N., O’Reilly, M., Olivetti, M., De Tommaso, M., Navarro,

J., et al. (2011). A learning assessment procedure as a test supplement for

monitoring progress with two post-coma persons with a diagnosis of vegetative

state. Dev. Neurorehabil. 14, 358–365. doi: 10.3109/17518423.2011.605076

Lancioni, G., Singh, N., O’Reilly, M., Signorino, M., Alberti, G., Scigliuzzo, F., et al.

(2010b). Adapting a computer-assisted program to help a post-coma man with

extensive multiple disabilities choose stimulus events. Dev. Neurorehabil. 13,

433–439. doi: 10.3109/17518423.2010.519761

Lancioni, G. E., Belardinelli, M. O., Chiapparino, C., Angelillo, M. T., Stasolla,

F., Singh, N. N., et al. (2008a). Learning in post-coma persons with profound

multiple disabilities: Two case evaluations. J. Dev. Phys. Disabil. 20, 209–216.

doi: 10.1007/s10882-008-9105-9

Lancioni, G. E., Belardinelli, M. O., Stasolla, F., Singh, N. N., O’Reilly, M. F.,

Sigafoos, J., et al. (2008b). Promoting engagement, requests and choice by a

man with post-coma pervasive motor impairment and minimally conscious

state through a technology-based program. J. Dev. Phys. Disabil. 20, 379–388.

doi: 10.1007/s10882-008-9104-x

Lancioni, G. E., Bosco, A., Belardinelli, M. O., Singh, N. N., O’Reilly, M.

F., and Sigafoos, J. (2010a). An overview of intervention options for

promoting adaptive behavior of persons with acquired brain injury

and minimally conscious state. Res. Dev. Disabil. 31, 1121–1134.

doi: 10.1016/j.ridd.2010.06.019

Lancioni, G. E., Bosco, A., Belardinelli, M. O., Singh, N. N., O’Reilly, M. F.,

Sigafoos, J., et al. (2014a). Technology-based intervention programs to promote

stimulation control and communication in post-coma persons with diﬀerent

levels of disability. Front. Hum. Neurosci. 8:48. doi: 10.3389/fnhum.2014.00048

Lancioni, G. E., Bosco, A., O’Reilly, M. F., Sigafoos, J., and Belardinelli,

M. O. (2017). Assessment and intervention with patients with severe

disorders of consciousness. Adv. Neurodev. Disord. 1, 196–202.

doi: 10.1007/s41252-017-0025-5

Lancioni, G. E., Singh, N. N., O’Reilly, M. F., Green, V. A., Buonocunto,

F., Sacco, V., et al. (2014b). Microswitch-aided programs with

contingent stimulation versus general stimulation programs for post-

coma persons with multiple disabilities. Dev. Neurorehabil. 17, 251–258.

doi: 10.3109/17518423.2013.793751

Lancioni, G. E., Singh, N. N., O’Reilly, M. F., Sigafoos, J., Chiapparino, C., Stasolla,

F., et al. (2007a). Using an optic sensor and a scanning keyboard emulator

to facilitate writing by persons with pervasive motor disabilities. J. Dev. Phys.

Disabil. 19, 593–603. doi: 10.1007/s10882-007-9073-5

Lancioni, G. E., Singh, N. N., O’Reilly, M. F., Sigafoos, J., Chiapparino, C., Stasolla,

F., et al. (2007b). Enabling a young man with minimal motor behavior to

manage independently his leisure television engagement. Percept. Mot. Skills

105, 47–54. doi: 10.2466/pms.105.1.47-54

Lancioni, G. E., Singh, N. N., O’Reilly, M. F., Sigafoos, J., Green, V., Chiapparino,

C., et al. (2009a). A voice-detecting sensor and a scanning keyboard emulator

to support word writing by two boys with extensive motor disabilities. Res. Dev.

Disabil. 30, 203–209. doi: 10.1016/j.ridd.2008.03.001

Lancioni, G. E., Singh, N. N., O’reilly, M. F., Sigafoos, J., Ricciuti, R. A., Trignani,

R., et al. (2015). Extending technology-aided leisure and communication

programs to persons with spinal cord injury and post-coma multiple

disabilities. Disability Rehabilit. 10, 32–37. doi: 10.3109/17483107.2013.860635

Lech, M., Czyzewski, A., and Kucewicz, M. T. (2021). Cybereye: New eye-tracking

interfaces for assessment and modulation of cognitive functions beyond the

brain. Sensors 21:e7605. doi: 10.3390/s21227605

Lee, I., Jung, W., Lee, Y., Wallraven, C., and Chae, Y. (2021). Operant and classical

learning principles underlying mind–body interaction in pain modulation: A

pilot fMRI study. Sci. Rep. 11:81134. doi: 10.1038/s41598-021-81134-6

Levin, M. F., and Demers, M. (2021). Motor learning in neurological rehabilitation.

Disabil. Rehabil. 43, 3445–3453. doi: 10.1080/09638288.2020.1752317

Maggio, M. G., Naro, A., La Rosa, G., Cambria, A., Lauria, P., Billeri, L., et al.

(2020). Virtual reality based cognitive rehabilitation in minimally conscious

state: A case report with EEG ﬁndings and systematic literature review. Brain

Sci. 10, 1–16. doi: 10.3390/brainsci10070414

Momsen, A. H., Ørtenblad, L., and Maribo, T. (2022). Eﬀective rehabilitation

interventions and participation among people with multiple sclerosis:

An overview of reviews. Ann. Phys. Rehabil. Med. 65:e101529.

doi: 10.1016/j.rehab.2021.101529

Ngadimon, I. W., Aledo-Serrano, A., Arulsamy, A., Mohan, D., Khoo, C. S.,

Cheong, W. L., et al. (2022). An interplay between post-traumatic epilepsy and

associated cognitive decline: A systematic review. Front. Neurol. 13:e827571.

doi: 10.3389/fneur.2022.827571

Pan, J., Wu, J., Liu, J., Wu, J., and Wang, F. (2021). A systematic review of sleep

in patients with disorders of consciousness: From diagnosis to prognosis. Brain

Sci. 11:e1072. doi: 10.3390/brainsci11081072

Pape, T. L., Rosenow, J. M., Steiner, M., Parrish, T., Guernon, A., Harton, B.,

et al. (2015). Placebo-controlled trial of familiar auditory sensory training for

acute severe traumatic brain injury. Neurorehabil. Neural Repair 29, 537–547.

doi: 10.1177/1545968314554626

Pistoia, F., Sacco, S., Palmirotta, R., Onorati, P., Carolei, A., and Sar,à, M. (2008).

Mismatch of neurophysiological ﬁndings in partial recovery of consciousness:

A case report. Brain Injury 22, 633–637. doi: 10.1080/02699050802189693

Pistoia, F., Sacco, S., Sarà, M., and Carolei, A. (2013). The perception of

pain and its management in disorders of consciousness topical collection

on psychiatric management of pain. Curr. Pain Headache Rep. 17:374.

doi: 10.1007/s11916-013-0374-3

Pistoia, F., and Sarà, M. (2012). Is there a cartesian renaissance of the mind or

is it time for a new taxonomy for low responsive states? J. Neurotrauma 29,

2328–2331. doi: 10.1089/neu.2009.1257

Ponsford, J., Janzen, S., McIntyre, A., Bayley, M., Velikonja, D., and Tate, R. (2014).

INCOG recommendations for management of cognition following traumatic

brain injury, part I: Posttraumatic amnesia/delirium. J. Head Trauma Rehabilit.

29, 307–320. doi: 10.1097/HTR.0000000000000074

Pruvost-Robieux, E., Marchi, A., Martinelli, I., Bouchereau, E., and

Gavaret, M. (2022). Evoked and event-related potentials as biomarkers

of consciousness state and recovery. J. Clin. Neurophysiol. 39, 22–31.

doi: 10.1097/WNP.0000000000000762

Raso, M. G., Arcuri, F., Liperoti, S., Mercurio, L., Mauro, A., Cusato, F., et al.

(2021). Telemonitoring of patients with chronic traumatic brain injury: A pilot

study. Front. Neurol. 12:e598777. doi: 10.3389/fneur.2021.598777

Ricchi, I., Tarun, A., Maretic, H. P., Frossard, P., and Van De Ville, D. (2022).

Dynamics of functional network organization through graphmixturelearning.

Neuroimage 252:e119037. doi: 10.1016/j.neuroimage.2022.119037

Sanz, L. R. D., Thibaut, A., Edlow, B. L., Laureys, S., and Gosseries, O. (2021).

Update on neuroimaging in disorders of consciousness. Curr. Opin. Neurol. 34,

488–496. doi: 10.1097/WCO.0000000000000951

Savoia, S. E., Stern, E. H., Salardini, A., and Dorai, C. (2021). Usability and

functional validation of artiﬁcial intelligence-enabled technology services in

dementia care using a controlled trial in clinical practice. Alzheimer Dementia

17:e056547. doi: 10.1002/alz.056547

Frontiers in Psychology | www.frontiersin.org 5July 2022 | Volume 13 | Article 905811

Stasolla et al. AT and VR in Post-coma

Schwarzbauer, C., and Schafer, B. (2011). FMRI in disorders of consciousness:

Future diagnostic opportunities, methodological and legal challenges. Cortex

47, 1243–1245. doi: 10.1016/j.cortex.2011.04.014

Stasolla, F. (2021). Virtual reality and wearable technologies to support adaptive

responding of children and adolescents with neurodevelopmental disorders:

A critical comment and new perspectives. Front. Psychol. 12:e720626.

doi: 10.3389/fpsyg.2021.720626

Stasolla, F., Caﬀò, A. O., Bottiroli, S., and Ciarmoli, D. (2022). An

assistive technology program for enabling ﬁve adolescents emerging

from a minimally conscious state to engage in communication,

occupation, and leisure opportunities. Dev. Neurorehabil. 25, 193–204.

doi: 10.1080/17518423.2021.2011457

Stasolla, F., Caﬀò, A. O., Damiani, R., Perilli, V., Di Leone, A., and Albano, V.

(2015). Assistive technology-based programs to promote communication and

leisure activities by three children emerged from a minimal conscious state.

Cogn. Process. 16, 69–78. doi: 10.1007/s10339-014-0625-1

Stasolla, F., Caﬀò, A. O., Perilli, V., Boccasini, A., Stella, A., Damiani, R.,

et al. (2017). a microswitch-based program for promoting initial ambulation

responses: An evaluation with two girls with multiple disabilities. J. Appl. Behav.

Anal. 50, 345–356. doi: 10.1002/jaba.374

Stasolla, F., Damiani, R., and Caﬀò, A. O. (2014a). Promoting constructive

engagement by two boys with autism spectrum disorders and high functioning

through behavioral interventions. Res. Autism Spectr. Disord. 8, 376–380.

doi: 10.1016/j.rasd.2013.12.020

Stasolla, F., and De Pace, C. (2014). Assistive technology to promote leisure

and constructive engagement by two boys emerged from a minimal

conscious state. NeuroRehabilitation 35, 253–259. doi: 10.3233/NRE-14

1112

Stasolla, F., Matamala-Gomez, M., Bernini, S., Caﬀò, A. O., and Bottiroli,

S. (2021). Virtual reality as a technological-aided solution to support

communication in persons with neurodegenerative diseases and acquired

brain injury during COVID-19 pandemic. Front. Public Health 8:e635426.

doi: 10.3389/fpubh.2020.635426

Stasolla, F., Perilli, V., Damiani, R., Caﬀò, A. O., Di Leone, A., Albano,

V., et al. (2014b). A microswitch-cluster program to enhance

object manipulation and to reduce hand mouthing by three boys

with autism spectrum disorders and intellectual disabilities. Res.

Autism Spectr. Disord. 8, 1071–1078. doi: 10.1016/j.rasd.2014.0

5.016

Zucchella, C., Di Santis, M., Ciccone, B., Pelella, M., Scappaticci, M., Badalassi, G.,

et al. (2018). Is telemonitoring useful for supporting persons with consciousness

disorders and caregivers? A preliminary observational study in a real-life

population. J. Telemed. Telecare 24, 56–62. doi: 10.1177/1357633X16673273

Conﬂict of Interest: The authors declare that the research was conducted in the

absence of any commercial or ﬁnancial relationships that could be construed as a

potential conﬂict of interest.

Publisher’s Note: All claims expressedin this article are solely those of the authors

and do not necessarily represent those of their aﬃliated organizations, or those of

the publisher, the editors and the reviewers. Any product that may be evaluated in

this article, or claim that may be made by its manufacturer, is not guaranteed or

endorsed by the publisher.

distributed under the terms of the Creative Commons Attribution License (CC BY).

The use, distribution or reproduction in other forums is permitted, provided the

original author(s) and the copyright owner(s) are credited and that the original

publication in this journal is cited, in accordance with accepted academic practice.

No use, distribution or reproduction is permitted which does not comply with these

terms.

Frontiers in Psychology | www.frontiersin.org 6July 2022 | Volume 13 | Article 905811

fpsyg-13-976661 September 1, 2022 Time: 17:28 # 1

TYPE Study Protocol

PUBLISHED 01 September 2022

DOI 10.3389/fpsyg.2022.976661

OPEN ACCESS

EDITED BY

Casandra Isabel Montoro,

University of Jaén, Spain

REVIEWED BY

Andrea Polari,

Orygen Youth Health, Australia

Emily Bilek,

University of Michigan, United States

*CORRESPONDENCE

Trinidad Peláez

mtrinidad.pelaez@sjd.es

SPECIALTY SECTION

This article was submitted to

Psychology for Clinical Settings,

a section of the journal

Frontiers in Psychology

RECEIVED 23 June 2022

ACCEPTED 27 July 2022

PUBLISHED 01 September 2022

CITATION

Peláez T, López-Carrillero R,

Ferrer-Quintero M, Ochoa S and

Osma J (2022) Application of the

uniﬁed protocol for the transdiagnostic

treatment of comorbid emotional

disorders in patients with ultra-high

risk of developing psychosis:

A randomized trial study protocol.

Front. Psychol. 13:976661.

doi: 10.3389/fpsyg.2022.976661

Ferrer-Quintero, Ochoa and Osma.

This is an open-access article

distributed under the terms of the

Creative Commons Attribution License

(CC BY). The use, distribution or

reproduction in other forums is

permitted, provided the original

author(s) and the copyright owner(s)

are credited and that the original

publication in this journal is cited, in

accordance with accepted academic

practice. No use, distribution or

reproduction is permitted which does

not comply with these terms.

Application of the uniﬁed

protocol for the transdiagnostic

treatment of comorbid

emotional disorders in patients

with ultra-high risk of

developing psychosis: A

randomized trial study protocol

Trinidad Peláez1,2*, Raquel López-Carrillero1,2,

Marta Ferrer-Quintero1,2, Susana Ochoa1,2 and Jorge Osma3,4

1Parc Sanitari Sant Joan de Déu, Barcelona, Spain, 2Investigación Biomédica en Red de Salud

Mental (CIBERSAM) Instituto de Salud Carlos III, Madrid, Spain, 3Departamento de Psicología y

Sociología, Universidad de Zaragoza, Zaragoza, Spain, 4Instituto de Investigación Sanitaria de

Aragón, Zaragoza, Spain

Background: Cognitive Behavioral Therapy is delivered in most of the early

intervention services for psychosis in different countries around the world.

This approach has been demonstrated to be effective in decreasing or at

least delaying the onset of psychosis. However, none of them directly affect

the comorbidity of these types of patients that is often the main cause of

distress and dysfunctionality. The Uniﬁed Protocol for the Transdiagnostic

Treatment of Emotional Disorders (UP) is a psychological intervention that

combines cognitive-behavioral and third-generation techniques that address

emotional dysregulation as an underlying mechanism that these disorders

have in common. The application of this intervention could improve the

comorbid emotional symptoms of these patients.

Materials and methods: The study is a randomized controlled trial in which

one group receives immediate UP plus standard intervention and the other is

placed on a waiting list to receive UP 7 months later, in addition to standard

care in one of our early psychosis programs. The sample will be 42 patients

with UHR for psychosis with comorbid emotional symptoms. The assessment

is performed at baseline, at the end of treatment, and at 3-months’ follow-up,

and includes: general psychopathology, anxiety and depression, positive and

negative emotions, emotional dysregulation, personality, functionality, quality

of life, cognitive distortions, insight, and satisfaction with the UP intervention.

Discussion: This will be the ﬁrst study of the efﬁcacy, acceptability, and

viability of the UP in a sample of young adults with UHR. The results of this

Frontiers in Psychology 01 frontiersin.org

fpsyg-13-976661 September 1, 2022 Time: 17:28 # 2

Peláez et al. 10.3389/fpsyg.2022.976661

study may have clinical implications, contributing to improving the model of

care for young people who consult for underlying psychotic, anxiety, and/or

depressive symptoms that can lead to high distress and dysfunctionality.

Clinical trial registration: [https://clinicaltrials.gov/], identiﬁer

[NCT04929938].

KEYWORDS

anxiety disorders, depressive disorders, ultra high risk for psychosis, uniﬁed protocol,

transdiagnostic, psychotic disorders, emotional focused therapy

Introduction

In recent decades one of the main topics of research

in mental health has been psychotic disorders and, more

speciﬁcally, their ﬁrst clinical manifestations. Cumulative

research has found evidence highly suggestive of a relationship

between the duration of untreated psychosis (DUP) and both

short- and long-term prognosis, of a relationship between longer

DUP and more severe positive symptoms, and of more severe

negative symptoms and lower chance of remission at follow-

up (Howes et al.,2021). For this reason, early detection and

intervention programs in psychosis have been disseminated in

several countries around the world (Taylor,2016).

In this sense, most psychotic disorders do not appear

abruptly. Rather, the onset is preceded by subclinical

manifestations of symptoms that gradually increase in

frequency and intensity (Shah et al.,2017). This period has

been deﬁned and operationalized as “Ultra High Risk” (UHR;

Yung et al.,2005). Subsequently, numerous studies have been

carried out with the main objective of determining the rates

of transition to psychosis as the main measure of outcome

(Fusar-Poli et al.,2012). Research has indicated that people

that meet UHR criteria are at increased risk of developing

psychosis in the short term, and this risk increases over time

(Salazar de Pablo et al.,2021). Despite this association, the UHR

paradigm is not without criticism. Some previous literature has

argued that the “UHR for psychosis” label can be stigmatizing.

First, because setting the goal of preventing transition to

psychosis/schizophrenia could create a paradox or a self-

fulﬁlling prophecy of failure. Second, because initial research

led to clinical trials prescribing antipsychotic medication in the

UHR/CHR population (Van Os and Guloksuz,2017).

There are recent studies that relate the stigma associated

with the diagnosis of UHR with a worse prognosis (Colizzi

et al.,2020) and stimulated negative stereotypes (Woodberry

et al.,2021). For this reason, alternative terms like pre-diagnosis

stage’ (PDS), potential of developing a mental illness (PDMI),

and disposition for developing a mental illness (DDMI) have

Abbreviations: UP, Uniﬁed Protocol; UHR, Ultra-high Risk (for psychosis);

TAU, Treatment as Usual; ED, Emotional disorders; WL, Waiting list.

been proposed that generate less discomfort among patients

and families (Polari et al.,2021a). Generally, pharmacological

therapy with antipsychotic medication is not recommended,

and psychological interventions represent a more appropriate

alternative to oﬀer treatment to people at UHR (NICE,

2014;Orygen The National Center of Excellence in Youth,

2016). In a recent meta-analysis, it was found that early

interventions reduced transition rate and attenuated positive

psychotic symptoms at 12 months. In addition, psychological

interventions demonstrated a signiﬁcant reduction in transition

rates compared to pharmacological therapy (Mei et al.,2021).

Further, transition to psychosis is not the only outcome for

people at UHR (Polari et al.,2021b). Some studies have shown

diﬀerent clinical trajectories beyond the transition to psychosis,

such as no transition, chroniﬁcation of attenuated symptoms

or their recurrence (Lin et al.,2015), positive improvement,

moderate impairment, and severe impairment (Allswede et al.,

2020). Another study found 17 diﬀerent trajectories in UHR

patients from complete recovery to transition to psychosis,

including no remission, relapse, and recurrence (Polari et al.,

2018). A recent study observed that 56.8% of patients with UHR

met criteria for a non-psychotic disorder at 6-years’ follow-

up (Rutigliano et al.,2016). Similarly, a sample from Spain

yielded comparable results (Barajas et al.,2019). Persistence or

recurrence of non-psychotic comorbid disorders was associated

with worse overall functioning. At baseline, they found that

70.3% of this sample had some comorbidity with a non-

psychotic disorder (aﬀective disorder 36.5%, anxiety disorder

10.8%, mixed anxiety-depressive disorder 5.4%, and personality

disorder 6.8%). These results showed that although transition to

psychosis may be a frequent outcome of patients at UHR, there

is also a very high risk of developing another psychiatric disorder

(Rutigliano et al.,2016).

Recently, a growing body of literature has suggested that

the classiﬁcation of mental disorders needs to shift from a

categorical model (such as DSM or CIE) to a dimensional one

(Van Os and Guloksuz,2017;McGorry et al.,2018). Under this

paradigm, it is proposed that there is a continuum between a

complete absence of symptoms and severe psychopathology. In

this continuum, patients may exhibit symptoms ranging from

mild to distressing to indicating the need for specialized help

Frontiers in Psychology 02 frontiersin.org

fpsyg-13-976661 September 1, 2022 Time: 17:28 # 3

Peláez et al. 10.3389/fpsyg.2022.976661

(McGorry et al.,2018), including people meeting the criteria

for UHR. This clinical staging and transdiagnostic framework

has led to a broader classiﬁcation of subthreshold risk states

(CHARMS) including the bipolar trait vulnerability group,

the attenuated (hypo)manic symptom group, the moderate

(attenuated) depression group, and the attenuated borderline

personality group (Hartmann et al.,2019).

The development of a dimensional approach has fostered

the creation of diﬀerent treatment alternatives, such as

transdiagnostic models (Rosellini and Brown,2019). These

models are focused on treating the etiological and maintenance

mechanisms shared by diﬀerent mental disorders (Sauer-Zavala

et al.,2017). The Uniﬁed Protocol for the Transdiagnostic

Treatment of Emotional Disorders (UP; Barlow et al.,2017,

2019) focuses on improving emotional regulation through

acceptance, tolerance of intense emotions, and behavior

modiﬁcation (Barlow et al.,2019). To achieve these objectives, it

uses traditional cognitive-behavioral techniques (e.g., cognitive

restructuring or exposure) combined with third-generation

techniques (e.g., mindfulness) (Barlow et al.,2019). This

treatment is indicated for people with diﬃculties in emotional

regulation with or without diagnostic comorbidity. Most

recently, a clinical trial is under way in Spain to test the

eﬃcacy, cost-eﬀectiveness, and implementation characteristics

(acceptability, usability, and utility) of a blended intervention

which will enhance face-to-face treatment by incorporating an

app-based intervention in onsite treatment (Osma et al.,2021).

Current psychological interventions targeting attenuated

psychotic symptoms have proved eﬀective in reducing the rates

of transition to psychosis in the medium term (Morrison et al.,

2004;Van der Gaag et al.,2013b;Ising et al.,2017) and in

reducing the severity of psychotic symptoms (Morrison et al.,

2012) compared with treatment as usual (TAU). However,

further studies are needed to demonstrate more robust results

(Van der Gaag et al.,2013b). Cognitive behavioral therapy

(CBT) manuals designed to treat UHR (French and Morrison,

2004;Van der Gaag et al.,2013a) do not speciﬁcally include

comorbid emotional symptoms (aﬀective or anxiety disorders)

among treatment goals, although it is known that they are

highly prevalent and, in many cases, are the main cause of

dysfunctionality (Rutigliano et al.,2016). In the PACE-Manual-

Writing Group (Nelson and Orygen Youth Health Research

Centre Issuing Body,2012) there is a module that addresses

comorbidity, but in the case of the UP intervention, that is based

on the transdiagnostic approach, all psychotherapy techniques

have been chosen because they are associated with the core

vulnerabilities and processes shared by all emotional disorders

(Sauer-Zavala et al.,2017).

The UP has shown a reduction in symptoms of anxiety

and depression in several mental disorders (such as major

depressive disorder, obsessive-compulsive disorder, and social

anxiety, among others) with a large eﬀect size, an increase in

adaptive emotional regulation strategies paired with a decrease

in maladaptive regulation strategies at a moderate eﬀect size,

and an increase in functioning and quality of life (Sakiris and

Berle,2019;Cassiello-Robbins et al.,2020;Carlucci et al.,2021).

Furthermore, the beneﬁts of the UP seem to be maintained at

6 months’ follow-up for clinical outcomes and at 12–18 months’

follow-up in functioning (Bullis et al.,2014;Osma et al.,

2021). The UP is a standardized and manualized intervention

that can be delivered in individual (Barlow et al.,2017) and

group formats (Osma et al.,2022). The manual consists of 8

treatment modules.

To date, there is no published study that has used the

UP in the treatment of comorbid emotional symptoms in

patients that meet UHR criteria, save for a single case study

using the UP in a person with treatment-resistant schizophrenia

(Grasa and Corripio,2019) with promising results. There were

signiﬁcant decreases between pre- and post-test measures of

anxiety, depressive symptoms, emotional dysregulation, loss

of control, rejection, interference scales, and hallucinations,

measured with the PSYRATS, as well as a signiﬁcant increase

in quality of life. This work aroused interest in applying the UP

to psychotic disorders.

Ultra-high risk (for psychosis) patients often have diﬃculties

in engaging with mental health services (Ben-David et al.,

2019) being this situation an important obstacle to receiving

appropriate treatment. In addition, young people are often

familiar with new technologies (Lupton,2021) and because

of that, we believe that the application of the UP in an

online format may reduce the barriers to accessing treatment

(Osma et al.,2021). We already know that evidence-based

therapies can be administered online without sacriﬁcing their

eﬀectiveness (Andersson et al.,2014;McLaren et al.,2021).

In a recent review, videoconferencing interventions proved to

be reliable and highly acceptable for patients with psychotic

disorders (Santesteban-Echarri et al.,2020). With regard to

emotional disorders, no signiﬁcant diﬀerences were found in

the eﬀectiveness of face-to-face CBT and online CBT. Both of

them were shown to be eﬀective in reducing the symptoms of

anxiety, depression, and stress, as well as in improving quality of

life (Stubbings et al.,2013). The UP has also been shown to be

eﬀective in an online format (Carlucci et al.,2021).

A pilot study is under way to assess the feasibility

and eﬃcacy of group UP to reduce comorbid emotional

symptoms in patients that meet UHR criteria. In the context

of the SARS-CoV-2 pandemic, the group sessions are being

conducted online. Given the evidence cited above, this method

of delivering the intervention would be as eﬀective as in-

person treatment with the added beneﬁt of encouraging the

attendance of people residing in diﬀerent geographic territories

(Singh and Sagar,2022).

Study aims

Given the high level of comorbid anxiety and depression

among individuals meeting UHR criteria, the main objective

Frontiers in Psychology 03 frontiersin.org

fpsyg-13-976661 September 1, 2022 Time: 17:28 # 4

Peláez et al. 10.3389/fpsyg.2022.976661

of our study is to evaluate the eﬃcacy of UP in addition to

TAU as compared to TAU in targeting symptoms of anxiety and

depression in UHR for psychosis patients.

As secondary objectives, changes in attenuated symptoms

of psychosis, transition to psychosis rates, cognitive distortions,

quality of life, metacognition, personality, and psychosocial

functioning, and the satisfaction of participants with the

treatment, are being assessed. Further, assessment will be made

of whether the results are maintained at 3 months’ follow-up.

Diﬀerent clinical trajectories will be analyzed for comparison

with previous studies (Polari et al.,2018).

Materials and methods

Study design

This study is a randomized controlled trial. All patients are

assessed at baseline. Patients are assigned to either treatment

group, one receiving immediate treatment with UP and

treatment as usual, or to a waiting list, only receiving treatment

as usual. All patients are assessed post-treatment and at 3-

months’ follow-up. After the ﬁnal assessment, patients in the

waiting list are oﬀered the UP. A list of random numbers created

for this purpose is used.

Participants

Participants are people who meet criteria of UHR for

psychosis and who have comorbid symptoms of emotional

disorders and are receiving treatment in one of the Early

Pychosis Programs (PIPPEP) in Parc Sanitari Sant Joan de Déu.

The inclusion criteria are (1) age between 18 and 35 years

old, (2) a diagnosis of UHR for psychosis in the last 3 years and

inclusion in our early intervention program, (3) symptoms of a

comorbid emotional disorder, (4) ﬂuent Spanish or Catalan, and

(5) signing the informed consent (IC).

The exclusion criteria are (1) a frank psychotic episode in the

past or in the present, (2) intellectual disability, (3) an organic

disorder that explains current symptomatology.

Measures

The variables to be studied are evaluated using the

instruments described in Supplementary Appendix A.

Data collection

The evaluation is being carried out at 3 time points.

A detailed description of the measures used in each evaluation

is reported in Supplementary Appendix B.

Evaluators have been trained in psychological evaluation

and speciﬁcally in the administration of the CAARMS (Yung

et al.,2005). They also are blind to the condition of

the study to which the participants have been assigned.

In order to ensure internal consistency of the evaluations,

interobserver reliability will be calculated with the Cohen

kappa. At the beginning of each session with the group

receiving UP, two scales are administered, the ODSIS and

the OASIS, to measure the severity of depression and

anxiety experienced during the previous week, in order to

observe ﬂuctuations during treatment. This procedure is

performed following the recommendations of the UP manual

(Barlow et al.,2019).

CAARMS scores are also collected at the time patients begin

treatment in our early psychosis program, prior to entering the

baseline assessment of the present study. Medication changes

and number or TAU sessions will be recorded as a control

variables. Types and dosages of medication will be recorded at

the three time points of assessment.

Interventions

All study participants receive TAU, within our early

intervention program. It includes the following interventions:

psychological therapy (about 20–40 sessions of CBT) as well

as psychiatric treatment (with antidepressants, benzodiazepines

and only when needed antipsychotic medication), social work

intervention (vocational orientation and support), nursing

care (side eﬀects monitoring and healthy habits), individual

cognitive remediation (if necessary), and family therapy. The

number of sessions received in TAU depends on the clinical

status of each patient. The maximum duration of TAU is

5 years. CBT delivered in TAU consists of techniques such

as behavioral experiments, socratic questioning, and some

exposure techniques mainly focused on subthreshold psychotic

symptoms (French and Morrison,2004;Nelson and Orygen

Youth Health Research Centre Issuing Body,2012;Van der

Gaag et al.,2013a). Patients receive weekly or fortnightly

sessions of psychotherapy. UP includes establishment of

the speciﬁc therapeutic aims of each participant, motivation

techniques, emotional psychoeducation, teaching functional

analysis of the emotional experiences, mindfulness techniques,

cognitive ﬂexibility, analysis of emotional behaviors, and

training in opposite behaviors. We also use interoceptive

exposure, which we never use in individual CBT, and we

teach patients to create exposure hierarchies for intense

emotions so that they can follow treatment without the

continuous supervision of a therapist. The UP consists of

15 online group sessions of 2 h each week. The groups

include 5–8 participants. Participants receive an additional

follow-up session 1 and 3 months after the end of the

program. The sessions work on the 8 modules of UP

Frontiers in Psychology 04 frontiersin.org

fpsyg-13-976661 September 1, 2022 Time: 17:28 # 5

Peláez et al. 10.3389/fpsyg.2022.976661

for the transdiagnostic treatment of emotional disorders,

as detailed in the reference manuals (Barlow et al.,2019).

A summary of each session of the UP is detailed in

Supplementary Appendix C.

Fidelity to the UP treatment protocol is guaranteed through

weekly supervision with an accredited therapist. In addition,

the therapists who will perform the UP intervention have

undergone a 20-h training course. The contents of each module

are summarized in an infographic and delivered to patients after

the UP session to improve adherence to the intervention and

acquisition of the techniques.

Once the participants are recruited to the study, they are

randomized into one of the two conditions: TAU +immediate

UP (TAU +imm UP) or TAU +WL (TAU +Waiting

List). The ﬁrst group will receive UP immediately in addition

to the TAU. The second group will do the TAU while

doing the assessments. During this period these patients

act as a control group. Seven months later, these patients

will receive UP, in addition to TAU. Assessments will

also be made at the same time points. This type of

study has been carried out previously (Carl et al.,2014).

The number of sessions of all the services used in TAU

(in both conditions) will be recorded in order to be

taken into account when statistical analyses are made. The

ﬂowchart of the trial and its diﬀerent stages is detailed in

Supplementary Appendix D.

Sample size calculation

Accepting an alpha risk of 0.05 and a beta risk of less than 0.2

in a bilateral contrast, 21 subjects in the PU +TAU group and

21 in the WL +TAU group are needed to detect a diﬀerence

equal to or greater than 1.57 units. The common standard

deviation is assumed to be 1.6. A follow-up loss rate of 20%

has been estimated.

Data analysis

To analyze the improvements in primary and secondary

variables throughout the study, linear mixed model analysis

will be used. This analysis will allow us to study the

main eﬀects of time (pre-test, post-test, 3-month follow-up),

treatment condition (TAU vs. TAU +UP), and number

of sessions received (CBT sessions in TAU, UP sessions,

etc.). We will also calculate interaction eﬀects (e.g., treatment

condition ×time, or treatment condition ×number of

sessions ×time, and the type and dosage of medication)

which will reveal whether the treatment condition and the

number of sessions received interacted with time in the

prediction of changes in study outcomes. In the event that

we observe a signiﬁcant interaction, post hoc analysis will

be conducted. Due to the nature of the present study, we

expect to identify subgroups of patients presenting diﬀering

evolutions in study variables according to the number of

sessions they have received.

The rate of transition to psychosis of patients in each

condition and the CAARMS symptom severity will be

calculated in order to evaluate changes in subthreshold

psychotic symptoms (Morrison et al.,2012). We will also

analyze the diﬀerent clinical trajectories in the two groups

following other previous studies (Polari et al.,2018). If

any participant makes a transition to psychosis during their

participation in the study, they will be excluded (full-

blown psychosis is an exclusion criterion) and their data

will be taken into account for further analysis. Finally,

satisfaction of patients undergoing group treatment will

also be analyzed.

Discussion

To the best of our knowledge, this is the ﬁrst study

to investigate the eﬀectiveness of UP in a sample of young

adults diagnosed with UHR who also have comorbid emotional

symptoms. If the results of this study show that UP is

eﬀective in treating the comorbid symptoms of UHR for

psychosis, this ﬁnding could contribute to expanding the

psychotherapeutic approaches that can be used with young

people presenting with an at-risk mental state. UP may

be complementary and/or an alternative to standard CBT

approaches. This study would need to be replicated with

a bigger sample.

Uniﬁed protocol has been shown to be eﬀective in patients

with a primary diagnosis of emotional disorders, including

cases with comorbidity, according to the systematic review

studies and meta-analyses conducted to date (Sakiris and Berle,

2019;Cassiello-Robbins et al.,2020;Carlucci et al.,2021). We

hypothesize that this intervention could be equally eﬀective in

young patients because there has not been a chroniﬁcation of

their symptoms yet.

The UP contents and the way each emotion regulation

technique is introduced and trained for helps patients to

normalize their emotional symptoms or disorders, because

all people can experience intense emotions and respond

with emotional behaviors. It is positive for all people to

improve their emotional regulation skills. This perspective

focused on training skills can also help to reduce mental

health stigmatization. In addition, UP uses expressions like

“emotional experiences,” “intense emotions,” and “emotion

driven behaviors” instead of other terms like “aggressive

response” or “negative emotions.” All these aspects can help

reduce the rejection of treatment by mental health services,

especially in young people.

Frontiers in Psychology 05 frontiersin.org

fpsyg-13-976661 September 1, 2022 Time: 17:28 # 6

Peláez et al. 10.3389/fpsyg.2022.976661

The telematic group format of the UP could improve

therapeutic adherence in young people, as they are familiar with

new technologies (Lupton,2021). Further, it may encourage

the recruitment of patients residing in remote areas and those

without specialized care resources.

Cognitive behavioral therapy delivered in TAU is an

intervention that has already been shown to be eﬀective in

patients of this type (Van der Gaag et al.,2013a). However,

signiﬁcant diﬀerences are expected in clinical variables in those

patients who additionally receive the intervention with the UP,

mainly in comorbid emotional symptomatology.

Given the high comorbidity of emotional disorders in

patients with UHR (Rutigliano et al.,2016) and the presence

of errors in information processing in both groups of

disorders, such as jumping to conclusion, selective care, and

catastrophization, as well as avoidant behaviors (Livet et al.,

2020), it is likely that these share common transdiagnostic

mechanisms with ED. It may be the case that improving

emotional regulation will have a positive eﬀect on cognitive

biases implicated in the onset and maintenance of both

emotional disorders and symptoms of psychosis. If the

results show positive associations between improved emotional

regulation and improvement in cognitive biases, this would

provide a strong theoretical and clinical basis for oﬀering UP

to people at the UHR of psychosis.

One limitation of the study is the potential diﬃculty in

isolating the impact of UP on outcome measures. This study is a

naturalistic study, that is, it is carried out in the context of public

mental health, which is why it is comparable with what has been

done up to now. Clinically, it would be more ethical to oﬀer

an intervention like UP to all the participants of the study. We

expect that the statistical analyses mentioned above can increase

the robustness of the study results and solve this limitation.

The UP has already been shown to be eﬀective with similar

symptoms (Sakiris and Berle,2019;Cassiello-Robbins et al.,

2020;Carlucci et al.,2021) and this brings us closer to the clinical

reality of mental health services for young people in public

health. Furthermore, this design may make it easier to collect

more samples since this type of patient is not very prevalent.

There are several previous studies that have used this type of

methodology (e.g., Carl et al.,2014). In this sense, this would

be the ﬁrst step toward obtaining preliminary data on whether

the UP adds something to what is already done and to assess

aspects of viability and user satisfaction. The next steps will be

to compare the UP in isolation with the TAU.

Ethics statement

This study has been evaluated and approved by the Drug

Research Ethics Committee (CEIm) of the Parc Sanitari Sant

Joan de Déu. All participants are being provided with an

information sheet explaining the objectives and procedure of

the study as well as the conﬁdentiality of the data collected.

All participants are being asked in writing for their consent in

accordance with the Declaration of Helsinki (WMA,2013) and

Law 14/2007 on Biomedical Research.

Author contributions

TP was the principal investigator of the project, led the

development of the manuscript, and investigator in charge of

recruiting patients. TP, JO, SO, RL-C, and MF-Q did the study

design and decided upon all the outcome measures. MF-Q

was carrying out the evaluation process and conﬁgured the

data collection system. RL-C and TP were carrying out the

therapy of the UP and made the infographics of each module.

JO was supervising all the therapy process and content of the

infographics. SO was responsible for determining sample size

and power calculation and proposing all the statistical analyses.

RL-C and MF-Q kept the patients linked to the study. JO

and SO supervised the development of the study. All authors

contributed to the manuscript and approved the ﬁnal version of

the manuscript.

Funding

This work was ﬁnanced by a grant from Parc Sanitari Sant

Joan de Déu [AR202003]. Jorge Osma has received support by

Gobierno de Aragon and FEDER 2014–2020 “Construyendo

Europa desde Aragon” [Grant Number Research team

S31_20D].

Acknowledgments

We would like to acknowledge all the patients who have

agreed to participate in this trial and all the mental health

professionals who are referring participants to our study. We

also would like to thank Parc Sanitari Sant Joan de Déu and

Sant Joan de Déu Foundation for encouraging research and

facilitating the means and funding to carry out this research.

Conﬂict of interest

The authors declare that the research was conducted in the

absence of any commercial or ﬁnancial relationships that could

be construed as a potential conﬂict of interest.

Publisher’s note

All claims expressed in this article are solely those of

the authors and do not necessarily represent those of their

Frontiers in Psychology 06 frontiersin.org

fpsyg-13-976661 September 1, 2022 Time: 17:28 # 7

Peláez et al. 10.3389/fpsyg.2022.976661

aﬃliated organizations, or those of the publisher, the

editors and the reviewers. Any product that may be

evaluated in this article, or claim that may be made by

its manufacturer, is not guaranteed or endorsed by the

publisher.

Supplementary material

The Supplementary Material for this article can be

found online at: https://www.frontiersin.org/articles/10.3389/

fpsyg.2022.976661/full#supplementary-material

References

Allswede, D. M., Addington, J., Bearden, C. E., Cadenhead, K. S., Cornblatt,

B. A., Mathalon, D. H., et al. (2020). Characterizing covariant trajectories of

individuals at clinical high risk for psychosis across symptomatic and functional

domains. Am. J. Psychiatry 177, 164–171. doi: 10.1176/appi.ajp.2019.18111290

Andersson, G., Cuijpers, P., Carlbring, P., Riper, H., and Hedman, E. (2014).

Guided Internet-based vs. face-to-face cognitive behavior therapy for psychiatric

and somatic disorders: A systematic review and meta-analysis. World Psychiatry

13, 288–295. doi: 10.1002/wps.20151

Barajas, A., Pelaez, T., González, O., Usall, J., Iniesta, R., Arteaga, M., et al.

(2019). Predictive capacity of prodromal symptoms in ﬁrst-episode psychosis of

recent onset. Early Interv. Psychiatry 13, 414–424. doi: 10.1111/eip.12498

Barlow, D. H., Farchione, T. J., Bullis, J. R., Gallagher, M. W., Murray-Latin, H.,

Sauer-Zavala, S., et al. (2017). The uniﬁed protocol for transdiagnostic treatment

of emotional disorders compared with diagnosis-speciﬁc protocols for anxiety

disorders: A randomized clinical trial. JAMA Psychiatry 74:875. doi: 10.1001/

jamapsychiatry.2017.2164

Barlow, D. H., Farchione, T. J., Sauer-Zavala, S., Murray, H., Ellard, K. K., Bullis,

J. R., et al. (2019). Protocolo Uniﬁcado Para el Tratamiento Transdiagnóstico de los

Trastornos Emocionales. Madrid: Alianza Editorial.

Beck, A. T., Baruch, E., Balter, J. M., Steer, R. A., and Warman, D. M. (2004). A

new instrument for measuring insight: The Beck Cognitive Insight Scale (BCIS).

Schizophre. Res. 68, 319–329. doi: 10.1016/S0920-9964(03)00189-0

Beck, A. T., and Steer, R. A. (1993). Beck Anxiety Inventory Manual. San

Antonio, TX: Psychological Corporation.

Beck, A. T., Steer, R. A., and Brown, G. K. (1996). BDI-II. Beck Depression

Inventory-Second Edition Manual. San Antonio, TX: Psychological Corporation.

Ben-David, S., Cole, A. R., Brucato, G., Girgis, R., and Munson, M. R. A.

(2019). conceptual model of mental health service utilization among young adults

at clinical high-risk for developing psychosis. Psychiatr. Rehabil. J. 42, 17–25.

doi: 10.1037/prj0000336

Bentley, K. H., Gallagher, M. W., Carl, J. R., and Barlow, D. H. (2014).

Development and validation of the Overall Depression Severity and Impairment

Scale. Psychol. Assess. 26, 815–830. doi: 10.1037/a0036216

Bullis, J. R., Fortune, M. R., Farchione, T. J., and Barlow, D. H. A. (2014).

preliminary investigation of the long-term outcome of the Uniﬁed Protocol

for Transdiagnostic Treatment of Emotional Disorders. Compr. Psychiatry 55,

1920–1927. doi: 10.1016/j.comppsych.2014.07.016

Carl, J. R., Gallagher, M. W., Sauer-Zavala, S. E., Bentley, K. H., and Barlow,

D. H. A. (2014). preliminary investigation of the eﬀects of the uniﬁed protocol on

temperament. Compr. Psychiatry 55, 1426–1434. doi: 10.1016/j.comppsych.2014.

04.015)

Carlucci, L., Saggino, A., and Balsamo, M. (2021). On the eﬃcacy of the uniﬁed

protocol for transdiagnostic treatment of emotional disorders: A systematic review

and meta-analysis. Clin. Psychol. Rev. 87:101999. doi: 10.1016/j.cpr.2021.101999

Cassiello-Robbins, C., Southward, M. W., Tirpak, J. W., and Sauer-Zavala,

S. A. (2020). systematic review of Uniﬁed Protocol applications with adult

populations: Facilitating widespread dissemination via adaptability. Clin. Psychol.

Rev. 78:101852. doi: 10.1016/j.cpr.2020.101852

Colizzi, M., Ruggeri, M., and Lasalvia, A. (2020). Should we be concerned about

stigma and discrimination in people at risk for psychosis? A systematic review.

Psychol. Med. 50, 705–726. doi: 10.1017/S0033291720000148

Corral, L., Labad, J., Ochoa, S., Cabezas, A., Muntané, G., Valero, J., et al.

(2018). Cognitive Biases Questionnaire for Psychosis (CBQ-sp): Spanish validation

and relationship with cognitive insight in psychotic patients. Front. Psychiatry

11:596625. doi: 10.3389/fpsyt.2020.596625

Costa, P. T., and Mccrae, R. R. (1992). Manual For the Revised NEO Personality

Inventory (NEO-PIR) and NEO Five-Factor Inventory (NEO-FFI)Professional

Manual. Odessa, FL: Psychological Assessment Resources.

Echeburúa, E., Corral, P. D., and Fernández-Montalvo, J. (2000). Escala de

inadaptación (EI): Propiedades psicométricas en contextos clínicos. Análisis

yModiﬁcación de Conducta 26, 325–340.

Ferrando, L., Bobes, J., Gibert, M., Soto, M., and Soto, O. (1998). M.I.N.I. Mini

International Neuropsychiatric Interview. Madrid: Instituto IAP.

Ferrans, C. E., and Powers, M. J. (1985). Quality of life index: Development and

psychometric properties. ANS Adv. Nurs. Sci. 8, 15–24. doi: 10.1097/00012272-

198510000-00005

French, P., and Morrison, A. P. (2004). Early Detection and Cognitive Therapy

for People at High Risk of Developing Psychosis: A Treatment Approach. London:

John Wiley and Sons.

Fusar-Poli, P., Bonoldi, I., Yung, A. R., Borgwardt, S., Kempton, M. J.,

Valmaggia, L., et al. (2012). Predicting psychosis: Meta-analysis of transition

outcomes in individuals at high clinical risk: Meta-analysis of transition outcomes

in individuals at high clinical risk. Arch. Gen. Psychiatry 69, 220–229. doi: 10.1001/

archgenpsychiatry.2011.1472

Grasa, E., and Corripio, I. (2019). En: Aplicaciones del Protocolo Uniﬁcado para

el tratamiento transdiagnóstico de la disregulación emocional. Madrid Alianza

Editor. 2019, 115–138.

Gratz, K. L., and Roemer, L. (2004). Multidimensional assessment of emotion

regulation and dysregulation: Development, factor structure, and initial validation

of the Diﬃculties in Emotion Regulation Scale. J. Psychopathol. Behav. Assess. 26,

41–54. doi: 10.1023/B:JOBA.0000007455.08539.94

Gutierrez-Zotes, A., Valero, J., Ruiz, C., Labad, M., Ochoa, A., Ahuir,

S., et al. (2012). Spanish Adaptation of the Beck Cognitive Insight

Scale (BCIS) for Schizophrenia. Actas Españolas De Psiquiatría. 40,

2–9.

Hartmann, J. A., Nelson, B., Spooner,R., Paul Amminger, G., Chanen, A., Davey,

C. G., et al. (2019). Broad clinical high-risk mental state (CHARMS): Methodology

of a cohort study validating criteria for pluripotent risk. Early Interv. Psychiatry 13,

379–386. doi: 10.1111/eip.12483

Hervás, G., and Jódar, R. (2008). The Spanish version of the Diﬃculties in

Emotion Regulation Scale. Clínica y Salud. 2, 139–156.

Howes, O. D., Whitehurst, T., Shatalina, E., Townsend, L., Onwordi, E. C., Mak,

T. L. A., et al. (2021). The clinical signiﬁcance of duration of untreated psychosis:

An umbrella review and random-eﬀects meta-analysis. World Psychiatry 20,

75–95. doi: 10.1002/wps.20822

Inchausti, F., Mole, J., Fonseca-Pedrero, E., and Ortuño-Sierra, J. (2015).

Validity of personality measurement in adults with anxiety disorders:

Psychometric properties of the Spanish NEO-FFI-R using Rasch analyses.

Front. Psychol. 6:465. doi: 10.3389/fpsyg.2015.00465

Ising, H. K., Lokkerbol, J., Rietdijk, J., Dragt, S., Klaassen, R. M. C., Kraan,

T., et al. (2017). Four-year cost-eﬀectiveness of cognitive behavior therapy

for preventing ﬁrst-episode psychosis: The dutch early detection intervention

evaluation (EDIE-NL) trial. Schizophr. Bull. 43, 365–374. doi: 10.1093/schbul/

sbw084

Lin, A., Wood, S. J., Nelson, B., Beavan, A., McGorry, P., and Yung, A. R.

(2015). . Outcomes of nontransitioned cases in a sample at ultra-high risk

for psychosis. Am. J. Psychiatry 172, 249–258. doi: 10.1176/appi.ajp.2014.1303

0418

Livet, A., Navarri, X., Potvin, S., and Conrod, P. (2020). Cognitive biases in

individuals with psychotic-like experiences: A systematic review and a meta-

analysis. Schizophr. Res. 222, 10–22. doi: 10.1016/j.schres.2020.06.016

López-Gómez, I., Hervás, G., and Vázquez, C. (2015). Adaptación de la “Escala

de Afecto Psotivo y Negativo” (PANAS) en una muestra general española. Behav.

Psychol. 3, 529–548.

Lupton, D. (2021). Young people’s use of digital health technologies in the

Global North: Narrative review. J. Med. Int. Res. 23:e18286. doi: 10.2196/18286

Frontiers in Psychology 07 frontiersin.org

fpsyg-13-976661 September 1, 2022 Time: 17:28 # 8

Peláez et al. 10.3389/fpsyg.2022.976661

McGorry, P. D., Hartmann, J. A., Spooner, R., and Nelson, B. (2018). Beyond the

“at risk mental state” concept: Transitioning to transdiagnostic psychiatry. World

Psychiatry 17, 133–142. doi: 10.1002/wps.20514

McLaren, T., Peter, L.-J., Tomczyk, S., Muehlan, H., Stolzenburg, S., Schomerus,

G., et al. (2021). How can the utilisation of help for mental disorders be improved?

A quasi-experimental online study on the changeability of stigmatising attitudes

and intermediate variables in the process of utilisation. BMC Public Health

21:2124. doi: 10.1186/s12889-021- 12125-5

Mei, C., van der Gaag, M., Nelson, B., Smit, F., Yuen, H. P., Berger, M., et al.

(2021). Preventive interventions for individuals at ultra high risk for psychosis: An

updated and extended meta-analysis. Clin. Psychol. Rev. 86:102005. doi: 10.1016/j.

cpr.2021.102005

Mezzich, J. E., Ruipérez, M. A., Pérez, C., Yoon, G., Liu, J., and Mahmud,

S. (2000). The Spanish version of the Quality of Life index: Presentation and

validation. J. Nerv. Ment. Dis. 188, 301–305. doi: 10.1097/00005053-200005000-

00008

Morrison, A. P., French, P., Stewart, S. L. K., Birchwood, M., Fowler,D., Gumley,

A. I., et al. (2012). Early detection and intervention evaluation for people at risk of

psychosis: Multisite randomised controlled trial. BMJ 344:e2233. doi: 10.1136/bmj.

e2233

Morrison, A. P., French, P., Walford, L., Lewis, S. W., Kilcommons, A., Green,

J., et al. (2004). Cognitive therapy for the prevention of psychosis in people at

ultra-high risk: Randomised controlled trial. Br. J. Psychiatry 185, 291–297. doi:

10.1192/bjp.185.4.291

Nelson, B., and Orygen Youth Health Research Centre Issuing Body. (2012). The

PACE Clinic Manual: A Treatment Approach for Young People at Ultra High Risk

of Psychosis. Parkville, Vic: Orygen Youth Health Research Centre.

Norman, S. B., Hami Cissell, S., Means-Christensen, A. J., and Stein, M. B.

(2006). Development and validation of an overall anxiety severity and impairment

scale (OASIS). Depress. Anxiety 23, 245–249. doi: 10.1002/da.20182

Orygen The National Center of Excellence in Youth (2016). Australian

Clinical Guidelines for Early Psychosis Available onlone at: https://www.orygen.

org.au/Campus/Expert-Network/Resources/Free/Clinical- Practice/Australian-

Clinical-Guidelines- for-Early- Psychosis/Australian-Clinical-Guidelines-for-

Early-Psychosis.aspx?ext (accessed June, 2016).

Osma, J., Martínez-García, L., Peris-Baquero, Ó, Navarro-Haro, M. V.,

González-Pérez, A., and Suso-Ribera, C. (2021). Implementation, eﬃcacy and cost

eﬀectiveness of the uniﬁed protocol in a blended format for the transdiagnostic

treatment of emotional disorders: A study protocol for a multicentre, randomised,

superiority controlled trial in the Spanish National Health System. BMJ Open

11:e054286. doi: 10.1136/bmjopen-2021- 054286.2021

Osma, J., Peris-Baquero, O., Suso-Ribera, C., Farchione, T. J., and Barlow,

D. H. (2022). Eﬀectiveness of the Uniﬁed Protocol for transdiagnostic treatment

of emotional disorders in group format in Spain: Results from a randomized

controlled trial with 6-months follow-up. Psychother. Res. 32, 329–342. doi: 10.

1080/10503307.2021.1939190

Osma, J., Quilez-Orden, A., Suso-Ribera, C., Peris-Baquero, O., Norman, S. B.,

Bentley, K. H., et al. (2019). Psychometric properties and validation of the Spanish

versions of the overall anxiety and depression severity and impairment scales.

J. Aﬀect. Disord. 252, 9–18. doi: 10.1016/j.jad.2019.03.063

NICE (2014). Psychosis and Schizophrenia in Adults: Prevention and

Management | Guidance | NICE. Available online at: http://www.nice.org.uk/

guidance/cg178 (accessed March 1, 2014).

Peters, E., Moritz, S., Wiseman, Z., Greenwood, K., Kuipers, E., Schwannauer,

M., et al. (2010). The Cognitive Biases Questionnaire for Psychosis. (CBQP).

Schizophr. Res. 117, 413–413. doi: 10.1016/j.schres.2010.02.759

Polari, A., Lavoie, S., Yuen, H.-P., Amminger, P., Berger, G., Chen, E., et al.

(2018). Clinical trajectories in the ultra-high risk for psychosis population.

Schizophr. Res. 197, 550–556. doi: 10.1016/j.schres.2018.01.022

Polari, A., Street, R., Conus, P., Finkelstein, A., Hartmann, J. A., Kim, S.-W.,

et al. (2021a). Patients’, carers’ and clinicians’ attitudes towards alternative terms to

describe the at-risk for psychosis state. Schizophr. Res. 237, 69–75. doi: 10.1016/j.

schres.2021.08.031

Polari, A., Yuen, H. P., Amminger, P., Berger, G., Chen, E., deHaan, L., et al.

(2021b). Prediction of clinical outcomes beyond psychosis in the ultra-high risk for

psychosis population. Early Interv. Psychiatry 15, 642–651. doi: 10.1111/eip.13002

Rosellini, A. J., and Brown, T. A. (2019). The Multidimensional Emotional

Disorder Inventory (MEDI): Assessing transdiagnostic dimensions to validate a

proﬁle approach to emotional disorder classiﬁcation. Psychol. Assess. 31, 59–72.

doi: 10.1037/pas0000649

Rutigliano, G., Valmaggia, L., Landi, P., Frascarelli, M., Cappucciati, M., Sear,

V., et al. (2016). Persistence or recurrence of non-psychotic comorbid mental

disorders associated with 6-year poor functional outcomes in patients at ultra high

risk for psychosis. J. Aﬀect. Disord. 203, 101–110. doi: 10.1016/j.jad.2016.05.053

Sakiris, N., and Berle, D. A. (2019). A.systematic review and meta-analysis of the

Uniﬁed Protocol as a transdiagnostic emotion regulation based intervention. Clin.

Psychol. Rev. 2019:72. doi: 10.1016/j.cpr.2019.101751

Salazar de Pablo, G., Radua, J., Pereira, J., Bonoldi, I., Arienti, V., Besana, F.,

et al. (2021). Probability of transition to psychosis in individuals at clinical high

risk: An updated meta-analysis: An updated meta-analysis. JAMA Psychiatry 78,

970–978. doi: 10.1001/jamapsychiatry.2021.0830

Santesteban-Echarri, O., Piskulic, D., Nyman, R. K., and Addington, J. (2020).

Telehealth interventions for schizophrenia-spectrum disorders and clinical high-

risk for psychosis individuals: A scoping review. J Telemed. Telecare 26, 14–20.

doi: 10.1177/1357633X18794100

Sanz, J., García-Vera, M. P., Espinosa, R., Fortún, M., and Vázquez, C. (2005).

Adaptación española del Inventario para la Depresión de Beck-II (BDI-II):

Propiedades psicométricas en pacientescon trastornos psicológicos. Clínica y

Salud 16, 121–142.

Sanz, J., García-Vera, M. P., and Fortún, M. (2012). El “Inventario de Ansiedad

de Beck” (BAI): Propiedades psicométricas de la Versión Española en pacientes

con Trastornos Psicológicos. Behav. Psychol. 20, 563–583.

Sauer-Zavala, S., Gutner, C. A., Farchione, T. J., Boettcher, H. T., Bullis, J. R.,

and Barlow, D. H. (2017). Current deﬁnitions of “transdiagnostic” in treatment

development: A search for consensus. Behav. Ther. 48, 128–138. doi: 10.1016/j.

beth.2016.09.004

Shah, J. L., Crawford, A., Mustafa, S. S., Iyer, S. N., Joober, R., and Malla, A. K.

(2017). Is the clinical high-risk state a valid concept? Retrospective examination in

a ﬁrst-episode psychosis sample. Psychiatr. Serv. 68, 1046–1052. doi: 10.1176/appi.

ps.201600304

Sheehan, D. V., Lecrubier, Y., Sheehan, K. H., Amorim, P., Janavs, J., Weiller, E.,

et al. (1998). The Mini-International Neuropsychiatric Interview (M.I.N.I.): The

development and validation of a structured diagnostic psychiatric interview for

DSM-IV and ICD-10. J. Clin. Psychiatry 59, 34–57. doi: 10.1037/t18597-000

Singh, S., and Sagar, R. (2022). Online Psychotherapy During the COVID-19

Pandemic: The Good, the Bad, and the Ugly. Indian J. Psychol. Med. 44, 177–180.

doi: 10.1177/02537176211070427

Stubbings, D. R., Rees, C. S., Roberts, L. D., and Kane, R. T. (2013). Comparing

in-person to videoconference-based cognitive behavioral therapy for mood and

anxiety disorders: Randomized controlled trial. J. Med. Int. Res. 15:e258. doi:

10.2196/jmir.2564

Taylor, E. (2016). Program Directory of Early Psychosis Intervention Programs.

Portland, OR: EASA Center for Excellence.

Van der Gaag, M., Nieman, D., and van den Berg, D. (2013a). CBT for Those at

Risk of a First Episode Psychosis: Evidence-Based Psychotherapy for People with an

At Risk Mental State. New York: Routledge.

Van der Gaag, M., Smit, F., Bechdolf, A., French, P., Linszen, D. H., Yung, A. R.,

et al. (2013b). Preventing a ﬁrst episode of psychosis: Meta-analysis of randomized

controlled prevention trials of 12 month and longer-term follow-ups. Schizophr.

Res. 149, 56–62. doi: 10.1016/j.schres.2013.07.004

Van Os, J., and Guloksuz, S. A. (2017). critique of the “ultra-high risk” and

“transtion” paradigm. World Psychiatry 16, 200–206. doi: 10.1002/wps.20423

Watson, D., Clark, L. A., and Tellegen, A. (1988). Development and validation

of brief measures of positive and negative aﬀect: The PANAS scales. J. Pers. Soc.

Psychol. 54, 1063–1070. doi: 10.1037//0022-3514.54.6.1063

WMA (2013). “Declaration of helsinki-ethical principles for medical research

involving human subjects,” in 64th General Assembly, Fortaleza.

Woodberry, K. A., Powers, K. S., Bryant, C., Downing, D., Verdi, M. B., Elacqua,

K. M., et al. (2021). Emotional and stigma-related experiences relative to being told

one is at risk for psychosis. Schizophr. Res. 238, 44–51. doi: 10.1016/j.schres.2021.

09.017

Yung, A. R., Yuen, H. P., Phillips, L. J., Francey, S., and McGorry, P. D. (2005).

Mapping the onset of psychosis: The comprehensive assessment of at risk mental

states (CAARMS). Schizophr. Res. 60, 30–31. doi: 10.1016/s0920-9964(03)80090- 7

Frontiers in Psychology 08 frontiersin.org

Frontiers in Psychology 01 frontiersin.org

Eects of virtual reality natural

experiences on factory workers’

psychological and physiological

stress

Mu-HsingHo

1, Meng-ShinWu

2 and Hsin-YenYen

1 School of Nursing, LKS Faculty of Medicine, The University of Hong Kong, Pokfulam, Hong Kong SAR,

China, 2 School of Gerontology and Long-Term Care, College of Nursing, Taipei Medical University,

Taipei, Taiwan

Introduction: Manufacturing facilities and factories are stressful work

environments. Interventions to improve factory workers’ stress is necessary to

promote occupational health. This study aimed to examine the eects of virtual

reality natural experiences on furniture factory employees’ psychological and

physiological stress.

Methods: A single-blinded, non-randomised quasi-experimental study was

conducted between July and December 2021. Factory workers were recruited

from two factories, and all participants at a given factory were assigned to either an

experimental group or a comparison group. The intervention was conducted in a

clean conference room once a week for 12 weeks during the worker’s break time.

The experimental group received virtual reality natural experiences consisting

of 30-minute nature-based 360° videos which were played in a headset. The

generalised estimating equations were performed for the statistical analyses.

Results: In total, 35 participants completed the intervention. As to psychological

stress, the experimental group showed improvements in distress, depression, and

anxiety, and a positive aect after the intervention compared to the comparison

group. As to physiological stress, the experimental group showed improvements

in indicators of heart rate variability compared to the comparison group, including

standard deviations of all normal-to-normal intervals, low-frequency power, and

high-frequency power.

Discussion: Virtual reality is an innovative platform to bring the natural environment

into an indoor environment to create similar health eects.

KEYWORDS

green space, heart rate variability, immersion, mental health, occupational health,

virtual reality

1. Introduction

Occupational stress is a well-known issue worldwide that inuences both developed and

developing countries. Workplace stress occurs when work-related demands surpass a worker’s

capacity to manage them (World Health Organization, 2020). Moreover, globalization and

dramatic changes have had direct impacts on the variety of work in terms of technological

developments, higher job demands, and workloads. Also, aging populations and the

demographic and systemic structure of the workforce, such as a poor work-life balance, job

insecurity, and precarious employment, have resulted in a signicant occupational stress

epidemic worldwide (Sorensen et al., 2021). Work stress is particularly important and

OPEN ACCESS

EDITED BY

Lisa A. Osborne,

The Open University,

UnitedKingdom

REVIEWED BY

Delia Virga,

West University of Timișoara, Romania

Benjamin Huang,

University of California, San Diego,

UnitedStates

Anabela da Conceição Pereira,

University Institute of Lisbon (ISCTE), Portugal

*CORRESPONDENCE

Hsin-Yen Yen

yenken520@gmail.com;

kenji@tmu.edu.tw

SPECIALTY SECTION

This article was submitted to

Psychology for Clinical Settings,

a section of the journal

Frontiers in Psychology

RECEIVED 13 July 2022

ACCEPTED 15 February 2023

PUBLISHED 06 March 2023

CITATION

Ho M-H, Wu M-S and Yen H-Y (2023) Eects of

virtual reality natural experiences on factory

workers’ psychological and physiological

stress.

Front. Psychol. 14:993143.

doi: 10.3389/fpsyg.2023.993143

access article distributed under the terms of

the Creative Commons Attribution License

(CC BY). The use, distribution or reproduction

in other forums is permitted, provided the

original author(s) and the copyright owner(s)

are credited and that the original publication in

this journal is cited, in accordance with

accepted academic practice. No use,

distribution or reproduction is permitted which

does not comply with these terms.

TYPE Original Research

PUBLISHED 06 March 2023

DOI 10.3389/fpsyg.2023.993143

Ho et al. 10.3389/fpsyg.2023.993143

Frontiers in Psychology 02 frontiersin.org

signicantly impacts individuals and organizations. Workplace stress

causes a variety of ailments, including cardiovascular and metabolic

disorders, psychological issues, musculoskeletal discomfort,

reproductive issues, and occupational injuries, and also leads to a poor

quality of life (Mohamed etal., 2022). ese health issues are related

to increased absenteeism and presenteeism, as well as decreased

motivation, contentment, and commitment. ese can produce

increases in employee turnover and a desire to resign, resulting in low

business productivity and increased medical, healthcare, and social

welfare expenditures (Asplund etal., 2022; Mohamed etal., 2022;

Sznajder etal., 2022).

Manufacturing is not exempt from stressful work environments.

Workers in the manufacturing industry become stressed as a result of

high job expectations, lengthy and irregular working hours, and tough

work shi patterns in order to reach production objectives and

maintain customer satisfaction. Constant work stress and pressure

result in both physical and mental exhaustion, a lack of work-life

balance, and decreased employee productivity (Bhui et al., 2016;

Bolliger etal., 2022; Kim and Jung, 2022). Hopelessness, not feeling

useful, and feeling depressed in the work environment are considered

factors associated with symptoms of work-related stress among factory

workers (Sznajder etal., 2022), highlighting the need for interventions

to alleviate poor mental health symptoms among workers in high-

pressure occupational environments.

Nature-based interventions have been studied and are considered

eective strategies for alleviating stress and mental health illnesses

(Picton etal., 2020; Coventry etal., 2021). Nature-based interventions

provide individuals with an opportunity to explore their relationship

with nature in terms of connecting to and being impacted by the

natural environment to reduce negative mental health issues (Hartig

etal., 2014; Owens and Bunce, 2022). However, infusing the natural

environment in the workplace is challenging due to urbanization, and

people who live and work in urban areas have very limited

opportunities to connect with nature. A scarcity of research has been

undertaken to implement nature-based interventions and build a

natural environment in the factory workplace.

Virtual reality (VR) is becoming an increasingly popular

technology, and a growing body of research has demonstrated the

eect of using VR as a tool to enable engagement with natural

environments (Li et al., 2021; Adhyaru and Kemp, 2022;

Spangenberger etal., 2022). Several nature videos and applications can

also beeasily accessed and applied as VR technology (Adhyaru and

Kemp, 2022; Calogiuri et al., 2022). Implementing a natural

environment using VR is a novel approach and likely to produce

psycho-physiological benets by bringing nature into an indoor

environment (Browning etal., 2019). A previous evidence-based study

revealed that using VR natural experiences had positive impacts on

psychological stress in terms of mood, anxiety, perceived stress, and

physiological stress such as the heart rate (HR) (Adhyaru and Kemp,

2022). However, few investigations have been conducted into the

impacts of VR natural experiences on biofeedback and physiological

stress. A more in-depth discussion and evidence are required of the

physiological changes such as the autonomic nervous system (ANS)

and blood pressure (BP) measurements through VR natural

experiences (Lüddecke and Felnhofer, 2022). Moreover, research

touched on both psychological and physiological stress-related

outcome is scarce. VR natural experiences can beconsidered as a

simulation-based intervention contributing to mental-state attribution

through the simulation of perception. According to the simulation

theory, activity in sensory cortex that resembles the perception of

external stimuli can be elicited from other parts of the brain.

Particularly from a simulation-based intervention, imagining, hearing,

or feeling something is essentially the same as actually seeing, hearing,

or feeling it (Hesslow, 2012). erefore, VR natural experiences

intervention which brought a natural environment into a workplace

has a great potential and contribution to psychological and

physiological stress improvement. However, research on utilizing VR

natural experience on alleviating occupational stress are limited. More

empirical studies on investigating the eect of VR natural experience

on occupational stress are warranted. us, the main purpose of this

study was to explore the eects of VR natural experiences on furniture

factory employees’ stress. Using an innovative intervention of VR

natural experiences during their break time, the factory workers in the

experimental group were expected to show improvements in their

psychological and physiological stress compared to the comparison

group. Psychological stress-related outcomes included distress,

depression, anxiety, somatization, positive and negative aects, and

perceived stress. Physiological stress-related outcomes included BP

and HR variability (HRV).

2. Materials and methods

2.1. Study design

is was a single-blinded, two-armed non-randomized, quasi-

experimental study conducted from July to December 2021.

Participants were recruited from two furniture factories by

convenience sampling in southern Taiwan. e supervisors of the

two factories were contacted by the principal investigator, and the

oral consent was obtained to invite eligible workers in the factories.

en, a researcher explained the aim and procedure of the study to

all workers. All workers were required to sign an informed consent

form before data collection and the intervention. e workers in the

two factories were either assigned to an experimental group (VR

group) and a comparison group by drawing. e minimal sample

size was 34 which was calculated by medium eect sizes via

G*power soware (Faul et al., 2007). Each factory recruited

21 participants who were either in the same experimental or

comparison group. Participants were not aware of the other group.

Ethical approval for this study was obtained from the Taiwan

Medical University-Joint Institutional Review Board (N202103114).

is study was conducted in accordance with the principles of the

Declaration of Helsinki.

2.2. Participants

Participants were factory workers who had break time in the

aernoon. e inclusion criteria were participants (1) aged

20–60 years, (2) who worked on the production line, (3) who had no

visual or hearing impairment, and (4) who had no serious health

problems, mental illness, or disability that might inuence the

experiment and outcomes. e exclusion criteria were participants (1)

who had gone to a natural environment for recreation in the past year,

(2) who went to parks or green spaces weekly, (3) who had experiences

in using any VR devices in the past year, and (4) who experienced

serious VR sickness. e study ow is illustrated in Figure1.

100

Ho et al. 10.3389/fpsyg.2023.993143

Frontiers in Psychology 03 frontiersin.org

2.3. The intervention

e intervention was conducted once a week for 12 weeks. To not

interrupt the working time, the intervention in the experimental VR

group was conducted during workers’ break time. Five participants

with the same break time schedule were grouped together. e group

received the VR intervention in a clean conference room at the same

time. e conference room was free of interference from any external

visual or auditory stimulation. Participants in the experimental group

were required to sit in a chair and wear a VR headset (Oculus Quest

2, META, UnitedStates) to watch 360° videos for 30 min (Figure2).

Based on the coronavirus disease 2019 (COVID-19) policy at that

time, participants were also required to wear a face mask in

indoor environments.

Nature-based VR videos were pre-recorded in a 360° format,

including such areas as parks, hiking trails, forest paths, and bikeways

(Supplementary Figures S1, S2). All videos were recorded on a sunny

day in the aernoon. A dierent 30 min nature-based video was

played in the VR headset every week. During the session, participants

could freely move the direction of their head to watch the video from

various angles. Participants were asked not to talk to each other during

the session. A trained college student supervised every session. When

using the VR headset, if a participant felt a little uncomfortable, they

were told to temporarily close their eyes and then open their eyes

FIGURE1

Flow diagram detailing the progress of enrolment to analysis.

FIGURE2

Photo of the virtual reality (VR) natural experience intervention.

101

Ho et al. 10.3389/fpsyg.2023.993143

Frontiers in Psychology 04 frontiersin.org

again. If the uncomfortable sensation continued, they were told to stop

the session. e comparison group received no interventions for

12 weeks. Participants in the comparison group were free to do any

activities of their choosing during their aernoon break time.

2.4. Measures

Measurements were conducted in the conference room once

before and once aer the 12 week intervention by a trained college

student. Stress-related outcomes were measured by self-reported

questionnaires, a sphygmomanometer, and an HRV analyzer.

Participants’ background information was collected in self-reported,

structured questionnaires, including age, gender, education level,

marital status, main job content in the factory, alcohol use, smoking,

and chronic diseases.

Psychological measures. e Four-Dimensional Symptom

Questionnaire (4DSQ) measures four common mental health

problems: distress, anxiety, depression, and somatization. In total,

50 items were measured on a ve-point scale. A higher score

indicates worse symptoms. e 4DSQ previously presented good

content validity, criterion-related validity, and construct validity

(Terluin etal., 2016). Cronbach’s α was 0.802 in this study. e

Positive and Negative Aect Scales (PANAS) were used for

measuring participants’ emotions in two dimensions, including

positive and negative aects. In total, 50 items were measured on a

ve-point scale. A higher score indicates a higher perceived aective

status. e PANAS have good reliability and construct validity

(Watson etal., 1988). e Perceived Stress Scale (PSS) was used for

a self-evaluation of stress in the past month. In total, 10 items were

measured on a ve-point scale. A higher score indicates higher

perceived stress. e PSS previously had good reliability, construct

validity, and criterion-related validity (Cohen et al., 1983).

Cronbach’s α was 0.903in this study.

Physiological measures. A BP monitor (HEM-7310, OMRON,

Japan) was used to measure participants’ systolic BP (SBP) and

diastolic BP (DBP). e BP monitor was validated European Society

of Hypertension International Protocol. by A handheld

electrocardiogram (ECG) Monitor (8Z11, Wegene Technology,

Taiwan) was used for ECG signal acquisition, storage, and processing

of resting HRV and the ANS with the good validity (Tseng etal.,

2020). Participants were requested to sit still for 5 min to record short-

term HRV. Aer the algorithm, the selective parameters of HRV

included the HR, a standard deviation (SD) of all normal-to-normal

intervals (SDNN), total power (TP, 0–0.5 Hz), low-frequency (LF)

power (0.04–0.15 Hz), high-frequency (HF) power (0.15–0.40 Hz),

and the ratio of LF to HF (LF/HF). Participants with a higher SDNN

and LF/HF were more likely to have better ANS and lower stress,

anxiety, and depression (Malik etal., 1996).

2.5. Statistical analysis

Descriptive analyzes used the frequency and percentage for

categorized variables and the mean and SD of continuous

variables. Chi-square tests were performed to compare

participants’ backgrounds between the experimental and

comparison groups. Cramer’s V was calculated for effect sizes of

the Chi-squared tests. Independent t-tests were performed to

compare differences in participants’ age and stress-related

outcomes of psychological and physiological measures at the

baseline. Cohen’s d was calculated for effect sizes of the

independent t-tests. The natural logarithms (Ln) of HRV data (TP,

LF, HF, and LF/HF) were calculated for further analyzes. Analyzes

were performed based on intention-to-treat principle. ITT

approach provides unbiased comparisons among the treatment

groups and this technique was done to avoid the effects of

dropout, which the number of participants after group allocation

was included in the final analysis (i.e., VR group N = 20 and

Comparison group N = 20). Generalized estimating equations

(GEEs) were performed to analyze the effect of group, time, and

group-by-time interactions on stress-related outcomes. The GEEs

were adjusted for participants’ age and the score at the baseline.

SPSS 18.0 (SPSS, UnitedStates) was used for all statistical analyzes.

3. Results

3.1. Participants’ backgrounds

Table 1 reveals the participants’ backgrounds. No signicant

dierences were found in gender, smoking behavior, or chronic

diseases between the VR and comparison groups. However,

participants’ age, education levels, marital status, and job content

exhibited signicant dierences between the two groups. erefore,

the signicant continuous variable (age) of participants’ backgrounds

was adjusted for in the subsequent GEE analysis. In total,

18 participants in the VR group and 17 participants in the comparison

group completed the intervention.

3.2. Stress-related outcomes at the

baseline

Table2 demonstrates participants’ stress-related outcomes at the

baseline. Negative aect (p = 0.043) and SBP (p = 0.043) were found to

signicantly dier between the VR and comparison groups. No other

variables of psychological or physiological measures were found to

signicantly dier between the VR and comparison groups.

3.3. Outcomes

Table3 demonstrates the GEE-adjusted model which indicates the

eects of group, time, and group-by-time interactions on stress-related

outcomes, and the model was adjusted for participants’ age and the

outcome score at the baseline. For psychological measures, signicant

group eects (p = 0.021) and group-by-time interactions (p = 0.015)

were found for distress. Signicant group eects (p = 0.039) and

group-by-time interactions (p = 0.042) were also found for anxiety.

Signicant group eects were found for depression (p = 0.005) and

positive aect (p = 0.035). Mean dierences indicated that distress,

depression, anxiety, and positive aect in the VR group improved aer

the intervention compared to the comparison group. In contrast,

102

Ho et al. 10.3389/fpsyg.2023.993143

Frontiers in Psychology 05 frontiersin.org

somatization, negative aect, and perceived stress revealed no

signicant eects.

For physiological measures, signicant group eects (p = 0.031)

and group-by-time interactions (p < 0.001) were found for DBP. A

signicant group-by-time interaction (p = 0.007) was found for

SBP. HRV outcomes revealed that SDNN had a signicant group eect

(p = 0.030). Both LF (p = 0.041) and HF (p = 0.028) had signicant

group-by-time interactions. Mean dierences indicated that the

TABLE1 Participants’ backgrounds.

Parameter VR group (N = 20) Comparison group

(N = 20) 𝒙2pV1

n(%) n(%)

Gender Male 7 (36.84%) 12 (60.00%) 2.09 0.148 0.232

Female 12 (63.16%) 8 (40.00%)

Educational level <high school 13 (81.25%) 6 (33.33%) 7.89 0.005 0.482

>college 3 (18.75%) 12 (66.67%)

Marital status Single/divorced 3 (15.79%) 10 (50.00%) 5.13 0.023 0.363

Married 16 (84.21%) 10 (50.00%)

Job content Tail or 5 (26.32%) 5 (25.00%) 8.62 0.013 0.470

Sewing 8 (42.11%) 1 (5.00%)

Others 6 (31.58%) 14 (70.00%)

Alcohol use No 13 (86.67%) 10 (50.00%) 5.12 0.024 0.382

Yes 2 (13.33%) 10 (50.00%)

Smoking No 3 (16.67%) 2 (10.00%) 0.37 0.544 0.098

Yes 15 (83.33%) 18 (90.00%)

Chronic diseases No 14 (73.68%) 17 (85.00%) 0.77 0.382 0.140

Yes 5 (26.32%) 3 (15.00%)

1Cramer’s V for eect size. VR, virtual reality.

TABLE2 Participants’ stress-related outcomes at the baseline.

Variable VR group (N = 20) Comparison group

(N = 20) t p d

Mean (SD) Mean (SD)

Age (years) 55.21 (7.71) 36.10 (11.12) 6.27 <0.001 1.998

Psychological

measures

Distress 23.24 (6.51) 23.05 (4.43) 0.10 0.919 0.033

Depression 8.12 (3.30) 6.70 (1.42) 1.65 0.114 0.559

Anxiety 15.24 (4.52) 14.80 (3.65) 0.32 0.748 0.106

Somatization 25.29 (8.07) 24.10 (5.31) 0.54 0.593 0.175

Positive aect 2.71 (1.06) 2.68 (0.70) 0.09 0.930 0.028

Negative aect 2.26 (0.85) 1.76 (0.64) 2.09 0.043 0.668

Perceived stress 25.63 (4.57) 26.40 (4.84) −0.51 0.614 0.163

Physiological

measures

SBP (mmHg) 132.90 (15.73) 118.63 (16.09) 2.80 0.008 0.897

DBP (mmHg) 80.35 (11.72) 73.05 (10.61) 2.04 0.050 0.653

HR (bpm) 75.20 (10.13) 80.00 (13.73) −1.25 0.220 0.398

SDNN (ms) 39.02 (17.16) 44.75 (19.85) −0.97 0.341 0.309

TP [Ln (ms2)] 6.94 (0.99) 7.32 (0.95) −1.22 0.229 0.392

LF [Ln (ms2)] 5.58 (1.31) 6.06 (0.84) −1.36 0.181 0.439

HF [Ln (ms2)] 4.97 (1.35) 5.46 (1.23) −1.17 0.249 0.376

LF/HF [Ln (ratio)] 0.61 (0.70) 0.61 (0.79) 0.01 0.990 0.004

Cohen’s d for eect size; VR, virtual reality; SD, standard deviation; SBP, systolic blood pressure; DBP, diastolic blood pressure; HR, heart rate; SDNN, standard deviation of all normal-to-

normal intervals; TP, total power (0–0.5 Hz); LF, low-frequency power (0.04–0.15 Hz); HF, high-frequency power (0.15–0.40 Hz); LF/HF, the ratio of low frequency to high frequency.

103

Ho et al. 10.3389/fpsyg.2023.993143

Frontiers in Psychology 06 frontiersin.org

comparison group had higher blood pressure aer 12 weeks. e

SDNN, LF, and HF of the VR group had improved aer the

intervention compared to the comparison group. However, HR and

LF/HF exhibited no signicant eects.

4. Discussion

is study applied a VR device to bring the natural environment

to furniture factory workers. VR natural experiences were introduced

in the aernoon break time weekly for 12 weeks. Participants’

psychological stress, including distress, depression, anxiety, and

positive aect, improved aer long-term VR natural experiences.

Participants’ physiological stress, including partial indicators of HRV

and stabilized BP, improved aer long-term exposure to VR natural

experiences. VR natural experiences could potentially ameliorate

factory workers’ stress levels.

is study found that VR natural experiences had positive eects

on psychological stress of factory workers, including distress,

depression, anxiety, and positive aect. ese results are in line with

previous studies and demonstrate the eects of VR natural experiences

on alleviating psychological stress (Li etal., 2021; Calogiuri et al.,

2022; Spangenberger etal., 2022) and that it is a feasible approach

which can beapplied during break time in the workplace, particular

in manufacturing factory settings. However, VR natural experiences

did not have an eect on perceived stress. A possible reason might

be that other social stressors exist, such as job content, family

situations, social dynamics among colleagues, and so on, that were

potentially aecting perceived stress (World Health Organization,

2020). Both the work content and work context need to beassessed;

for example, the work context includes career development, economic

and payment issues, role in the organization, interpersonal

relationships, organizational culture, and work-life balance (Wor ld

Health Organization, 2020; Sanchez-Gomez et al., 2021). Future

studies can consider evaluating other stressors to exclude stressed-out

participants in order to examine the eect of VR natural experiences

on perceived stress.

In addition, this study also found VR natural experiences had

positive impacts on the physiological stress of factory workers,

including the SDNN, LF, and HF. e SDNN is one of the important

indicators of ANS functions which presents overall physiological

stress measured by the HRV (Kim etal., 2018). On the other hand,

participants in the experimental group had stable SBP and DBP aer

the intervention, while participants in the comparison group had

increased SBP and DBP aer 12 weeks. is is a relatively less

explored area, and more biofeedback and physiological measures are

encouraged to bemanaged by VR nature interventions. Our study

adopted precise and accurate physiological measures in assessing

ANS function with an evidence-based evaluation to summarize the

eects of VR natural experiences (Francis etal., 2009; Guo etal.,

2022). Study ndings highlighted that changes in physiological

outcomes can beachieved by applying a VR natural experience

intervention in workplace settings which is central to promoting

occupational health. As mentioned, VR natural experiences as a

simulation-based intervention did contribute to psychological and

physiological stress improvement. is study can bea fundamental

TABLE3 Results of the generalized estimating equation (GEE).

Variable VR group (N = 20) Comparison group

(N = 20)

Group

eect Time eect Group × Time

interaction

MD (95% CI) MD (95% CI) p p p

Physiological measures

Distress −1.77 (−0.85, 1.57) 2.07 (−0.72, 4.86) 0.021 0.077 0.015

Depression −1.08 (−5.17, 1.63) 0.50 (−0.51, 1.51) 0.005 0.470 0.062

Anxiety −0.92 (−3.66, 1.50) 1.14 (−1.70, 3.99) 0.039 0.167 0.042

Somatization 0.85 (−2.55, 0.70) 1.71 (−1.44, 4.86) 0.340 0.062 0.452

Positive aect 0.22 (−1.36, 3.05) −0.24 (−0.79, 0.31) 0.035 0.267 0.095

Negative aect 0.19 (−0.54, 0.99) −0.21 (−0.57, 0.16) 0.518 0.371 0.297

Perceived stress −0.71 (−2.40, 0.97) 0.86 (−1.27, 2.98) 0.475 0.315 0.116

Physiological measures

SBP (mmHg) 0.88 (−5.91, 7.66) 9.73 (−0.45, 0.84) 0.357 0.590 0.007

DBP (mmHg) 0.13 (−7.65, 7.90) 2.47 (3.18, 16.29) 0.165 0.031 <0.001

HR (bpm) −1.93 (−0.70, 5.63) 3.63 (−8.69, 15.94) 0.883 0.404 0.393

SDNN (ms) 6.59 (−5.98, 2.12) −8.10 (−34.22, 18.02) 0.030 0.587 0.184

TP (Ln (ms2)) 0.53 (0.13, 0.92) −0.13 (−1.41, 1.15) 0.585 0.011 0.545

LF (Ln (ms2)) 0.70 (0.15, 1.25) −0.13 (−1.16, 0.89) 0.461 0.001 0.041

HF (Ln (ms2)) 0.66 (0.03, 1.28) −0.36 (−1.74, 1.02) 0.279 0.006 0.028

LF/HF (Ln (ratio)) 0.05 (−0.44, 0.53) 0.23 (−0.40, 0.86) 0.473 0.534 0.933

e GEE was adjusted for participants’ age and score at the baseline. e statistical signicance level is set at 0.05 and the value of statistical signicance is emphasized in bold. VR, virtual reality;

MD, mean dierences between pre-and post-test scores; CI, condence interval; SBP, systolic blood pressure; DBP, diastolic blood pressure; HR, heart rate; SDNN, standard deviation of all

normal-to-normal intervals; TP, total power (0–0.5 Hz); LF, low-frequency power (0.04–0.15 Hz); HF, high-frequency power (0.15–0.40 Hz); LF/HF, the ratio of low frequency to high frequency.

104

Ho et al. 10.3389/fpsyg.2023.993143

Frontiers in Psychology 07 frontiersin.org

work in the component of simulation of perception in the simulation

theory. Future studies and interventions are warranted to investigate

the impacts of the simulation-based intervention on other

components of the simulation theory, namely simulation of behavior

and anticipation (Hesslow, 2012). It is believes that the benets of VR

natural experiences can go beyond the psychological and

physiological stress improvement, for example, the enhancement in

working memory, motor, sensory and cognitive function may

be observed as well. In our study, the VR natural experience

intervention was only implemented during break time at a factory for

30 min per session. us, the frequency and length were reasonable

and feasible, and did not occupy much time during their break.

Wesuggest that employers could provide VR natural experience

headsets in the break space or conference room in the factory so that

factory workers can have options to relieve their tense mood and

improve physiological stress during their break time and have a better

occupational health status.

To our best knowledge, this is the rst study to explore the eects

of natural experiences via VR devices on psychological and

physiological stress for occupational health. However, there were

several limitations. e study design was a clustered RCT, instead of

an RCT. A few variables of participant’s backgrounds signicantly

diered between the experimental and comparison groups, although

most stress-related outcomes at the baseline did not dier between the

two groups. e diverse population might have inuenced the

interpretation of results. e small sample size and loss of several

participants aer 12 weeks were also a problem. e 360° video might

have been not clear enough compared to the real world. During the

intervention, participants who complained about VR sickness might

have missed several parts of the videos because they closed their eyes

to rest. Finally, the activities of participants in the comparison group

during the break time were not monitored, which might have

generated a bias.

is study has implications for factory employers and occupational

healthcare professionals such as public health nurses and general

practitioners, and informs future studies for developing relevant

interventions. Manufacturing industry employers should pay attention

to employees’ occupational health, especially stress. Planning regular

break time in relaxed and comfortable places is important for

employees’ rest and further work ecacy. VR is an interesting

platform that can provide an opportunity to connect with nature,

activities, and games. Weekly real-world and VR-based natural

experiences are both recommended for stressed workers. However,

when using VR devices, the image resolution of 360° videos is still a

problem that might cause VR sickness that should beovercome by

future technological advances. For future studies, a study design of

RCTs might provide stronger evidence. Increasing the sample size and

decreasing dierences in participants’ backgrounds between groups

are important considerations. e intervention can beconducted

more frequently and longer, for example, two or three times a week for

6 months. Diverse natural environments, such as forests, waterfalls,

and mountains, can be recorded in a 360° format to increase

enjoyment and attraction. Besides the visual and auditory stimulation

from VR, olfactory stimulation could be considered in

future interventions.

VR is an innovative opportunity to bring the natural environment

into an indoor environment. VR natural experiences can provide

similar eects as real-world natural experiences for relaxation. is

study indicated that VR natural experiences had positive eects on

furniture factory workers’ psychological and physiological stress aer

a 12 week intervention. VR natural experiences are recommended to

release stress and promote the occupational health of factory workers

and heavy labors.

Data availability statement

e datasets generated during and analysed during the current

study are available from the corresponding author on reasonable request.

Ethics statement

e studies involving human participants were reviewed and

approved by Taiwan Medical University-Joint Institutional Review

Board (N202103114). e patients/participants provided their written

informed consent to participate in this study.

Author contributions

M-HH: formal analysis and writing—review and editing. M-SW:

resources and data curation. H-YY: formal analysis, conceptualization,

supervision, and writing—review and editing. All authors contributed

to the article and approved the submitted version.

Funding

is work was supported by the Ministry of Science and

Technology, Taiwan under grant (MOST 109-2314-B-038-077-MY3).

Conﬂict of interest

e authors declare that the research was conducted in the

absence of any commercial or nancial relationships that could

beconstrued as a potential conict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors

and do not necessarily represent those of their aliated organizations,

or those of the publisher, the editors and the reviewers. Any product

that may be evaluated in this article, or claim that may be made by its

manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

e Supplementary material for this article can befound online

at: https://www.frontiersin.org/articles/10.3389/fpsyg.2023.993143/

full#supplementary-material

105

Ho et al. 10.3389/fpsyg.2023.993143

Frontiers in Psychology 08 frontiersin.org

References

Adhyaru, J. S., and Kemp, C. (2022). Virtual reality as a tool to promote wellbeing

in the workplace. Digit. Health 8:20552076221084473. doi:

10.1177/20552076221084473

Asplund, S., Åhlin, J., Åström, S., and Lindgren, B. M. (2022). Experiences of work-

related stress among highly stressed municipal employees in rural northern Sweden. Int.

J. Qual. Stud. Health Well-being 17:2056957. doi: 10.1080/17482631.2022.2056957

Bhui, K., Dinos, S., Galant-Miecznikowska, M., de Jongh, B., and Stansfeld, S. (2016).

Perceptions of work stress causes and eective interventions in employees working in

public, private and non-governmental organisations: a qualitative study. BJPsych Bull.

40, 318–325. doi: 10.1192/pb.bp.115.050823

Bolliger, L., Lukan, J., Colman, E., Boersma, L., Luštrek, M., De Bacquer, D., et al.

(2022). Sources of occupational stress among oce workers-a focus group study. Int. J.

Environ. Res. Public Health 19:1075. doi: 10.3390/ijerph19031075

Browning, M., Mimnaugh, K. J., van Riper, C. J., Laurent, H. K., and LaValle, S. M.

(2019). Can simulated nature support mental health? Comparing short, single-doses of

360-degree nature videos in virtual reality with the outdoors. Front. Psychol. 10:2667.

doi: 10.3389/fpsyg.2019.02667

Calogiuri, G., Keegan, B. J., Birkheim, S. L., Rydgren, T. L., Flaten, O. E., Fröhlich, F.,

et al. (2022). A mixed-methods exploration of virtual reality as a tool to promote green

exercise. Sci. Rep. 12:5715. doi: 10.1038/s41598-022-09622-x

Cohen, S., Kamarck, T., and Mermelstein, R. (1983). A global measure of perceived

stress. J. Health Soc. Behav. 24, 385–396. doi: 10.2307/2136404

Coventry, P. A., Brown, J. E., Pervin, J., Brabyn, S., Pateman, R., Breedvelt, J., et al.

(2021). Nature-based outdoor activities for mental and physical health: systematic

review and meta-analysis. SSM Popul. Health 16:100934. doi: 10.1016/j.

ssmph.2021.100934

Faul, F., Erdfelder, E., Lang, A. G., and Buchner, A. (2007). G*power 3: a exible

statistical power analysis program for the social, behavioral, and biomedical sciences.

Behav. Res. Methods 39, 175–191. doi: 10.3758/bf03193146

Francis, J. L., Weinstein, A. A., Krantz, D. S., Haigney, M. C., Stein, P. K., Stone, P. H.,

et al. (2009). Association between symptoms of depression and anxiety with heart rate

variability in patients with implantable cardioverter debrillators. Psychosom. Med. 71,

821–827. doi: 10.1097/PSY.0b013e3181b39aa1

Guo, T., Guo, H., Fu, L., Chen, X., Ma, Y., Pang, B., et al. (2022). e modifying eect

of trait anxiety on the association of ne particulate matter with heart rate variability

variables. Int. J. Hyg. Environ. Health 241:113933. doi: 10.1016/j.ijheh.2022.

113933

Hartig, T., Mitchell, R., de Vries, S., and Frumkin, H. (2014). Nature and health. Annu.

Rev. Public Health 35, 207–228. doi: 10.1146/annurev-publhealth-032013-182443

Hesslow, G. (2012). e current status of the simulation theory of cognition. Brain

Res. 1428, 71–79. doi: 10.1016/j.brainres.2011.06.026

Kim, H. G., Cheon, E. J., Bai, D. S., Lee, Y. H., and Koo, B. H. (2018). Stress and heart

rate variability: a meta-analysis and review of the literature. Psychiatry Investig. 15,

235–245. doi: 10.30773/pi.2017.08.17

Kim, J., and Jung, H. S. (2022). e eect of employee competency and organizational

culture on employees' perceived stress for better workplace. Int. J. Environ. Res. Public

Health 19:4428. doi: 10.3390/ijerph19084428

Li, H., Zhang, X., Wang, H., Yang, Z., Liu, H., Cao, Y., et al. (2021). Access to nature

via virtual reality: a mini-review. Front. Psychol. 12:725288. doi: 10.3389/

fpsyg.2021.725288

Lüddecke, R., and Felnhofer, A. (2022). Virtual reality biofeedback in health: a scoping

review. Appl. Psychophysiol. Biofeedback 47, 1–15. doi: 10.1007/s10484-021-09529-9

Malik, M., Bigger, J. T., Camm, A. J., Kleiger, R. E., Malliani, A., Moss, A. J., et al.

(1996). Heart rate variability: standards of measurement, physiological interpretation,

and clinical use. Eur. Heart J. 17, 354–381. doi: 10.1093/oxfordjournals.eurheartj.

a014868

Mohamed, A. F., Isahak, M., Awg Isa, M. Z., and Nordin, R. (2022). e eectiveness

of workplace health promotion program in reducing work-related depression, anxiety

and stress among manufacturing workers in Malaysia: mixed-model intervention. Int.

Arch. Occup. Environ. Health 95, 1113–1127. doi: 10.1007/s00420-022-01836-w

Owens, M., and Bunce, H. L. I. (2022). e potential for outdoor nature-based

interventions in the treatment and prevention of depression. Front. Psychol. 13:740210.

doi: 10.3389/fpsyg.2022.740210

Picton, C., Fernandez, R., Moxham, L., and Patterson, C. F. (2020). Experiences of

outdoor nature-based therapeutic recreation programs for persons with a mental illness:

a qualitative systematic review. JBI Evid. Synth. 18, 1820–1869. doi: 10.11124/

jbisrir-d-19-00263

Sanchez-Gomez, M., Giorgi, G., Finstad, G. L., Alessio, F., Ariza-Montes, A.,

Arcangeli, G., et al. (2021). Economic stress at work: its impact over absenteeism and

innovation. Int. J. Environ. Res. Public Health 18:5265. doi: 10.3390/ijerph18105265

Sorensen, G., Dennerlein, J. T., Peters, S. E., Sabbath, E. L., Kelly, E. L., and

Wagner, G. R. (2021). e future of research on work, safety, health and wellbeing: a

guiding conceptual framework. Soc. Sci. Med. 269:113593. doi: 10.1016/j.

socscimed.2020.113593

Spangenberger, P., Geiger, S. M., and Freytag, S. C. (2022). Becoming nature: eects

of embodying a tree in immersive virtual reality on nature relatedness. Sci. Rep. 12:1311.

doi: 10.1038/s41598-022-05184-0

Sznajder, K. K., Harlow, S. D., Wang, J., Tso, L., Ashagre, Y., and Han, C. (2022).

Factors associated with symptoms of poor mental health among women factory workers

in China's supply chain. Int. Arch. Occup. Environ. Health 95, 1209–1219. doi: 10.1007/

s00420-021-01820-w

Terluin, B., Unalan, P. C., Turfaner Sipahioğlu, N., Arslan Özkul, S., and van

Marwijk, H. W. (2016). Cross-cultural validation of the Turkish four-dimensional

symptom questionnaire (4DSQ) using dierential item and test functioning (DIF and

DTF) analysis. BMC Fam. Pract. 17:53. doi: 10.1186/s12875-016-0449-4

Tseng, T. H., Chen, H. C., Wang, L. Y., and Chien, M. Y. (2020). Eects of exercise

training on sleep quality and heart rate variability in middle-aged and older adults with

poor sleep quality: a randomized controlled trial. J. Clin. Sleep Med. 16, 1483–1492. doi:

10.5664/jcsm.8560

Watson, D., Clark, L. A., and Tellegen, A. (1988). Development and validation of brief

measures of positive and negative aect: the PANAS scales. J. Pers. Soc. Psychol. 54,

1063–1070. doi: 10.1037//0022-3514.54.6.1063

World Health Organization. (2020). Occupational health: Stress at the workplace

Available at: https://www.who.int/news-room/questions-and-answers/item/ccupational-

health-stress-at-the-workplace

106

+41 (0)21 510 17 00

frontiersin.org/about/contact

Avenue du Tribunal-Fédéral 34

1005 Lausanne, Switzerland

frontiersin.org

Frontiers

Paving the way for a greater understanding of

human behavior

The most cited journal in its field, exploring

psychological sciences - from clinical research to

cognitive science, from imaging studies to human

factors, and from animal cognition to social

psychology.

Discover the latest

Research Topics

Frontiers in

Psychology

ResearchGate has not been able to resolve any citations for this publication.

The Effect of Employee Competency and Organizational Culture on Employees’ Perceived Stress for Better Workplace

Article

Full-text available

Apr 2022
Int J Environ Res Publ Health

Although the meaningful relationship between organizational culture and employee performance is a widely-researched topic, there is scant research available how organizational culture impacts on employees’ perceived stress in the workplace, affecting their performance. This might cause a difficulty to guide practitioners as to what organizational elements can be applied to reduce employee perceived stress. To add the level of robustness and fill the gap in the literature, the present research explores the effect of organizational culture with employee competency on workers’ perceived stress which has emerged as a common occupational disease and affected employees psychologically and physically; thus, affecting their performance. Using 641 responses, the statistical findings of the present research insists that HR practitioners should match the type of organizational culture and employee competency situationally to reduce employee stress. The current authors suggest that organizations desiring the adaptability competencies (Professional competency) for their employees should build a clan organizational culture. In contrast, organizations should encourage a market organizational culture for their employees who possess customer orientation competency (Simply result-oriented competency). The research outcomes provide additional knowledge to the existing literature, enhance academicians’ understanding of the research topic, and serve as a significant knowledge base for further empirical research.

Experiences of work-related stress among highly stressed municipal employees in rural northern Sweden

Article

Full-text available

Mar 2022

Purpose The aim of this qualitative study was to describe experiences of work-related stress among highly stressed municipal employees in rural northern Sweden. Methods We interviewed 15 employees in the municipal sector in rural northern Sweden using a semi-structured guide and subjected the interviews to qualitative content analysis. Results Under the main theme of Suffering Through Endless Chaos, we summarized four themes: facing incompatible interests and high demands due to lack of time and resources; feeling powerless, trapped, and ignored due to lack of control; feeling insufficient, insecure, and guilty due to challenging relations and high expectations; and struggling with consequences such as health problems, spillover effects on family life, and difficulty coping. Conclusion Findings from this study suggest the importance of acknowledging suffering among municipal employees in a stressful work environment. An imbalance between job demands and resources is affecting both health and family lives of employees, and also their ability to work. It seems important to improve the work environment through supportive leadership, job control, and reasonable job demands to prevent stress, reduce suffering, and create a healthy organization.

The Potential for Outdoor Nature-Based Interventions in the Treatment and Prevention of Depression

Article

Full-text available

Mar 2022

There is growing interest in nature-based interventions (NBI) to improve human health and wellbeing. An important nascent area is exploring the potential of outdoor therapies to treat and prevent common mental health problems like depression. In this conceptual analysis on the nature–depression nexus, we distil some of the main issues for consideration when NBIs for depression are being developed. We argue that understanding the mechanisms, or ‘active ingredients’ in NBIs is crucial to understand what works and for whom. Successfully identifying modifiable mediating intervention targets will pave the way for interventions with increased efficacy. We highlight a non-exhaustive list of five clinically relevant putative, candidate mechanisms which may underly the beneficial effects of NBIs on depression: stress, rumination, mindfulness, sleep and exercise. We also make the case that when developing NBIs it is important to not neglect young people, explore personalised approaches and focus on both treatment and prevention approaches. To achieve these aims methodologically rigorous programmes of clinical research are needed that include well-powered and controlled experimental designs including randomised controlled trials, qualitative research, longitudinal studies and large prospective cohorts.

A mixed-methods exploration of virtual reality as a tool to promote green exercise

Article

Full-text available

Apr 2022

The salutogenic effects of green exercise are widely recognised, yet many individuals do not engage in this health-related behaviour. Using a convergent mixed methods approach, this study explored the impact of Immersive Virtual Nature (IVN) on the decision-making process relating to green exercise. Three experimental trials were conducted (overall n = 136), in which healthy adults were exposed to different types of IVN reproducing an existing urban green area. Participants reported medium-high rating of intent to visit the location. Significant pre-to-post increments in future green exercise intention were observed after the IVN exposure, though a significance difference was not achieved in comparison with control. Qualitative analysis revealed the impact of IVNs on behaviour regulation, and highlighted the pivotal role of anticipated emotional benefits. Despite scepticism regarding the IVN experience, it was effective in arousing curiosity to explore natural environments, which was associated with environmental perceptions, nostalgic and socio-cultural perspectives.

Implementing New Technologies to Improve Visual–Spatial Functions in Patients with Impaired Consciousness

Article

Full-text available

Mar 2022
Int J Environ Res Publ Health

The quality of life of patients with severe brain damage is compromised by, e.g., impaired cognitive functions and ocular dysfunction. The paper contains research findings regarding participants of an oculomotor training course aimed at the therapy of visual–spatial functions. Five male patients with brain damage who did not communicate, verbally or motorically, participated in the study. Over a six-week period, the subjects solved tasks associated with recognising objects, size perception, colour perception, perception of object structures (letters), perception of object structures (objects), detecting differences between images and assembling image components into the complete image with the use of an eye tracker. The findings present evidence of oculomotor training effectiveness based on a longer duration of the work with the eye tracker and improved visual–spatial functions.

Virtual reality as a tool to promote wellbeing in the workplace

Article

Full-text available

Mar 2022

Background NHS staff are disproportionately impacted by workplace stress, threatening not only quality of service delivery, but the sustainability, of this vital healthcare system. There is an urgent need for accessible and cost-effective interventions that promote the wellbeing of this integral workforce. Aim The aim of this study was to explore the potential propensity of a short virtual reality (VR) nature experience, delivered during the workday, to induce positive mood states NHS clinicians. Method NHS clinicians working in a fast-paced trauma service were offered the opportunity to take part in a short virtual reality (VR) relaxation session, during their working day, in which they were able to explore the “Green Meadows” experience available within the Nature Treks application. An indication of physiological arousal (heart rate) was obtained, and subjective measures of emotional state were employed to assess the effect of the intervention on participants’ mood. Further, feedback was gathered to provide an initial indication of the acceptability of the experience. Results Analysis of data from 39 clinicians revealed that, following the VR experience, participants reported significantly increased feelings of happiness ( p < 0.001) and relaxation ( p < 0.001), and significantly decreased feelings of sadness ( p = 0.003), anger ( p < 0.001) and anxiety ( p < 0.001). No significant differences in reported levels of surprise ( p = 0.603) or vigour ( p = 0.566) were found. Further, the experience was associated with a significant reduction in heart-rate ( p = 0.025), and had high acceptability ratings, despite participants’ varied experience with computers and VR technology. Conclusions The findings of this study indicate that the administration of VR for the promotion of NHS staff wellbeing in the workplace is a potentially fruitful avenue of exploration that warrants further investigation.

Dynamics of Functional Network Organization Through Graph Mixture Learning

Article

Full-text available

Feb 2022
NEUROIMAGE

Understanding the organizational principles of human brain activity at the systems level remains a major challenge in network neuroscience. Here, we introduce a fully data-driven approach based on graph learning to extract meaningful repeating network patterns from regionally-averaged timecourses. We use the Graph Laplacian Mixture Model (GLMM), a generative model that treats functional data as a collection of signals expressed on multiple underlying graphs. By exploiting covariance between activity of brain regions, these graphs can be learned without resorting to structural information. To validate the proposed technique, we first apply it to task fMRI with a known experimental paradigm. The probability of each graph to occur at each time-point is found to be consistent with the task timing, while the spatial patterns associated to each epoch of the task are in line with previously established activation patterns using classical regression analysis. We further on apply the technique to resting state data, which leads to extracted graphs that correspond to well-known brain functional activation patterns. The GLMM allows to learn graphs entirely from the functional activity that, in practice, turn out to reveal high degrees of similarity to the structural connectome. The Default Mode Network (DMN) is always captured by the algorithm in the different tasks and resting state data. Therefore, we compare the states corresponding to this network within themselves and with structure. Overall, this method allows us to infer relevant functional brain networks without the need of structural connectome information. Moreover, we overcome the limitations of windowing the time sequences by feeding the GLMM with the whole functional signal and neglecting the focus on sub-portions of the signals.

An Interplay Between Post-Traumatic Epilepsy and Associated Cognitive Decline: A Systematic Review

Article

Full-text available

Feb 2022

Background Post-traumatic epilepsy (PTE) is a devastating neurological outcome of traumatic brain injury (TBI), which may negatively impact the quality of life of patients with TBI, and may impose a huge socioeconomic burden. This burden may be due to long-term functional outcomes associated with PTE, particularly cognitive dysfunction. To date, the relationship between TBI and PTE remains unclear, with little known about how the effect of their link on cognitive function as well. Objective Thus, this systematic review aimed at elucidating the relationship between PTE and cognitive impairment in adults after TBI based on available clinical studies, in hopes to aid in the development of therapeutic strategies for PTE. Methods A systematic literature search was performed using 6 databases; MEDLINE, Embase, CINAHL, Psych INFO, Web of Science, and Cochrane to retrieve relevant clinical studies investigating the link between PTE and cognition in the context of TBI. The Newcastle-Ottawa Scale (NOS) was used to assess the methodological quality of relevant studies. Results A total of six eligible studies were included for critical appraisal in this review after performing the inclusion and exclusion criteria, which involved 1,100 individuals, from 1996 to 2021. The selected studies were derived from the civilian and military population, with a follow-up period that ranged from 6 months to 35 years. The average quality of the involved studies was moderate (6.6, SD = 1.89). Five out of six studies found poorer cognitive performance in people with PTE, compared with those without PTE. Although the association between PTE and cognitive impairment was insignificant after controlling for specific covariates, there was a statistical trend toward significance. Conclusion This systematic review suggests that there may be a possible link between PTE and cognitive decline in TBI patients, with the latter being reported to occur up to 35 years post injury. Variations in sample sizes, follow-up periods, and neuropsychological assessment tools may be the limitations affecting the interpretation and significance of this relationship. Therefore, future studies with standard cognitive assessment tools may be warranted to solidify the link between TBI-PTE-cognitive dysfunction, prior to the development of therapeutic strategies. Systematic Review Registration: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020221702, prospero identifier: CRD42020221702.

Online Psychotherapy During the COVID-19 Pandemic: The Good, the Bad, and the Ugly

Article

Full-text available

Feb 2022

Exploring literacy engagement in a significant disability context

Article

May 2022
J Early Child Literacy

This study tackles the question: how is literacy engagement enacted in the context of significant disability? We delve into the complex literacy practices of Kalika, a three-year-old child with Rett syndrome, a rare neurodevelopmental disorder, to elucidate how she engages with printed text. Rett syndrome leads to near total loss of verbal communication and limited functional hand use, making it particularly challenging to participate in traditionally recognized forms of literacy engagement. Using in-depth qualitative data from both in- and out-of-school settings, we conduct a micro-level analysis of Kalika’s behaviours during story time rituals. In order to bring analytic coherence to the data, we classified her modalities of literacy engagement under two broad categories: 1) kinesics, which included a) corporal (entailing full body positioning and motion), b) oral (involving contact with mouth or expression), c) oculesics (relating to eye gaze), and d) haptic (relating to hands) elements as well as 2) vocalics (pertaining to vocal tone and vocalisation). Our analysis elucidates the sophisticated, complex multimodal practices that Kalika enacts to engage with texts. For far too long, students with significant disabilities have been viewed from deficit perspectives, neglected within the literature as well as in the classroom, and thought to require additional instruction to learn how to engage with texts. We suggest that perhaps it is a question, instead, of educators and scholars learning to expand their own frames of reference.

Methods and applications in psychology for clinical settings

Recommended publications

Assessment of Attention in Children with Attention Deficit Hyperactivity Problems in Primary Care Se...

Promoting psychological flexibility in clinical settings

What's wrong with medicalising behaviour?

Setting up a group for adults with a learning disability who have been bullied