Fig 2 - uploaded by Won-Seok Kim
Content may be subject to copyright.
Example of unrecognizable hyoid bone located in the square in black. A. The hyoid bone moves too fast, resulting in the almost equal grey scale value in the area around it. B. The strong reflective light makes the hyoid bone invisible.
Source publication
Motion analysis of the hyoid bone via videofluoroscopic study has been used in clinical research, but the classical manual tracking method is generally labor intensive and time consuming. Although some automatic tracking methods have been developed, masked points could not be tracked and smoothing and segmentation, which are necessary for functiona...
Contexts in source publication
Context 1
... described by the previous study [24], the y-axis of the patient-centric coordinate system is defined as a straight line connecting the anterior- interior border of the fourth cervical vertebra to that of the second cervical vertebra. The x- axis is defined as a line vertical to the y-axis crossing the origin, C4, as seen in Fig 3B. The points C4ðx c 4 ; y c 4 Þ and C2ðx c 2 ; y c 2 Þ can be tracked at the same time using the methods illus- trated in the previous subsections over the entire video sequences (Fig 3A). ...
Context 2
... x- axis is defined as a line vertical to the y-axis crossing the origin, C4, as seen in Fig 3B. The points C4ðx c 4 ; y c 4 Þ and C2ðx c 2 ; y c 2 Þ can be tracked at the same time using the methods illus- trated in the previous subsections over the entire video sequences (Fig 3A). To reduce the tracking errors, smoothing is carried out for both the target point in the hyoid bone and the two tracking points in the cervical vertebra using a cubic smoothing spline. ...
Context 3
... reduce the tracking errors, smoothing is carried out for both the target point in the hyoid bone and the two tracking points in the cervical vertebra using a cubic smoothing spline. The degree of smoothing is controlled by the smoothing parameter, which can be adjusted by the operator to avoid over-fitting (Fig 3D). Then all the data is calibrated by the vertical distance from C4 to C2. ...
Citations
... For comparison with previous studies [21,22,28], the root mean squared error (RMSE) of predicted trajectory of HBE was calculated. The range of motion (ROM) of HBE, defined as the maximal distance of any of the two points of the HBE trajectory, was measured and the differences between predicted values (ROM Predicted ) and ground truth (ROM ground truth ) were compared by calculating the relative errors of ROM (%) = (ROM Predicted -ROM ground truth )/ ROM ground truth × 100%. ...
... Because our algorithm localizes a single position point of the hyoid bone, the results of detection accuracy could not be directly compared with most of previous CNN-based algorithms using a bounding box method [18,21,22,28]. These studies used mean average precision, the area below the precision-recall curve, to estimate the detection accuracy. ...
... Three studies provided the RMSE and/or relative errors of ROM to evaluate prediction accuracy [21,22,28]. Our algorithm showed an RMSE of 3.05 ± 2.29 mm using transformed coordinates and an RMSE of 1.64 ± 1.34 using untransformed coordinates. ...
The hyoid bone excursion is one of the most important gauges of larynx elevation in swallowing, contributing to airway protection and bolus passage into the esophagus. However, the implications of various parameters of hyoid bone excursion, such as the horizontal or vertical displacement and velocity, remain elusive and raise the need for a tool providing automatic kinematics analysis. Several conventional and deep learning-based models have been applied automatically to track the hyoid bone, but previous methods either require partial manual localization or do not transform the trajectory by anatomic axis. This work describes a convolutional neural network-based algorithm featuring fully automatic hyoid bone localization and tracking and spine axis determination. The algorithm automatically estimates the hyoid bone trajectory and calculates several physical quantities, including the average velocity and displacement in horizontal or vertical anatomic axis. The model was trained in a dataset of 365 videos of videofluoroscopic swallowing from 189 patients in a tertiary medical center and tested using 44 videos from 44 patients with different dysphagia etiologies. The algorithm showed high detection rates for the hyoid bone. The results showed excellent inter-rater reliability for hyoid bone detection, good-to-excellent inter-rater reliability for calculating the maximal displacement and the average velocity of the hyoid bone in horizontal or vertical directions, and moderate-to-good reliability in calculating the average velocity in horizontal direction. The proposed algorithm allows for complete automatic kinematic analysis of hyoid bone excursion, providing a versatile tool with high potential for clinical applications.
... Data misalignment is also a common issue in FDA. For example, in the motion analysis of the hyoid bone of stroke patients [13] discussed in Section 4, when the trajectories of the hyoid bone movement during swallowing are recorded, peaks and valleys for phases of the trajectories include elevation, anterior, remain and return occur at different times for different individuals. A large fraction of the variability in a sample of trajectories is then best explained as time variation [21]. ...
... In this study, a total of 30 subjects' data containing two groups, i.e. one for normal people and the other for patients after stroke, are obtained from X-ray video clips in the VFS swallowing study. Then the trajectories of hyoid bone movement are obtained by using a semi-automatic programme proposed in [13], the detailed description of the procedures can be found in [36]. The trajectories are considered to be useful in classifying the dysphagia, predicting the prognosis or assessing the treatment effects. ...
In this paper, we consider the problem of classification of misaligned multivariate functional data. We propose to use a model-based approach for the joint registration and classification of such data. The observed functional inputs are modeled as a functional nonlinear mixed effects model containing a nonlinear functional fixed effect constructed upon warping functions to account for curve alignment, and a nonlinear functional random effects component to address the variability among subjects. The warping functions are also modeled to accommodate common effect within groups and the variability between subjects. Then, a functional logistic regression model defined upon the representation of the aligned curves and scalar inputs is used to account for curve classification. EM-based algorithms are developed to perform maximum likelihood inference of the proposed models. The identifiability of the registration model and the asymptotical properties of the proposed method are established. The performance of the proposed procedure is illustrated via simulation studies and an analysis of a hyoid bone movement data application. The statistical developments proposed in this paper were motivated by the hyoid bone movement study, the methodology is designed and presented generality and can be applied to numerous areas of scientific research.
... The importance of VFSS has stimulated many researchers to develop automated approaches to support clinicians in the interpretation of the recorded video, for example, by detecting structures and anatomical points of interest [17], [21], [32], tracking hyoid bone movements [18], [20], [22], [33]- [35], segmenting the bolus during swallow events [15], and detecting the pharyngeal phase from the raw videos [12]- [14]. The advent of deep learning has considerably advanced the state of the art of this field. ...
... The advent of deep learning has considerably advanced the state of the art of this field. In fact, approaches for tracking hyoid bone movement, which were semi-automatic until a few years ago [22], [33]- [35], have recently became fully automated with improved detection accuracies [18], [20]. This was possible due to the availability of powerful object detection neural networks such as Faster R-CNN [36] and YOLO [37]. ...
The videofluoroscopic swallowing study (VFSS) is a gold-standard imaging technique for assessing swallowing, but analysis and rating of VFSS recordings is time consuming and requires specialized training and expertise. Researchers have demonstrated that it is possible to automatically detect the pharyngeal phase of swallowing and to localize the bolus in VFSS recordings via computer vision approaches, fostering the development of novel techniques for automatic VFSS analysis. However, training of algorithms to perform these tasks requires large amounts of annotated data that are seldom available. We demonstrate that the challenges of pharyngeal phase detection and bolus localization can be solved together using a single approach. We propose a deep-learning framework that jointly tackles pharyngeal phase detection and bolus localization in a weakly-supervised manner, requiring only the initial and final frames of the pharyngeal phase as ground truth annotations for the training. Our approach stems from the observation that bolus presence in the pharynx is the most prominent visual feature upon which to infer whether individual VFSS frames belong to the pharyngeal phase. We conducted extensive experiments with multiple convolutional neural networks (CNNs) on a dataset of 1245 VFSS clips from 59 healthy subjects. We demonstrated that the pharyngeal phase can be detected with an F1-score higher than 0.9. Moreover, by processing the class activation maps of the CNNs, we were able to localize the bolus with promising results, obtaining correlations with ground truth trajectories higher than 0.9, without any manual annotations of bolus location used for training purposes. Once validated on a larger sample of participants with swallowing disorders, our framework will pave the way for the development of intelligent tools for VFSS analysis to support clinicians in swallowing assessment.
... Paik et al. reported that the extent and pattern of hyoid movement varies according to the etiology of dysphagia in two-dimensional kinetic swallowing motion analysis and suggested its applicability in differentiating the mechanism of dysphagia and treatment for reversing the mechanism [10]. However, in a clinical setting, manual tracking and quantitative measurement of hyoid bone movement is a labor-intensive and time-consuming task [13]. Moreover, the hyoid bone usually has unclear margins and varies in shape for each person. ...
... Therefore, manual tracking of the hyoid bone is inevitably prone to human error due to fatigue and individual subjective judgment, and wide intrarater and interrater variation has been shown [14]. Automatic tracking models of hyoid motion have been used to reduce human error and workload, and computer-assisted methods for kinematic analysis of hyoid bone movement have been proposed in a few studies [13,[15][16][17]. However, these semiautomatic methods still require human judgment and manual input from clinicians and have a limitation of low performance and efficiency for application in clinical settings [13,[15][16][17]. ...
... Automatic tracking models of hyoid motion have been used to reduce human error and workload, and computer-assisted methods for kinematic analysis of hyoid bone movement have been proposed in a few studies [13,[15][16][17]. However, these semiautomatic methods still require human judgment and manual input from clinicians and have a limitation of low performance and efficiency for application in clinical settings [13,[15][16][17]. ...
Kinematic analysis of the hyoid bone in a videofluorosopic swallowing study (VFSS) is important for assessing dysphagia. However, calibrating the hyoid bone movement is time-consuming, and its reliability shows wide variation. Computer-assisted analysis has been studied to improve the efficiency and accuracy of hyoid bone identification and tracking, but its performance is limited. In this study, we aimed to design a robust network that can track hyoid bone movement automatically without human intervention. Using 69,389 frames from 197 VFSS files as the data set, a deep learning model for detection and trajectory prediction was constructed and trained by the BiFPN-U-Net(T) network. The present model showed improved performance when compared with the previous models: an area under the curve (AUC) of 0.998 for pixelwise accuracy, an accuracy of object detection of 99.5%, and a Dice similarity of 90.9%. The bounding box detection performance for the hyoid bone and reference objects was superior to that of other models, with a mean average precision of 95.9%. The estimation of the distance of hyoid bone movement also showed higher accuracy. The deep learning model proposed in this study could be used to detect and track the hyoid bone more efficiently and accurately in VFSS analysis.
... The Pearson correlation coefficient of the x and y-axis between ground truth and the inference of all frames was calculated to provide clinically relevant comparisons between the trackers. The range of motion (ROM) of the hyoid bone was calculated from hyoid onset to maximum displacement before hyoid offset, which represents the maximum elevation and anterior displacement of the hyoid during swallowing [49]. Furthermore, the relative error of ROM was calculated using Equation (5). ...
... Good correlations were shown between ground truth and inference location with a Pearson's correlation coefficient of 0.985 ± 0.013 and 0.919 ± 0.034 on the x and y-axis, respectively. The relative error of ROM was 9.5% ± 6.1%, compared to the relative error of 3.3% to 9.2% reported in the previous study [49]. ...
(1) Background: Ultrasound provides a radiation-free and portable method for assessing swallowing. Hyoid bone locations and displacements are often used as important indicators for the evaluation of swallowing disorders. However, this requires clinicians to spend a great deal of time reviewing the ultrasound images. (2) Methods: In this study, we applied tracking algorithms based on deep learning and correlation filters to detect hyoid locations in ultrasound videos collected during swallowing. Fifty videos were collected from 10 young, healthy subjects for training, evaluation, and testing of the trackers. (3) Results: The best performing deep learning algorithm, Fully-Convolutional Siamese Networks (SiamFC), proved to have reliable performance in getting accurate hyoid bone locations from each frame of the swallowing ultrasound videos. While having a real-time frame rate (175 fps) when running on an RTX 2060, SiamFC also achieved a precision of 98.9% at the threshold of 10 pixels (3.25 mm) and 80.5% at the threshold of 5 pixels (1.63 mm). The tracker’s root-mean-square error and average error were 3.9 pixels (1.27 mm) and 3.3 pixels (1.07 mm), respectively. (4) Conclusions: Our results pave the way for real-time automatic tracking of the hyoid bone in ultrasound videos for swallowing assessment.
... In many situations, however, such as the motion analysis of the hyoid bone of stroke patients (Kim et al., 2017) discussed in Sect. 4, a specific aspect of the curve data is the presence of misaligned problems. ...
... Patients after stroke usually suffer from oropharyngeal dysphagia, and the data obtained from the VFSS could be used to classify the dysphagia, to predict the prognosis or to assess the treatment effects (Kim et al., 2017). A total of 30 subjects' data containing two groups, i.e. one for normal people and the other or patients after stroke, were obtained. ...
Many classification techniques when the data are curves or functions have been recently proposed. However, the presence of misaligned problems in the curves can influence the performance of most of them. In this paper, we propose a model-based approach for simultaneous curve registration and classification. The method is proposed to perform curve classification based on a functional logistic regression model that relies on both scalar variables and functional variables, and to align curves simultaneously via a data registration model. EM-based algorithms are developed to perform maximum likelihood inference of the proposed models. We establish the identifiability results for curve registration model and investigate the asymptotic properties of the proposed estimation procedures. Simulation studies are conducted to demonstrate the finite sample performance of the proposed models. An application of the hyoid bone movement data from stroke patients reveals the effectiveness of the new models.
... Quantitative kinematic analysis for swallowing difficulty is carried out based on the VFSS images containing information of the anatomic and dynamic properties of swallowing [5,6]. Unfortunately, the swallowing kinematic analysis demands labor-intensive and time-consuming processes for the manual marking of the swallowing structures, hence limiting its clinical utility and applicability [7]. ...
... Previously, various methods have been proposed to automatically track the swallowing structures such as the hyoid bone for kinematic analysis through image processing and machine learning algorithms. These algorithms include sobel edge detection [7,8] and active shape matching [9], Haar classifier matching [10] and local binary patterns in the image processing, and recently, convolutional neural networks (CNN) in machine learning [11]. However, in these previously proposed methods, manual corrections of the swallowing kinematic analyses was still required because tracking the hyoid bone is frequently failed when the image contrast is abruptly changed in the duration of passing through the mandible. ...
... However, manual process may accompany with measurement errors as well as inter-and intra-rater variations [12]. It can also be time-consuming and labor-intensive especially in cases with prolonged swallowing duration or high frame rates [7]. These factors have led to limited application of swallowing kinematic analysis in clinical practice. ...
Swallowing difficulty is a major health concern of the elderly population. The gold standard examination to assess swallowing function is videofluoroscopic swallowing study (VFSS). Hyoid kinematic parameters extracted from VFSS images can be quantitative indicators of swallowing difficulty. In previous studies, its tracking failures are still not resolved when passing through the mandible. Furthermore, it is difficult to be applied in kinematic analysis because the hyoid trajectories can be susceptible to irrelevant neck movements during swallowing. The aim of this study is to develop a robust algorithm for obtaining high-accuracy trajectories of the hyoid bone during swallowing with adjustment of the neck movements. We propose a CNN-based hyoid tracking algorithm which consists of single-domain networks for hyoid tracking and an attention U-Net with conditional random fields for semantic segmentation of the hyoid bone and the cervical vertebrae. The results show that the proposed method can track the hyoid bone robustly compared to the previous methods as measured by a success plot of one-pass evaluation. In addition, the proposed semantic segmentation method achieved the highest dice coefficient for the hyoid bone and the cervical vertebrae. Finally, the obtained hyoid trajectories were evaluated by a root mean squared error, relative error of range of motion, and Pearson’s correlation analysis. The proposed algorithm can provide ability to automatically analyze the hyoid motions during swallowing in clinical practice and will potentially enable physician’s decision making on diagnostic and therapeutic modalities based on quantitative swallowing assessments.
... Finally, the latest works on hyoid bone kinematics put forward as more pertinent [28] the measurements of the hyoid bone motion peak. Semi-automatic measurement techniques are being developed [29] to encourage the use of these parameters in the future. ...
Purpose
A personalised transportable folding device for seating (DATP) on a standard seat was developed by an occupational therapist at the Toulouse University Hospital Centre (patent no. WO 2011121249 A1) based on the hypothesis that the use of a seat to assist with better positioning on any chair during meals modifies the sitting posture and has an impact on cervical statics which increases the amplitude of movements of the axial skeleton (larynx and hyoid bone) and benefits swallowing. The aim of this work is to demonstrate that an improvement in sitting posture with the help of the DATP, through Hyoid bone motion, has an impact on the quality of swallowing in a dysphagic population which benefits from the device in comparison to a dysphagic population which does not benefit from the device after 1 month of care. The secondary endpoints concern the evaluation of the impact on other characteristics of swallowing, posture, the acceptability of the device and the quality of life.
Methodology
This is a randomised comparative clinical trial. The blind was not possible for the patients but the examiner who evaluated the outcome criterion was blinded to the group to which the patient belonged. The outcome criterion was the measurement of the hyoid bone movement during swallowing. The other criteria were collected during the videofluoroscopic examination of swallowing and by use of a questionnaire. Fifty-six (56) patients were included: 30 in the group without device (D−) and 26 in the group with the device (D+). All the patients benefited from a training course on seating. Only the D+ patients participated in this course where the use of the device was explained and the device was then kept for use at home for 1 month.
Results
A significant improvement was noted in the postural criteria before and after use, in favour of a better posture for the two groups (p < 0.001) and more hyoid bone motion in the D+ group. The difference was significant in the bivariate analysis for horizontal movement (p = 0.04). After adjustment of potential factors of confusion, we noted a significant mean difference for the three distances in the D+ group in comparison to the D− group, of + 0.33 (95% CI [+ 0.17; + 0.48]) for horizontal movement, + 0.22 (95% CI [+ 0.03; + 0.40]) for vertical movement and + 0.37 (95% CI = [+ 0.20; + 0.53]) for horizontal movement. However, the other parameters, and notably the other swallowing markers were not significantly modified by the use of the device.
Conclusion
The personalised transportable folding device for seating developed to reduce dysphagia has an action on hyoid bone motion during swallowing. However, this positive effect on an intermediate outcome criterion of the quality of swallowing was not associated with an improvement in swallowing efficiency in the study population. The diversity of diseases with which the patients in this study were afflicted is a factor to be controlled in future studies with this device.
... Lee et al. developed a software platform that extracted the trajectory of the moving hyoid bone by calculating local binary patterns and multi-scale local binary patterns 19 . Kim et al. developed software which can track, smooth and segment the hyoid bone motion from VFSS 20 . ...
Recent publications have suggested that high-resolution cervical auscultation (HRCA) signals may provide an alternative non-invasive option for swallowing assessment. However, the relationship between hyoid bone displacement, a key component to safe swallowing, and HRCA signals is not thoroughly understood. Therefore, in this work we investigated the hypothesis that a strong relationship exists between hyoid displacement and HRCA signals. Videofuoroscopy data was collected for 129 swallows, simultaneously with vibratory/acoustic signals. Horizontal, vertical and hypotenuse displacements of the hyoid bone were measured through manual expert analysis of videofluoroscopy images. Our results showed that the vertical displacement of both the anterior and posterior landmarks of the hyoid bone was strongly associated with the Lempel-Ziv complexity of superior-inferior and anterior-posterior vibrations from HRCA signals. Horizontal and hypotenuse displacements of the posterior aspect of the hyoid bone were strongly associated with the standard deviation of swallowing sounds. Medial-Lateral vibrations and patient characteristics such as age, sex, and history of stroke were not significantly associated with the hyoid bone displacement. The results imply that some vibratory/acoustic features extracted from HRCA recordings can provide information about the magnitude and direction of hyoid bone displacement. These results provide additional support for using HRCA as a non-invasive tool to assess physiological aspects of swallowing such as the hyoid bone displacement.
... Figure 1(a) shows one frame from a X-ray video clip. The position of hyoid bone is tracked in each frame by a semiautomatic programme developed in Kim et al. (2017). The raw data before being preprocessed are shown in Figure 1(b). ...
The clustering for functional data with misaligned problems has drawn much attention in the last decade. Most methods do the clustering after those functional data being registered and there has been little research using both functional and scalar variables. In this paper, we propose a simultaneous registration and clustering (SRC) model via two-level models, allowing the use of both types of variables and also allowing simultaneous registration and clustering. For the data collected from subjects in different groups, a Gaussian process functional regression model with time warping is used as the first level model; an allocation model depending on scalar variables is used as the second level model providing further information over the groups. The former carries out registration and modeling for the multi-dimensional functional data (2D curves) at the same time. This methodology is implemented using an EM algorithm, and is examined on both simulated data and real data.
