Fig 5 - uploaded by Shuo Li
Content may be subject to copyright.
Relationship between vertebral localization, identification, and segmentation. The arrows point to the interval between adjacent vertebrae. The horizontal lines represent the positional relationship between the centroids of segmentation results and localization heatmaps, which is ideally parallel between vertebrae. The correct and incorrect localizations are shown by the right angle and acute angle respectively.
Source publication
Magnetic resonance imaging (MRI) vertebral localization, identification, and segmentation are important steps in the automatic analysis of spines. Due to the similar appearances of vertebrae, various pathological patterns and imaging artifacts, the accurate segmentation, localization and identification of vertebrae remain challenging. With the emer...
Citations
... Chen et al. [22] proposed a multi-task learning network for segmentation and classification of atria, and the results showed that by sharing features between related tasks, the multi-task network can obtain additional anatomical information about the atria and achieve more accurate segmentation of atria. Zhang et al. [23] proposed a multi-task relational learning network for segmentation, localization, and identification of vertebrae, which utilized the relationship between vertebrae and the correlation of three tasks to train the network and finally proved the effectiveness of the network on an MRI dataset. Zhou et al. [24] proposed a multi-task learning framework for joint classification and segmentation of tumors in ultrasound images. ...
Accurate classification and segmentation of polyps are two important tasks in the diagnosis and treatment of colorectal cancers. Existing models perform segmentation and classification separately and do not fully make use of the correlation between the two tasks. Furthermore, polyps exhibit random regions and varying shapes and sizes, and they often share similar boundaries and backgrounds. However, existing models fail to consider these factors and thus are not robust because of their inherent limitations. To address these issues, we developed a multi-task network that performs both segmentation and classification simultaneously and can cope with the aforementioned factors effectively. Our proposed network possesses a dual-branch structure, comprising a transformer branch and a convolutional neural network (CNN) branch. This approach enhances local details within the global representation, improving both local feature awareness and global contextual understanding, thus contributing to the improved preservation of polyp-related information. Additionally, we have designed a feature interaction module (FIM) aimed at bridging the semantic gap between the two branches and facilitating the integration of diverse semantic information from both branches. This integration enables the full capture of global context information and local details related to polyps. To prevent the loss of edge detail information crucial for polyp identification, we have introduced a reverse attention boundary enhancement (RABE) module to gradually enhance edge structures and detailed information within polyp regions. Finally, we conducted extensive experiments on five publicly available datasets to evaluate the performance of our method in both polyp segmentation and classification tasks. The experimental results confirm that our proposed method outperforms other state-of-the-art methods.
... The researchers employed non-saturating neurons and a highly efficient GPU implementation for convolution operations to expedite the training process. R. Zhang et al. [38] introduced a Multi-Task Relational Learning Network (MRLN) that establishes connections between vertebrae and prioritizes three key tasks. The authors utilized a dilation convolution group to expand the receptive field and incorporated long short-term memory (LSTM) to retain historical knowledge about the sequential relationships among vertebral bodies. ...
Scoliosis is a complicated spinal deformity, and millions of people are suffering from this disease worldwide. Early detection and accurate scoliosis assessment are vital for effective clinical management and patient outcomes. The Cobb Angle (CA) measurement is the most precise method for calculating scoliotic curvature, which plays an essential role in diagnosing and treating scoliosis. This letter has conducted a systematic review to analyze scoliosis detection by vertebra identification and CA estimation using the Preferred Reporting Item for Systematic Review and Meta-Analysis (PRISMA) guidelines. The major scientific databases such as Scopus, Web of Science (WoS), and IEEE Xplorer are explored, where 2017-2023 publications are considered. The article selection process is based on keywords like “Vertebra Identification,” “CA Estimation,” “Scoliosis Detection,” “Deep Learning (DL),” etc. After rigorous analysis, 413 articles are extracted, and 44 are identified for final consideration. Further, several investigations based on the previous work are discussed along with its Proposed Solutions (PS).
... surgical planning (Windsor et al., 2020;Zhang et al., 2020; (e) ( (a)∼(d) physiological similarity, (e)∼(i) variable spinal disease conditions, (j)∼(l) different qualities of MRI (various intensity, artifacts in (j), low resolution in (k)) and unpredictable FOV (T12-S1, T10-S1, T10-L5, etc.) will lead to abnormalities in the morphological appearance 24 of the spinal curve and vertebrae in MRI. This may lead 25 to the same vertebrae of different patients showing more 26 appearance discrepancy than different vertebrae of the same 27 patient, which may lead to detection errors (Windsor et al., 28 2020). ...
Automated vertebrae detection (identification and localization) aims to identify vertebrae and locate their centroids in medical images, which is a critical step of spinal computer-aided systems. However, due to unpredictable field-of-view and various pathology cases, the image content is diverse and the vertebral morphology can be abnormal in a variety of ways, which challenges the designed systems. In this paper, we propose an effective sequence-based framework robust to various tough cases for accurate vertebrae identification and localization. Our method consists of three sub-modules: (1) Local Feature Extraction (LFE) module designs a shape-compatible category-balanced sampler to collect patches to train a convolution neural network, which extracts representative local features and generates score maps. (2) Discriminative Sequential Image Description (DSID) module proposes a node screening strategy for reliable vertebral feature sequence construction based on feature maps and score maps. This effectively prevents false positives and false negatives in light-weighted dense prediction schemes and fuses local features into a hierarchical discriminative description of given images. (3) Spinal Pattern Exploitation (SPE) module designs an end-balanced relative position learning scheme to fuse hierarchical local-global information for comprehensively exploiting spinal patterns to overcome the FOV and pathological variation challenges in vertebrae detection. Extensive experiments on a challenging dataset consisting of 450 spinal MRIs show that the identification rate of FSDF reaches 0.974 ± 0.025 and the localization error is only 4.742 ± 2.928 pixels, which demonstrates the effectiveness of our method with pathological and field-of-view variations and its superiority over other state-of-the-art methods.
... They demonstrated that the proposed pipeline was able to outperform the SOTA methods by combining a series of techniques such as delayed sub-sampling, exponential linear units and highly restrictive regularization. Zhang et al. 19 proposed a multi-task relational learning network (MRLN) for vertebral localization, identification, and segmentation in Magnetic Resonance Images (MRI). Pang et al. 20 proposed a two-stage framework, named SpineParseNet, for automatic spine parsing for MR images, where a 3D graph CNN was used for coarse segmentation and a 2D residual U-Net was then used for refining the segmentation. ...
Reliable vertebrae annotations are key to perform analysis of spinal X-ray images. However, obtaining annotation of vertebrae from those images is usually carried out manually due to its complexity (i.e. small structures with varying shape), making it a costly and tedious process. To accelerate this process, we proposed an ensemble pipeline, VertXNet, that combines two state-of-the-art (SOTA) segmentation models (respectively U-Net and Mask R-CNN) to automatically segment and label vertebrae in X-ray spinal images. Moreover, VertXNet introduces a rule-based approach that allows to robustly infer vertebrae labels (by locating the 'reference' vertebrae which are easier to segment than others) for a given spinal X-ray image. We evaluated the proposed pipeline on three spinal X-ray datasets (two internal and one publicly available), and compared against vertebrae annotated by radiologists. Our experimental results have shown that the proposed pipeline outperformed two SOTA segmentation models on our test dataset (MEASURE 1) with a mean Dice of 0.90, vs. a mean Dice of 0.73 for Mask R-CNN and 0.72 for U-Net. To further evaluate the generalization ability of VertXNet, the pre-trained pipeline was directly tested on two additional datasets (PREVENT and NHANES II) and consistent performance was observed with a mean Dice of 0.89 and 0.88, respectively. Overall, VertXNet demonstrated significantly improved performance for vertebra segmentation and labeling for spinal X-ray imaging, and evaluation on both in-house clinical trial data and publicly available data further proved its generalization.
... (2) Trained a single multi-task encoder-decoder network to refine center coordinate and vertebra segmentation Zhang et al (2020) MRI MRLN In-house 95.38 ...
... MRI is an imaging modality that is frequently used for the diagnosis of a wide variety of spine-related diseases. Zhang et al (2020) proposed the multi-task relational learning network (MRLN) framework for the segmentation of the vertebrae in MRI scans based on segmentation, localization, and identification. This method used co-attention to learn the intrinsic connection of vertebrae and the inner link between localization and segmentation. ...
Orthopedic surgery remains technically demanding due to the complex anatomical structures and cumbersome surgical procedures. The introduction of image-guided orthopedic surgery (IGOS) has significantly decreased the surgical risk and improved the operation results. This review focuses on the application of recent advances in Artificial Intelligence (AI), Deep Learning (DL), Augmented Reality (AR) and Robotics in image-guided spine surgery, joint arthroplasty, fracture reduction and bone tumor resection. For the pre-operative stage, key technologies of AI and deep learning based medical image segmentation, 3D reconstruction and surgical planning procedures are systematically reviewed. For the intra-operative stage, the development of novel image registration, surgical tool calibration and real-time navigation are reviewed. Furthermore, the combination of the surgical navigation system with augmented reality (AR) and robotic technology is also discussed. Finally, the current issues and prospects of the IGOS system are discussed, with the goal of establishing a reference and providing guidance for surgeons, engineers, and researchers involved in the research and development of this area.
... Since the sequential correlation was introduced, this method has effectively handled the complex background and pathological or anatomic variations (47). Zhang et al. (86) further applied a multi-task relational learning to locate, identify, and segment the vertebrae simultaneously, which avoided the overfitting of a single task, corrected each other, and pushed the identification rate to 93.55%. Unlike the previous practice of training a dedicated network from a single modality or contrast, multi-modality or multi-contrast image information can improve the spine detection ability of DL. automatic cross-modality sacral region detection of IVDs and vertebrae is achieved (91). ...
Background and objective:
As the spine is pivotal in the support and protection of human bodies, much attention is given to the understanding of spinal diseases. Quick, accurate, and automatic analysis of a spine image greatly enhances the efficiency with which spine conditions can be diagnosed. Deep learning (DL) is a representative artificial intelligence technology that has made encouraging progress in the last 6 years. However, it is still difficult for clinicians and technicians to fully understand this rapidly evolving field due to the diversity of applications, network structures, and evaluation criteria. This study aimed to provide clinicians and technicians with a comprehensive understanding of the development and prospects of DL spine image analysis by reviewing published literature.
Methods:
A systematic literature search was conducted in the PubMed and Web of Science databases using the keywords "deep learning" and "spine". Date ranges used to conduct the search were from 1 January, 2015 to 20 March, 2021. A total of 79 English articles were reviewed.
Key content and findings:
The DL technology has been applied extensively to the segmentation, detection, diagnosis, and quantitative evaluation of spine images. It uses static or dynamic image information, as well as local or non-local information. The high accuracy of analysis is comparable to that achieved manually by doctors. However, further exploration is needed in terms of data sharing, functional information, and network interpretability.
Conclusions:
The DL technique is a powerful method for spine image analysis. We believe that, with the joint efforts of researchers and clinicians, intelligent, interpretable, and reliable DL spine analysis methods will be widely applied in clinical practice in the future.
... Wang, X [27] proposed an automatic method to localize and identify vertebrae by combining deep SSAE(stacked sparse auto encoder) contextual features and SRF(structured regression forest) to overcome the disadvantages of the approaches employing hand-crafted, low-level features. Li, S [28] proposed aco-attention module to learn the localization-guided segmentation attention and segmentation-guided localization attention based on combining a dilation convolution group and LSTM to learn the prior knowledge that the identification information was always in a fixed order for spine images. Shi,D.,et ...
Accurately reading spinal CT images is very important in clinical, but it usually costs some minutes and deeply depends on doctor’s individual experiences. In this paper, we construct a scheme for spinal fracture lesions segmentation based on U-net, by introducing attention module, combining dilated convolution and U-net to get accurate lesions segmentation. First, we present four network schemes to compete in same data set, then get the best one, DU-net(dilated convolution), which replaces original convolution layer with dilated convolution in both contraction path and expansion path of U-net, to increase receptive field for more lesions feature information. Second, we introduce attention module to DU-net for accurate lesions segmentation by focusing on specific regions to improve lesions recognition of training model. Finally, we get prediction results by trained model of lesions segmentation on test data test. The experimental results show that our presented network has a better lesions segmentation performance than U-net, which can save time and reduce patients’ suffering clinically.
... With the advent of deep learning, several new frameworks achieved great success in image segmentation [15][16][17][18]. In the community of OCT image segmentation, deep learning based strategies are also treated as the state-of-the-arts [19][20][21][22]. ...
Automatic segmentation of layered tissue is the key to esophageal optical coherence tomography (OCT) image processing. With the advent of deep learning techniques, frameworks based on a fully convolutional network are proved to be effective in classifying pixels on images. However, due to speckle noise and unfavorable imaging conditions, the esophageal tissue relevant to the diagnosis is not always easy to identify. An effective approach to address this problem is extracting more powerful feature maps, which have similar expressions for pixels in the same tissue and show discriminability from those from different tissues. In this study, we proposed a novel framework, called the tissue self-attention network (TSA-Net), which introduces the self-attention mechanism for esophageal OCT image segmentation. The self-attention module in the network is able to capture long-range context dependencies from the image and analyzes the input image in a global view, which helps to cluster pixels in the same tissue and reveal differences of different layers, thus achieving more powerful feature maps for segmentation. Experiments have visually illustrated the effectiveness of the self-attention map, and its advantages over other deep networks were also discussed.
... The iterative FCN is susceptible to cascading failure [19], i.e., failure to find or correctly segment a single vertebra may cause failure to find or correctly segment all subsequent vertebrae. Li et al. [20] presented a multi-task relational learning network (MRLN) that utilized the dependency between vertebrae by generative adversarial networks (GAN) to achieve vertebral localization, identification, and segmentation on 2D MR images. The dependency between vertebrae in MRLN can be implicitly captured by GAN in the prediction space, which demonstrates that the dependency between vertebrae is capable of improving the vertebrae segmentation performance. ...
Spine parsing (i.e., multi-class segmentation of vertebrae and intervertebral discs (IVDs)) for volumetric magnetic resonance (MR) image plays a significant role in various spinal disease diagnoses and treatments of spine disorders, yet is still a challenge due to the inter-class similarity and intra-class variation of spine images. Existing fully convolutional network based methods failed to explicitly exploit the dependencies between different spinal structures. In this paper, we propose a novel two-stage framework named SpineParseNet to achieve automated spine parsing for volumetric MR images. The SpineParseNet consists of a 3D graph convolutional segmentation network (GCSN) for 3D coarse segmentation and a 2D residual U-Net (ResUNet) for 2D segmentation refinement. In 3D GCSN, region pooling is employed to project the image representation to graph representation, in which each node representation denotes a specific spinal structure. The adjacency matrix of the graph is designed according to the connection of spinal structures. The graph representation is evolved by graph convolutions. Subsequently, the proposed region unpooling module re-projects the evolved graph representation to a semantic image representation, which facilitates the 3D GCSN to generate reliable coarse segmentation. Finally, the 2D ResUNet refines the segmen-tation. Experiments on T2-weighted volumetric MR images of 215 subjects show that SpineParseNet achieves impressive performance with mean Dice similarity coefficients of 87.32 ± 4.75%, 87.78 ± 4.64%, and 87.49 ± 3.81% for the segmentations of 10 vertebrae, 9 IVDs, and all 19 spinal structures respectively. The proposed method has great po-Qianjin Feng is the corresponding author. tential in clinical spinal disease diagnoses and treatments.
