Figure - uploaded by Arnaud Huaulmé
Content may be subject to copyright.
Source publication
Objective: According to a meta-analysis of 7 studies, the median number of patients with at least one adverse event during the surgery is 14.4%, and a third of those adverse events were preventable. The occurrence of adverse events forces surgeons to implement corrective strategies and, thus, deviate from the standard surgical process. Therefore, i...
Contexts in source publication
Context 1
... define three different hidden states: "no deviation" (compared to the standard surgical process), "context deviation," and "event deviation." These latter two are defined according to the definition given in Table 1. In our case, we have not "expert deviation" due to the fact that all surgeries were performed by a single surgeon. ...Context 2
... could help determine the exact moment when iAEs occur. Moreover, the objective of an event deviation is to "correct or limit the impacts of iAEs" (Table 1), so by studying the anatomical structure concerned by event deviations, it will be possible to determine which one is impacted. Of course, to make a complete identification of iAEs, further work will be necessary. ...Context 3
... define three different hidden states: "no deviation" (compared to the standard surgical process), "context deviation," and "event deviation." These latter two are defined according to the definition given in Table 1. In our case, we have not "expert deviation" due to the fact that all surgeries were performed by a single surgeon. ...Context 4
... could help determine the exact moment when iAEs occur. Moreover, the objective of an event deviation is to "correct or limit the impacts of iAEs" (Table 1), so by studying the anatomical structure concerned by event deviations, it will be possible to determine which one is impacted. Of course, to make a complete identification of iAEs, further work will be necessary. ...Similar publications
Machine learning has been used in hydrological applications for decades, and recently, it was proven to be more efficient than sophisticated physically based modelling techniques. In addition, it has been used in hybrid frameworks that combine hydrological and machine learning models. The concept behind the latter is the use of machine learning as...
The location of the grounding grid conductors is critical for performing corrosion diagnosis and maintenance work. An improved magnetic field differential method to locate the unknown grounding grid based on truncation errors and the round-off errors analysis is presented in this paper. It was proven that a different order of the magnetic field der...
This paper aims to know the type of errors that happened in writing daily activities of fifth grade of MI Al Khairiyah Kupang Teba Bandar Lampung. This research used qualitative design. This study was focused on analyzing the type of errors that's produce by students in simple present tense that have been collected from 20 students of fifth grade o...
This paper outlines the design and error analysis for a flux characterization device (i.e. flux wand) which traverses the aperture of a falling particle receiver cavity providing the incident power measurement needed for the calculation of receiver efficiency without significantly disrupting receiver output. The design features a linear actuator, w...
Coal mine water in China is mainly mine water containing suspended solids（SS）. The relationship between suspended solids and turbidity in mine water with different SS content gradients was studied and a model formula was established. Finally, the accuracy of the model formula is verified by the actual detection values of five kinds of mine water. T...
Citations
... As an example, deep learning-based automatic surgical record organization utilizes machine learning algorithms to analyze and categorize surgical records and documents. 19,21,22) One potential benefit of this approach is an improvement in the efficiency of medical record keeping and information management through the automatic classification and organization of surgical records. 19,22) This can allow medical professionals to more easily access and review relevant information, potentially saving time and resources. ...
... 19,21,22) One potential benefit of this approach is an improvement in the efficiency of medical record keeping and information management through the automatic classification and organization of surgical records. 19,22) This can allow medical professionals to more easily access and review relevant information, potentially saving time and resources. 19) Additionally, automatic surgical record organization using deep learning may be able to identify patterns and trends in surgical data, which could be useful for research and analysis. ...
Background
As the population ages, the rates of hip diseases and fragility fractures are increasing, making total hip arthroplasty (THA) one of the best methods for treating elderly patients. With the increasing number of THA surgeries and diverse surgical methods, there is a need for standard evaluation protocols. This study aimed to use deep learning algorithms to classify THA videos and evaluate the accuracy of the labelling of these videos.
Methods
In our study, we manually annotated 7 phases in THA, including skin incision, broaching, exposure of acetabulum, acetabular reaming, acetabular cup positioning, femoral stem insertion, and skin closure. Within each phase, a second trained annotator marked the beginning and end of instrument usages, such as the skin blade, forceps, Bovie, suction device, suture material, retractor, rasp, femoral stem, acetabular reamer, head trial, and real head.
Results
In our study, we utilized YOLOv3 to collect 540 operating images of THA procedures and create a scene annotation model. The results of our study showed relatively high accuracy in the clear classification of surgical techniques such as skin incision and closure, broaching, acetabular reaming, and femoral stem insertion, with a mean average precision (mAP) of 0.75 or higher. Most of the equipment showed good accuracy of mAP 0.7 or higher, except for the suction device, suture material, and retractor.
Conclusions
Scene annotation for the instrument and phases in THA using deep learning techniques may provide potentially useful tools for subsequent documentation, assessment of skills, and feedback.
... As a fundamental component in building a sophisticated assistive system for the operating room [6], the identification of surgical process can help the system in real-time monitoring and optimizing the surgical workflow. This skill can help surgeons make decisions, reduce surgical errors and warn them of potential complications [7,8]. ...
Liver rupture repair surgery serves as one tool to treat liver rupture, especially beneficial for cases of mild liver rupture hemorrhage. Liver rupture can catalyze critical conditions such as hemorrhage and shock. Surgical workflow recognition in liver rupture repair surgery videos presents a significant task aimed at reducing surgical mistakes and enhancing the quality of surgeries conducted by surgeons. A liver rupture repair simulation surgical dataset is proposed in this paper which consists of 45 videos collaboratively completed by nine surgeons. Furthermore, an end-to-end SA-RLNet, a self attention-based recurrent convolutional neural network, is introduced in this paper. The self-attention mechanism is used to automatically identify the importance of input features in various instances and associate the relationships between input features. The accuracy of the surgical phase classification of the SA-RLNet approach is 90.6%. The present study demonstrates that the SA-RLNet approach shows strong generalization capabilities on the dataset. SA-RLNet has proved to be advantageous in capturing subtle variations between surgical phases. The application of surgical workflow recognition has promising feasibility in liver rupture repair surgery.
... Surgical step recognition is necessary to enable downstream applications such as surgical workflow analysis [4,26], context-aware decision support [21], anomaly detection [14], and record-keeping purposes [28,5]. Some factors that make recognition of steps in surgical videos a challenging problem [20,21] include variability in patient anatomy and surgeon style [11], similarities across steps in a procedure [5,16], online recognition [25] and scene blur [21,12]. ...
Automated surgical step recognition is an important task that can significantly improve patient safety and decision-making during surgeries. Existing state-of-the-art methods for surgical step recognition either rely on separate, multi-stage modeling of spatial and temporal information or operate on short-range temporal resolution when learned jointly. However, the benefits of joint modeling of spatio-temporal features and long-range information are not taken in account. In this paper, we propose a vision transformer-based approach to jointly learn spatio-temporal features directly from sequence of frame-level patches. Our method incorporates a gated-temporal attention mechanism that intelligently combines short-term and long-term spatio-temporal feature representations. We extensively evaluate our approach on two cataract surgery video datasets, namely Cataract-101 and D99, and demonstrate superior performance compared to various state-of-the-art methods. These results validate the suitability of our proposed approach for automated surgical step recognition. Our code is released at: https://github.com/nisargshah1999/GLSFormer
... Only thirteen (41.2%) studies mentioned ethics committee approval [11][12][13][14][15][16][17][18]. ...
... Only Huaulmé et al. indicated that cases were consecutively included in the dataset during the inclusion time [18], decreasing the bias associated with non-consecutive case inclusion. ...
... Three studies focusing on the whole procedure did not detail the number of phases [15,18,26]. ...
Background
Annotated data are foundational to applications of supervised machine learning. However, there seems to be a lack of common language used in the field of surgical data science.
The aim of this study is to review the process of annotation and semantics used in the creation of SPM for minimally invasive surgery videos.
Methods
For this systematic review, we reviewed articles indexed in the MEDLINE database from January 2000 until March 2022. We selected articles using surgical video annotations to describe a surgical process model in the field of minimally invasive surgery. We excluded studies focusing on instrument detection or recognition of anatomical areas only. The risk of bias was evaluated with the Newcastle Ottawa Quality assessment tool. Data from the studies were visually presented in table using the SPIDER tool.
Results
Of the 2806 articles identified, 34 were selected for review. Twenty-two were in the field of digestive surgery, six in ophthalmologic surgery only, one in neurosurgery, three in gynecologic surgery, and two in mixed fields. Thirty-one studies (88.2%) were dedicated to phase, step, or action recognition and mainly relied on a very simple formalization (29, 85.2%). Clinical information in the datasets was lacking for studies using available public datasets. The process of annotation for surgical process model was lacking and poorly described, and description of the surgical procedures was highly variable between studies.
Conclusion
Surgical video annotation lacks a rigorous and reproducible framework. This leads to difficulties in sharing videos between institutions and hospitals because of the different languages used. There is a need to develop and use common ontology to improve libraries of annotated surgical videos.
... By automatically recognizing and evaluating current surgical scenarios, CAS systems can provide intraoperative decision support, improve operating room efficiency, assess surgical skills, and assist with surgical training and education [3]. Using surgical phase recognition during surgery, one can monitor the progress of the procedure, provide context-aware decision support, detect potential deviations and anomalies, perform objective and data-driven analysis of workflow and compare best practices [4]. However, even for advanced computer-assisted teaching systems [5,6], the task of identifying the surgical phase from intraoperative video remains challenging due to the diversity of patient anatomy and surgeon styles [7] and the limited availability and quality of video material [8]. ...
Background
Surgical video phase recognition is an essential technique in computer-assisted surgical systems for monitoring surgical procedures, which can assist surgeons in standardizing procedures and enhancing postsurgical assessment and indexing. However, the high similarity between the phases and temporal variations of cataract videos still poses the greatest challenge for video phase recognition.
Methods
In this paper, we introduce a global–local multi-stage temporal convolutional network (GL-MSTCN) to explore the subtle differences between high similarity surgical phases and mitigate the temporal variations of surgical videos. The presented work consists of a triple-stream network (i.e., pupil stream, instrument stream, and video frame stream) and a multi-stage temporal convolutional network. The triple-stream network first detects the pupil and surgical instruments regions in the frame separately and then obtains the fine-grained semantic features of the video frames. The proposed multi-stage temporal convolutional network improves the surgical phase recognition performance by capturing longer time series features through dilated convolutional layers with varying receptive fields.
Results
Our method is thoroughly validated on the CSVideo dataset with 32 cataract surgery videos and the public Cataract101 dataset with 101 cataract surgery videos, outperforming state-of-the-art approaches with 95.8% and 96.5% accuracy, respectively.
Conclusions
The experimental results show that the use of global and local feature information can effectively enhance the model to explore fine-grained features and mitigate temporal and spatial variations, thus improving the surgical phase recognition performance of the proposed GL-MSTCN.
... As the basis for the design of an intelligent auxiliary system in the operating room [7],the recognition of surgical workflow can assist the system in monitoring and optimizing the operation process in real time. It helps to alert surgeons when complications may arise and reduce their surgical errors as well as support their decision-making [8] [9]. ...
Purpose Surgical workflow recognition has emerged an important part of computer-assisted intervention systems for the modern operating room, and at the same time a very challenging problem. For sequence data, modelling of temporal information is essential for accurate identification. However, current phase recognition methods based on the CNN framework are not very adequate for the extraction of long-range dependencies.
Methods Therefore, to better capture long-range dependencies, we propose a Temporal-based Swin Transformer Network (TSTNet) for the surgical video workflow recognition task. TSTNet contains two main parts: the Swin Transformer and the LSTM. The Swin Transformer incorporates the attention mechanism to encode remote dependencies and learn highly expressive representations. The LSTM is capable of learning longrange dependencies and is used to extract temporal information. The TSTNet organically combines the two components to extract spatiotemporal features that contain more contextual information. In particular, based on a full understanding of the natural features of the surgical video, we propose a priori revision algorithm (PRA) using a priori information about the sequence of the surgical phase. This strategy optimises the output of TSTNet and further improves the recognition performance.
Results We have conducted extensive experiments using the Cholec80 dataset to validate the effectiveness of the TSTNet-PRA method. Our method achieves excellent performance on the Cholec80 dataset and can achieve an accuracy of 93.5%. This greatly exceeds the state-of-the-art methods.
Conclusion By modelling remote temporal information and multi-scale visual information, we proposed the TSTNet-PRA method. It was evaluated on a large public dataset, showing high recognition capability superior to other spatiotemporal networks.
... Causal algorithms, that do not require information from the future, can provide feedback to surgeons while performing surgery, can help staff in the operation room to detect anomalous events, and help to coordinate the surgical team [1][2][3]. In addition, offline phase analysis can be used for surgical deviation identification or automatic report generation [3,4]. In this work, we focus on causal algorithms as they can provide both postoperative but also real-time intra-operatively analytics. ...
Purpose
Surgical workflow estimation techniques aim to divide a surgical video into temporal segments based on predefined surgical actions or objectives, which can be of different granularity such as steps or phases. Potential applications range from real-time intra-operative feedback to automatic post-operative reports and analysis. A common approach in the literature for performing automatic surgical phase estimation is to decouple the problem into two stages: feature extraction from a single frame and temporal feature fusion. This approach is performed in two stages due to computational restrictions when processing large spatio-temporal sequences.
Methods
The majority of existing works focus on pushing the performance solely through temporal model development. Differently, we follow a data-centric approach and propose a training pipeline that enables models to maximise the usage of existing datasets, which are generally used in isolation. Specifically, we use dense phase annotations available in Cholec80 , and sparse scene (i.e., instrument and anatomy) segmentation annotation available in CholecSeg8k in less than 5% of the overlapping frames. We propose a simple multi-task encoder that effectively fuses both streams, when available, based on their importance and jointly optimise them for performing accurate phase prediction.
Results and conclusion
We show that with a small fraction of scene segmentation annotations, a relatively simple model can obtain comparable results than previous state-of-the-art and more complex architectures when evaluated in similar settings. We hope that this data-centric approach can encourage new research directions where data, and how to use it, plays an important role along with model development.
... Automatic recognition of surgical workflow is essential for the Operating Room (OR) increasing the patient's safety through early detection of surgical workflow variations and improving the surgical results [1]. By supplying surgeons with the necessary information for each surgical step, a cognitive OR can reduce the stress induced by an overload of information [2] and build the foundation for more efficient surgical scheduling systems. ...
Algorithmic surgical workflow recognition is an ongoing research field and can be divided into laparoscopic (Internal) and operating room (External) analysis. So far many different works for the internal analysis have been proposed with the combination of a frame-level and an additional temporal model to address the temporal ambiguities between different workflow phases. For the External recognition task, Clip-level methods are in the focus of researchers targeting the local ambiguities present in the OR scene. In this work we evaluate combinations of different model architectures for the task of surgical workflow recognition to provide a fair comparison of the methods for both Internal and External analysis. We show that methods designed for the Internal analysis can be transferred to the external task with comparable performance gains for different architectures.
... From the perspective of robotic surgery, similar research focused more on gesture recognition from kinematic data ( DiPietro et al., 2016;, and robotized surgeries ( Kitaguchi et al., 2019;Zia et al., 2018 ), system events , and the recognition of other events, such as the presence of smoke or bleeding ( Loukas and Georgiou, 2015 ). These surgical events are explored for the recognition of surgeon's deviation from standard processes in laparoscopic videos ( Huaulmé et al., 2020 ). ...
Out of all existing frameworks for surgical workflow analysis in endoscopic videos, action triplet recognition stands out as the only one aiming to provide truly fine-grained and comprehensive information on surgical activities. This information, presented as 〈instrument, verb, target〉 combinations, is highly challenging to be accurately identified. Triplet components can be difficult to recognize individually; in this task, it requires not only performing recognition simultaneously for all three triplet components, but also correctly establishing the data association between them. To achieve this task, we introduce our new model, the Rendezvous (RDV), which recognizes triplets directly from surgical videos by leveraging attention at two different levels. We first introduce a new form of spatial attention to capture individual action triplet components in a scene; called Class Activation Guided Attention Mechanism (CAGAM). This technique focuses on the recognition of verbs and targets using activations resulting from instruments. To solve the association problem, our RDV model adds a new form of semantic attention inspired by Transformer networks; called Multi-Head of Mixed Attention (MHMA). This technique uses several cross and self attentions to effectively capture relationships between instruments, verbs, and targets. We also introduce CholecT50 - a dataset of 50 endoscopic videos in which every frame has been annotated with labels from 100 triplet classes. Our proposed RDV model significantly improves the triplet prediction mAP by over 9% compared to the state-of-the-art methods on this dataset.
... The major limitation of SPM related state of the art publications [3,4,5,7,9,10] is the manual acquisition of the SPMs by human observers. This solution is observer-dependent, time-consuming, and subject to errors [11]. ...
This paper presents the design and results of the "PEg TRAnsfert Workflow recognition" (PETRAW) challenge whose objective was to develop surgical workflow recognition methods based on one or several modalities, among video, kinematic, and segmentation data, in order to study their added value. The PETRAW challenge provided a data set of 150 peg transfer sequences performed on a virtual simulator. This data set was composed of videos, kinematics, semantic segmentation, and workflow annotations which described the sequences at three different granularity levels: phase, step, and activity. Five tasks were proposed to the participants: three of them were related to the recognition of all granularities with one of the available modalities, while the others addressed the recognition with a combination of modalities. Average application-dependent balanced accuracy (AD-Accuracy) was used as evaluation metric to take unbalanced classes into account and because it is more clinically relevant than a frame-by-frame score. Seven teams participated in at least one task and four of them in all tasks. Best results are obtained with the use of the video and the kinematics data with an AD-Accuracy between 93% and 90% for the four teams who participated in all tasks. The improvement between video/kinematic-based methods and the uni-modality ones was significant for all of the teams. However, the difference in testing execution time between the video/kinematic-based and the kinematic-based methods has to be taken into consideration. Is it relevant to spend 20 to 200 times more computing time for less than 3% of improvement? The PETRAW data set is publicly available at www.synapse.org/PETRAW to encourage further research in surgical workflow recognition.
