Fig 2 - uploaded by Wen-Hong Zhu
Content may be subject to copyright.
Source publication
A robot-assisted system for medical diagnostic ultrasound has been
developed by the authors. The paper presents the visual servo controller
used in this system. While the ultrasound transducer is positioned by a
robot, the operator, the robot controller, and an ultrasound image
processor have shared control over its motion. Ultrasound image feature...
Contexts in source publication
Context 1
... Sec- tion IV presents the theory behind ultrasound image servoing, along with experimental results. Two applications of the system to the feature-based reconstruction of an ultrasound phantom in 3-D and to perform teleoperated ultrasound via the Internet are described in Section V. Finally, Section VI provides a summary and concluding remarks. Fig. 2 shows the block-diagram of the experimental setup and Fig. 3 shows the inter-communication and data flow in the ...
Context 2
... performance of the system while tracking two features (center of two pipes in the phantom ultrasound image) simul- taneously is shown in Fig. 20. Three degrees of freedom of the robot are controlled by the image servo controller in this exper- iment. The two features are moved away from their reference points at s by moving the robot along the -axis and are moved back by the image servoing action. In this experiment, Hz. We report ultrasound image servoing results using the ...
Context 3
... By using the Star- Kalman algorithm [27] to extract each pipe's contour in the ul- trasound image and the inverse kinematics of the robot to map each contour to the world coordinates, a 3-D image of each pipe is reconstructed. Fig. 21 shows a partially reconstructed image. The reconstructed image has an average absolute error of less than 0.7 mm. Fig. 22 shows two view points of a reconstructed 3-D image of the carotid artery by using the Star-Kalman reconstruction method. The robot was used to move the ultrasound probe along the neck of a volunteer subject. The Star-Kalman algorithm was used to extract the contours of the carotid artery. The figure clearly shows the shape and the ...
Context 4
... application of the system as a teleultrasound device through the Internet is presented here. Fig. 23 shows the archi- tecture of the experimental setup. A client-server application was written for this purpose under the Linux/Debian operating system. The server is responsible for relaying data between the user interface, the image processing system, the robot controller and the video camera. The server uses a JPEG compression ...
Context 5
... to transfer ultrasound images to the remote operation site over a limited bandwidth. This could be improved by using an MPEG video compression method, which is not included in the current implementation of the system. At the same time, different operator commands are sent from the remote operation site to the server through this application. Fig. 24 shows the data flow in this ...
Similar publications
Standardized acquisitions and diagnoses using robots and AI would potentially increase the general usability and reliability of medical ultrasound. Working towards this prospect, this paper presents the recent developments of a standardized acquisition workflow using a novel dual-probe ultrasound robot, for a project known as intelligent Fetal Imag...
A robot-assisted system for medical diagnostic ultrasound has been developed by the authors. This paper presents key features of the user interface used in this system. While the ultrasound transducer is positioned by a robot, the operator, the robot controller, and an ultrasound image processor have shared control over its motion. Ultrasound image...
Citations
... SRCC is adopted to evaluate the monotonicity of the IQA model [7], which can be calculated with Equation (10). ...
The quality of breast ultrasound images has a significant impact on the accuracy of disease diagnosis. Existing image quality assessment (IQA) methods usually use pixel-level feature statistical methods or end-to-end deep learning methods, which focus on the global image quality but ignore the image quality of the lesion region. However, in clinical practice, doctors’ evaluation of ultrasound image quality relies more on the local area of the lesion, which determines the diagnostic value of ultrasound images. In this study, a global–local integrated IQA framework for breast ultrasound images was proposed to learn doctors’ clinical evaluation standards. In this study, 1285 breast ultrasound images were collected and scored by experienced doctors. After being classified as either images with lesions or images without lesions, they were evaluated using soft-reference IQA or bilinear CNN IQA, respectively. Experiments showed that for ultrasound images with lesions, our proposed soft-reference IQA achieved PLCC 0.8418 with doctors’ annotation, while the existing end-to-end deep learning method that did not consider the local lesion features only achieved PLCC 0.6606. Due to the accuracy improvement for the images with lesions, our proposed global–local integrated IQA framework had better performance in the IQA task than the existing end-to-end deep learning method, with PLCC improving from 0.8306 to 0.8851.
... 5 In addition, ultrasound technology can enable robots to move around independently and avoid obstacles, providing timely information about the location and distance of objects. 6 Ultrasound communication can ensure the safety and dependability of data exchange in congested or loud settings, such as industrial plants and public transit systems. 7 It can also improve the relationship between humans and machines by increasing process efficiency, operational safety, and allowing for the introduction of new types of applications and user interfaces. ...
Ultrasound has been implemented as a powerful tool for noninvasive imaging and non-contact ranging. However, the size and single-function of traditional ultrasonic probes have limited their adaptability to complex application scenarios in the artificial intelligence era. This paper reports a multifunctional microsensor based on a piezoelectric micromachined ultrasonic transducer (pMUT) array that can be used for ranging and 2D imaging at the same time, in air, with a small working size (2 × 0.8 mm2) and low driving voltage (10 V). A back-end circuit was also designed to optimize the performance of the pMUT array, resulting in a peak-to-peak value of 158 mV. The characterization of the multifunctional microsensor demonstrated that it is capable of performing ranging and 2D imaging simultaneously with high spatial resolution (∼1 mm in ranging and ∼0.2 × 0.2 mm2 in 2D-imaging), even in complex application scenarios. This paper describes a system that integrates micro-electro-mechanical systems device, back-end circuit, imaging algorithm, and denoising algorithm, which allow the microsensor to provide multiple functions. This makes it a promising tool for a wide range of human–machine interaction applications, such as gesture recognition, robotics, and healthcare.
... Chatelain et al. (2015) propose the use of ultrasound quality maps to improve image quality in robotic ultrasound scanning applications. Autonomous visual servoing systems have also been proposed in support of teleoperated ultrasound diagnosis (Abolmaesumi et al., 2002;Li et al., 2012), but these techniques tend to rely on hand designed anatomical target detectors or confidence maps. The scanner introduced in this work is fully autonomous, and relies entirely on a reward signal learned from demonstration, in what we believe is a first for medical imaging. ...
Informative path-planning is a well established approach to visual-servoing and active viewpoint selection in robotics, but typically assumes that a suitable cost function or goal state is known. This work considers the inverse problem, where the goal of the task is unknown, and a reward function needs to be inferred from exploratory example demonstrations provided by a demonstrator, for use in a downstream informative path-planning policy. Unfortunately, many existing reward inference strategies are unsuited to this class of problems, due to the exploratory nature of the demonstrations. In this paper, we propose an alternative approach to cope with the class of problems where these sub-optimal, exploratory demonstrations occur. We hypothesise that, in tasks which require discovery, successive states of any demonstration are progressively more likely to be associated with a higher reward, and use this hypothesis to generate time-based binary comparison outcomes and infer reward functions that support these ranks, under a probabilistic generative model. We formalise this probabilistic temporal ranking approach and show that it improves upon existing approaches to perform reward inference for autonomous ultrasound scanning, a novel application of learning from demonstration in medical imaging while also being of value across a broad range of goal-oriented learning from demonstration tasks.
... A more general and comprehensive summary of force control can refer to (Haidegger et al., 2009; Zeng and Hemami, 1997). (2015) 1-DOF load cell handheld mechanism general phantom Zhu et al. (2000); Abolmaesumi et al. (2002) 6-DOF F/T sensor 6-DOF customized robot carotid phantom & volunteer Pierrot et al. (1999) 6-DOF F/T sensor 7-DOF PA-10 robot cardiovascular patient Ma G. et al. (2021) built-in torque sensor 6-DOF UR5e vascular structure phantom ...
... Thereby, the contact force can be dynamically connected to the probe position and velocity by selecting proper interface parameters. A similar hybrid position/force method based on an external 6-DOF force/torque (F/T) sensor was designed for 6-DOF RUSS (Zhu et al., 2000;Abolmaesumi et al., 2002). Their approaches can automatically switch between velocity and force control modes according to the contact condition (free or contact space). ...
Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques.
... To eliminate the requirement for the pre-operative images, Huang et al. proposed a method to automatically determine the scan path based on the extracted skin surface using a depth camera (Huang et al. 2018a). To utilize human experience, Abolmaesumi et al. developed a shared control method to visualize carotid arteries in 3D (Abolmaesumi et al. 2002). In addition, RUSS has been widely used in different applications, such as visualizing underlying bone surfaces (Jiang et al. 2020b) and imaging human spines (Zhang et al. 2021). ...
Ultrasound (US) imaging is widely used for biometric measurement and diagnosis of internal organs due to the advantages of being real-time and radiation-free. However, due to high inter-operator variability, resulting images highly depend on operators' experience. In this work, an intelligent robotic sonographer is proposed to autonomously "explore" target anatomies and navigate a US probe to a relevant 2D plane by learning from expert. The underlying high-level physiological knowledge from experts is inferred by a neural reward function, using a ranked pairwise image comparisons approach in a self-supervised fashion. This process can be referred to as understanding the "language of sonography". Considering the generalization capability to overcome inter-patient variations, mutual information is estimated by a network to explicitly extract the task-related and domain features in latent space. Besides, a Gaussian distribution-based filter is developed to automatically evaluate and take the quality of the expert's demonstrations into account. The robotic localization is carried out in coarse-to-fine mode based on the predicted reward associated to B-mode images. To demonstrate the performance of the proposed approach, representative experiments for the "line" target and "point" target are performed on vascular phantom and two ex-vivo animal organ phantoms (chicken heart and lamb kidney), respectively. The results demonstrated that the proposed advanced framework can robustly work on different kinds of known and unseen phantoms.
... Among the wide variety of robot configurations for US scanning, linkage-based or standard serial robotic arms have been widely explored. Examples of study include the ideas put up for image-servoing-based automated carotid artery inspections by Salcudean et al. [6] and Abolmaesumi et al. [7], and the proposal of redundant arms with passive compliance for fetal US by Wang et al. [8] and Housden et al. [9]. Moreover, recent years have also seen an increased use of commercially available serial robots to be used in the area of robotic US. ...
... However, this method requires a manual initialization of the object on the first frame. In addition, Abolmaesumi et al. compared the performance of five popular feature tracking approaches (the cross-correlation, the sequential similarity detection, the Star algorithm, the Star-Kalman algorithm, and the discrete snake algorithm) on carotid arteries [20]. The experimental results demonstrated that the Similarity Detection method and the Star-Kalman algorithm can achieve noted tracking performance, while the Correlation and Star algorithms result in poorer performance with higher computational cost. ...
Medical ultrasound (US) is widely used to evaluate and stage vascular diseases, in particular for the preliminary screening program, due to the advantage of being radiation-free. However, automatic segmentation of small tubular structures (e.g., the ulnar artery) from cross-sectional US images is still challenging. To address this challenge, this paper proposes the DopUS-Net and a vessel re-identification module that leverage the Doppler effect to enhance the final segmentation result. Firstly, the DopUS-Net combines the Doppler images with B-mode images to increase the segmentation accuracy and robustness of small blood vessels. It incorporates two encoders to exploit the maximum potential of the Doppler signal and recurrent neural network modules to preserve sequential information. Input to the first encoder is a two-channel duplex image representing the combination of the grey-scale Doppler and B-mode images to ensure anatomical spatial correctness. The second encoder operates on the pure Doppler images to provide a region proposal. Secondly, benefiting from the Doppler signal, this work first introduces an online artery re-identification module to qualitatively evaluate the real-time segmentation results and automatically optimize the probe pose for enhanced Doppler images. This quality-aware module enables the closed-loop control of robotic screening to further improve the confidence and robustness of image segmentation. The experimental results demonstrate that the proposed approach with the re-identification process can significantly improve the accuracy and robustness of the segmentation results (dice score: from 0:54 to 0:86; intersection over union: from 0:47 to 0:78).
... To solve the problems mentioned above, a wide range of robot-assisted technologies for ultrasound examinations have been developed. These robot systems have focused on several applications such as the shoulder [3], carotid artery [4], liver [5], fetal [6][7][8], cardiac tamponade [9], transesophageal echocardiography [10], echocardiography [11], and other generic sites [12,13]. However, most of these robot systems are designed to be used with the patient in the left lateral decubitus or supine position, which is the recommended position for conventional examination methods. ...
Purpose:
Conventional robotic ultrasound systems were utilized with patients in supine positions. Meanwhile, the limitation of the systems is that it is difficult to evacuate the patients in case of emergency (e.g., patient discomfort and system failure) because the patients are restricted between the robot system and bed. Therefore, we validated a feasibility study of seated-style echocardiography using a robot.
Method:
Preliminary experiments were conducted to verify the following two points: (1) diagnostic image quality due to the sitting posture angle and (2) physical load due to the sitting posture angle. For reducing the physical burden, two unique mechanisms were incorporated into the system: (1) a leg pendulum base mechanism to reduce the load on the legs when the lateral bending angle increases, and (2) a roll angle division by a lumbar lateral bending and thoracic rotation mechanisms.
Results:
Preliminary results demonstrated that adjusting the diagnostic posture angle allowed to obtain the views, including cardiac disease features, as in the conventional examination. The results also demonstrated that the body load reduction mechanism incorporated in the results could reduce the physical load in the seated echocardiography. Furthermore, this system was shown to provide greater safety and shorter evacuation times than conventional systems.
Conclusion:
These results indicate that diagnostic echocardiographic images can be obtained by seated-style echocardiography. It was also suggested that the proposed system can reduce the physical load and guarantee a sense of safety and emergency evacuation. These results demonstrated the possibility of the usage of the seated-style echocardiography robot.
... These proposed robots concentrated on limiting the contact forces rather than both forces and torques in a certain range, and additionally, did not integrate with the automatic pose adjustment of the US probe. Many vision-based controllers were proposed to control the robot system employing various visual features, such as points [13], moments [14], and wavelets [15]. In [16]- [17], the authors proposed a novel US feedback-based method to control three DOFs of the robot, enabling the US probe to move around a complex shape like the breast. ...
Robotic ultrasonography potentially acts as an essential aid to medical diagnosis. To overcome the limitations in robotic ultrasonography, in this paper, we proposed a novel self-adaptive parallel manipulator (SAPM) that can automatically adjust the ultrasound (US) probe pose to adapt to various contours of scanned areas, provide approximate constant operating forces/torques, achieve mechanical measurement, and cushion undesired produced forces. A novel parallel adjustment mechanism is proposed to attain automatic pose adjustment with 3 degrees of freedom (DOFs). This mechanism enables the US probe to adapt to different scanned areas and to perform the scanning with approximate constant forces and torques. Besides, we present a mechanical measurement and safety protection method that can be integrated into the SAPM and used as operation status monitoring and early warning during scanning procedures by capturing operating forces and torques. Experiments were carried out to calibrate the measurement and buffer units and evaluate the performance of the SAPM. Experimental results show the ability of the SAPM to provide 3-DOFs motion and operating force/torque measurement and automatically adjust the US probe pose to capture US images of equally good quality compared to a manual sonographer scan. Moreover, it has characteristics similar to soft robots that could significantly improve operation safety, and could be extended to some other engineering or medical applications.
... These involve a robotic arm with ultrasound probe end e↵ector which is teleoperated by a remote expert sonographer. Salcudean et al. presented a robot whose control was shared between the expert and a visual servoing system to maintain correct positioning on the carotid artery (Abolmaesumi, Salcudean, Zhu, Sirouspour, & DiMaio, 2002). Much interesting work has been carried out on autonomous robotic ultrasound to optimize ultrasound image quality (Chatelain, Krupa, & Navab, 2017). ...
div>
Current teleguidance methods include verbal guidance and robotic teleoperation, which present tradeoffs between precision and latency versus flexibility and cost. We present a novel concept of "human teleoperation" which bridges the gap between these two methods. A prototype teleultrasound system was implemented which shows the concept’s efficacy. An expert remotely "teloperates" a person (the follower) wearing a mixed reality headset by controlling a virtual ultrasound probe projected into the person’s scene. The follower matches the pose and force of the virtual device with a real probe. The pose, force, video, ultrasound images, and 3-dimensional mesh of the scene are fed back to the expert. In this control framework, the input and the actuation are carried out by people, but with near robot-like latency and precision. This allows teleguidance that is more precise and fast than verbal guidance, yet more flexible and inexpensive than robotic teleoperation. The system was subjected to tests that show its effectiveness, including mean teleoperation latencies of 0.27 seconds and errors of 7 mm and 6◦ in pose tracking. The system was also tested with an expert ultrasonographer and four patients and was found to improve the precision and speed of two teleultrasound procedures.
</div
