Fig 6
a) An input image (top) and a query image generated by Canny edge detection (bottom). b) and c) The two synthesized hypotheses. a) Query Image . b) . c) .
Source publication
![Fig. 2. FDCM algorithm [10] (left) and the proposed approach (right)...](profile/Xianqiao_Tong/publication/275155625/figure/fig2/AS:408997476749314@1474524005831/FDCM-algorithm-10-left-and-the-proposed-approach-right-applied-to-surgical_Q320.jpg)
We developed a unique robotic manipulation system that accurately singulates surgical instruments in a cluttered environment. A novel single-view computer vision algorithm identifies the next instrument to grip from a cluttered pile and a compliant electromagnetic gripper picks up the identified instrument. System is validated through extensive exp...
Citations
... Still, they did not consider grasping in the surgical environment. For example, in [13,[25][26][27], the authors demonstrated that robotic grasping for handling ferromagnetic surgical instruments using the electromagnetic gripper. The grippers could only handle selected tools when the tools were cluttered and overlaid. ...
This paper presents a conceptual design and implementation of a soft, compliant 3D printed gripper (SurgGrip), conceived for automated grasping of various surgery-based thin-flat instruments. The proposed solution includes (1) a gripper with a resilient mechanism to increase safety and better adaptation to the unstructured environment; (2) flat fingertips with mortise and tenon joint to facilitate pinching and enveloping based grasping of thin and random shape tools; (3) a soft pad on the fingertips to enable the high surface area to maintain stable grasping of the surgical instruments; (4) a four-bar linkage with a leadscrew mechanism to provide a precise finger movement; (5) enable automated manipulation of surgical tools using computer vision. Our gripper model is designed and fabricated by integrating soft and rigid components through a hybrid approach. The SurgGrip shows passive adaptation through inherent compliance of linear and torsional spring. The four-bar linkage mechanism controlled by a motor-leadscrew-nut drive provides precise gripper opening and closing movements. The experimental results show that the SurgGrip can detect, segment through a camera, and grasp surgical instruments (maximum 606.73 gs), with a 67% success rate (grasped 10 out of 12 selected tools) at 3.21 mm/s grasping speed and 15.81 s object grasping time autonomously. Besides, we demonstrated the pick and place abilities of SurgGrip on flat and nonflat surfaces in real-time.
Supplementary information:
The online version contains supplementary material available at 10.1007/s11012-022-01594-6.
... Changes in surface types and depth discontinuities are then used to segment the cluttered scene. A vision-based algorithm is proposed in [6] to resolve gripper-object collision by identifying and picking the topmost object in a pile composed of surgical instruments. Schwarz and Behnke [7] propose a deep learning approach for extracting large individual objects from a cluttered bin. ...
The picking of one or more objects from an unsorted pile continues to be non-trivial for robotic systems. This is especially so when the pile consists of a granular material (GM) containing individual items that tangle with one another, causing more to be picked out than desired. One of the key features of such tangle-prone GMs is the presence of protrusions extending out from the main body of items in the pile. This work characterises the role the latter play in causing mechanical entanglement and their impact on picking consistency. It reports experiments in which picking GMs with different protrusion lengths (PLs) results in up to 76% increase in picked mass variance, suggesting PL to be an informative feature in the design of picking strategies. Moreover, to counter this effect, it proposes a new spread-and-pick (SnP) approach that significantly reduces tangling, making picking more consistent. Compared to prior approaches that seek to pick from a tangle-free point in the pile, the proposed method results in a decrease in picking error (PE) of up to 51%, and shows good generalisation to previously unseen GMs.
... The most popular solution for the detection task is to utilize a camera. Among the camera-based approaches found in the literature is the work presented in [21], which relies on the use of markers attached to the instruments, and also the works of Zhou et al. [23,24] which identify instruments one at a time, employing an identification tray. Similarly, in [16], an RGB-D camera and 3D-printed models of surgical instruments are employed during the experiments. ...
... The instrument identification based on external means is studied in [21], where the authors rely on small barcodes attached to the instruments and a scanner to determine each instrument's class. This requires high resolution images and high proximity, as well as previously prepared instruments and visible barcodes during the identification process, which can increase the cost of the system, the identification time, and be susceptible to errors produced by overexposure. ...
Purpose
Robotic scrub nurses have the potential to become an attractive solution for the operating room. Surgical instrument detection is a fundamental task for these systems, which is the focus of this work. We address the detection of the complete surgery set for wisdom teeth extraction, and propose a data augmentation technique tailored for this task.
Methods
Using a robotic scrub nurse system, we create a dataset of 369 unique multi-instrument images with manual annotations. We then propose the Mask-Based Object Insertion method, capable of automatically generating a large amount of synthetic images. By using both real and artificial data, different Mask R-CNN models are trained and evaluated.
Results
Our experiments reveal that models trained on the synthetic data created with our method achieve comparable performance to that of models trained on real images. Moreover, we demonstrate that the combination of real and our artificial data can lead to a superior level of generalization.
Conclusion
The proposed data augmentation technique is capable of dramatically reducing the labelling work required for training a deep-learning-based detection algorithm. A dataset for the complete instrument set for wisdom teeth extraction is made available for the scientific community, as well as the raw information required for the generation of the synthetic data ( https://github.com/Jorebs/Deep-learning-based-instrument-detection-for-intra operative-robotic-assistance ).
... Specifically for instruments robot picking, works can be divided mainly into two streams: pre-installed tag-based and visual identificationbased. Tag-based methods are the mainstream of instrument picking [13], [14] due to the high similarity and variety of Fig. 1. Proposed instrument identification system pipeline surgical tools. ...
... By automating the process, the time efficiency can be improved, reducing also exposure of humans to hazardous waste. The automated tidying or removal of biohazardous materials requires direct manipulation by robotic end-effectors [10] that are typically sophisticated and thus expensive [11]. Such devices require regular sanitation and post-routine cleaning to meet sanitary requirements, while some designs are hard to clean. ...
Robots use their end-effectors to grasp and manipulate objects in unstructured and dynamic environments. Robot hands and grippers can vary from rigid and complex designs to soft, inflatable, and lightweight structures. In this paper, we focus on the modelling and development of a pneumatically driven soft robotic gripper with retractable telescopic fingers and finger bases with abduction / adduction capabilities. Both the main fingers and the base actuators use a pre-folded, telescopic structure facilitating passive retraction. The efficiency of the proposed device is experimentally validated through different types of experiments: i) grasping experiments that involve different everyday objects ranging from household objects and fragile items to medical waste and consumables, ii) force exertion experiments that capture the maximum forces that can be exerted by the proposed device when utilizing the different actuators of the gripper, and iii) grasp resistance experiments that focus on the effect of the inflatable structure on resisting environmental uncertainties and disturbances. The proposed gripper is able to grasp a plethora of objects, and can exert more than 14 N of grasping force. The design is so low-cost and modular that the soft fingers and palm pad of the gripper can be used in a disposable manner, facilitating the execution of specialized tasks (e.g., grasping in contaminated environments, handling of medical waste, etc). When it is not inflated, the gripper profile is thin and compact to facilitate storage.
... However, the involvement of medical robotics is undoubtedly helpful for our medical workers. Some of the works in medical robotics include application in an operating room [2], [3], a robotic arm with laparoscopic instruments [4], Da Vinci R, the most famous surgical robotic since the past decade [5], Robotic Scrub Nurse (RSN) [6], "pick and place" robotic system for surgical instruments by Peenelope CS [7] and Y. Xu [8], and Versius robotic arm for use in gynecology, upper GI surgery, collateral, and urology application [9]. The role of medical robotics in hospitals is essential and can reduce the burden of medical staff. ...
... [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15] ...
... Changes in surface types and depth discontinuities were then used to segment the cluttered scene. A novel vision-based algorithm was proposed in [2], which suggested resolving gripper-object collision by identifying and picking the topmost object in a pile. A deep learning approach for picking individual objects from a cluttered bin was proposed in [3]. ...
... 3-D point cloud matching techniques can be widely applied in the industry especially for production line operations. By employing machine vision for manipulators, automation tasks can be accomplished in bin-picking systems [45]. Although some information can be used for matching such as color, 3-D point cloud information is relatively more reliable and it is applicable for most objects. ...
In 3-D pattern matching, a reconstructed 3-D data point cloud of an object can be matched with the 3-D model point cloud to determine the object pose by employing the iterative closest point (ICP) algorithm. Typical ICP algorithm is time-consuming and may get stuck in local minimum if the pose difference between the two point clouds is not small enough. To resolve this uncertainty problem and enhance the matching capability, the candidate-based axially switching (CBAS) computed closer point (CCP) approach is proposed which is efficient, effective, and robust. It is based on evaluating origin candidates in the model point cloud to determine the approximate pose of the data point cloud. The proposed CBAS-CCP approach allows large uncertainty of the pose difference between the model and the object. The applicability and effectiveness of the proposed approach has been successfully validated by experimenting with 3-D data of real objects.
... Normally, vision based robotic grasping methods mainly include object detection, object segmentation, pose estimation and the selection of grasping point [1]. 2D features are used to estimate the pose of target objects [2][3] [4]. With the development of 3D vision technology, 3D visions are also used in robotic grasp in cluttered scene to detect the objects and estimate poses [5] [6]. ...
This paper focuses on a robotic picking tasks in cluttered scenario. Because of the diversity of objects and clutter by placing, it is much difficult to recognize and estimate their pose before grasping. Here, we use U-net, a special Convolution Neural Networks (CNN), to combine RGB images and depth information to predict picking region without recognition and pose estimation. The efficiency of diverse visual input of the network were compared, including RGB, RGB-D and RGB-Points. And we found the RGB-Points input could get a precision of 95.74%.
... A common approach to avoid surgical tool detection errors is the addition of external markers to the surgical tool which significantly eases the recognition task. Over the years several studies resorted this approach, using a variety of external markers such as recognizable patterns [101], color tags [102], light-emitting diodes [103], RFID tags [10] and 2D data matrix barcodes [3][104] [105]. ...
... Due to the objective of recognizing the tool the as fastest as possible, the initial approach to achieve the surgical object detection and sorting was similar to studies from Xu et al. [3][104] [105] and its scheme is presented in ...
Full Text ------------------ http://hdl.handle.net/10316/86257 ------------------
The main goal of this master dissertation is to classify and localize surgical tools in a cluttered tray, as well as perform occlusion reasoning to determine which tool should be removed first. These tasks are intended to be a part of a multi-stage robotic system able to sort surgical tools after disinfection, in order to assembly surgical kits and, hopefully, optimizing the nurses time in sterilization rooms, so that they can focus on more complex tasks.Initially, several classical approaches were tested to obtain 2D templates of each type of surgical tool, such as canny edges, otsu’s threshold and watershed algorithm. The idea was to place 2D data matrixes codes onto the surgical tools and whenever the code was detected, the respective template would be added to a virtual map, which would be posteriorly be assessed and determined which tool was on top by comparison with the original image. However, due to difficulties in acquiring a specific software, a modern approach was used instead, resorting to the YOLO (“you only look once”) deep learning neural network.In order to train the neural networks, a dataset was built, which was then published, along with the respective labels of the data and appropriate division into train and test groups. In total, 5 YOLOv2 neural networks were trained: 1 for object detection and classification and 1 for occlusion reasoning of each instrument (making a total of 4). Regarding object detection, it was also performed cross-validation, as well as trained the YOLOv3 network.A console application that applies the proposed algorithm was also developed, in which the first step is to run the object detector with either the trained YOLOv2 or YOLOv3 network, followed by sorting the detections in a decrescent order of confidence score. Afterward, the detections correspondent to the two higher confidence scores are chosen and the respective occlusion reasoning neural networks are run. Finally, the best combination of confidence scores between object detection and occlusion reasoning determines the surgical tool to be removed first from the cluttered tray.
