Fig 2 - uploaded by Robert Laganiere
Content may be subject to copyright.
Source publication
This paper addresses the problem of computing the three-dimensional (3-D) path of a moving rigid object using a calibrated stereoscopic vision setup. The proposed system begins by detecting feature points on the moving object. By tracking these points over time, it produces clouds of 3-D points that can be registered, thus giving information about...
Contexts in source publication
Context 1
... Even if this approach, involving the minimization of algebraic quantities, works well in practice, a geometric triangulation formulation is often preferred. The method finds the 3-D point that minimizes its 3-D distance with two noncrossing lines in space. In other words, it returns the middle point of the segment perpendicular to both rays. Fig. 2 shows the geometry of two cameras projecting the images x 1 and x 2 of the 3-D point X. In an ideal situation, the extension of the ...
Similar publications
Sensor-line cameras have been designed for space missions in the 1980s, and are used for various tasks, including panoramic imaging. Laser range-finders are able to generate dense depth maps (of isolated surface points). Panoramic sensor-line cameras and laser range-finders may both be implemented as rotating sensors, and we used them together this...
Citations
... In the case of Real Sense, the pixel coordinates of detected objects need to be recalculated using depth map data. Such a recalculation is carried out using the following expressions [66]: ...
Modern deep learning systems make it possible to develop increasingly intelligent solutions in various fields of science and technology. The electronics of single board computers facilitate the control of various robotic solutions. At the same time, the implementation of such tasks does not require a large amount of resources. However, deep learning models still require a high level of computing power. Thus, the effective control of an intelligent robot manipulator is possible when a computationally complex deep learning model on GPU graphics devices and a mechanics control unit on a single-board computer work together. In this regard, the study is devoted to the development of a computer vision model for estimation of the coordinates of objects of interest, as well as the subsequent recalculation of coordinates relative to the control of the manipulator to form a control action. In addition, in the simulation environment, a reinforcement learning model was developed to determine the optimal path for picking apples from 2D images. The detection efficiency on the test images was 92%, and in the laboratory it was possible to achieve 100% detection of apples. In addition, an algorithm has been trained that provides adequate guidance to apples located at a distance of 1 m along the Z axis. Thus, the original neural network used to recognize apples was trained using a big image dataset, algorithms for estimating the coordinates of apples were developed and investigated, and the use of reinforcement learning was suggested to optimize the picking policy.
... At present, the estimation of 6D object pose is usually based on visual sensors [8][9][10], which can achieve fast and accurate measurement by processing image information. The traditional methods [11,12] obtain the 6D object pose via establishing the corresponding relationships between the target image and the target model by matching feature points between the RGB image and the object 3D model. ...
6D pose estimation has been pervasively applied to various robotic applications, such as service robots, collaborative robots, and unmanned warehouses. However, accurate 6D pose estimation is still a challenge problem due to the complexity of application scenarios caused by illumination changes, occlusion and even truncation between objects, and additional refinement is required for accurate 6D object pose estimation in prior work. Aiming at the efficiency and accuracy of 6D object pose estimation in these complex scenes, this paper presents a novel end‐to‐end network, which effectively utilises the contextual information within a neighbourhood region of each pixel to estimate the 6D object pose from RGB‐D images. Specifically, our network first applies the attention mechanism to extract effective pixel‐wise dense multimodal features, which are then expanded to multi‐scale dense features by integrating pixel‐wise features at different scales for pose estimation. The proposed method is evaluated extensively on the LineMOD and YCB‐Video datasets, and the experimental results show that the proposed method is superior to several state‐of‐the‐art baselines in terms of average point distance and average closest point distance.
... The problem arises of converting the coordinates of objects in relative (to the upper left corner) pixels into millimeters relative to the center of the camera, since information about the distance in millimeters comes from the depth map. To solve this problem, it is possible to use the following relations [32]: ...
Despite the great possibilities of modern neural network architectures concerning the problems of object detection and recognition, the output of such models is the local (pixel) coordinates of objects bounding boxes in the image and their predicted classes. However, in several practical tasks, it is necessary to obtain more complete information about the object from the image. In particular, for robotic apple picking, it is necessary to clearly understand where and how much to move the grabber. To determine the real position of the apple relative to the source of image registration, it is proposed to use the Intel Real Sense depth camera and aggregate information from its depth and brightness channels. The apples detection is carried out using the YOLOv3 architecture; then, based on the distance to the object and its localization in the image, the relative distances are calculated for all coordinates. In this case, to determine the coordinates of apples, a transition to a symmetric coordinate system takes place by means of simple linear transformations. Estimating the position in a symmetric coordinate system allows estimating not only the magnitude of the shift but also the location of the object relative to the camera. The proposed approach makes it possible to obtain position estimates with high accuracy. The approximate root mean square error is 7–12 mm, depending on the range and axis. As for precision and recall metrics, the first is 100% and the second is 90%.
... Once the depth images are available, it is possible to obtain the point cloud using the intrinsic values of the camera with which the information was acquired. This process is known as deprojection; a point P with coordinates (X, Y, Z) can be obtained according to (2,3,4) from the depth information Dx,y being (x, y) the rectified position of the pixel in the sensor, where the variables cx, cy, fx, and fy are the intrinsic values of the camera used to acquire the information, with (fx, fy) as the components of the focal length and (cx, cy) the image projection center [29]. ...
... Once the depth images are available, it is possible to obtain the point cloud using the intrinsic values of the camera with which the information was acquired. This process is known as deprojection; a point P with coordinates (X, Y, Z) can be obtained according to (2,3,4) from the depth information D x,y being (x, y) the rectified position of the pixel in the sensor, where the variables c x , c y , f x , and f y are the intrinsic values of the camera used to acquire the information, with (f x , f y ) as the components of the focal length and (c x , c y ) the image projection center [29]. ...
The necessity for intelligent welding robots that meet the demand in real industrial production, according to the objectives of Industry 4.0, has been supported owing to the rapid development of computer vision and the use of new technologies. To improve the efficiency in weld location for industrial robots, this work focuses on trajectory extraction based on color features identification on three-dimensional surfaces acquired with a depth-RGB sensor. The system is planned to be used with a low-cost Intel RealSense D435 sensor for the reconstruction of 3D models based on stereo vision and the built-in color sensor to quickly identify the objective trajectory, since the parts to be welded are previously marked with different colors, indicating the locations of the welding trajectories to be followed. This work focuses on 3D color segmentation with which the points of the target trajectory are segmented by color thresholds in HSV color space and a spline cubic interpolation algorithm is implemented to obtain a smooth trajectory. Experimental results have shown that the RMSE error for V-type butt joint path extraction was under 1.1 mm and below 0.6 mm for a straight butt joint; in addition, the system seems to be suitable for welding beads of various shapes.
... The vision-based pose estimation [1][2][3] is to estimate the motion state of the moving object relative to the world coordinate system by using the moving target and its projection coordinates, mainly to obtain the three-dimensional pose. This can be attributed to the problem of solving the motion parameters of n-point perspective, which is the PnP (Perspective-n-Point) [4][5][6] problem. ...
We present UWSPSM, an algorithm of uncertainty weighted stereopsis pose solution method based on the projection vector which to solve the problem of pose estimation for stereo vision measurement system based on feature points. Firstly, we use a covariance matrix to represent the direction uncertainty of feature points, and utilize projection matrix to integrate the direction uncertainty of feature points into stereo-vision pose estimation. Then, the optimal translation vector is solved based on the projection vector of feature points, as well the depth is updated by the projection vector of feature points. In the absolute azimuth solution stage, the singular value decomposition algorithm is used to calculate the relative attitude matrix, and the above two stages are iteratively performed until the result converges. Finally, the convergence of the proposed algorithm is proved, from the theoretical point of view, by the global convergence theorem. Expanded into stereo-vision, the fixed relationship constraint between cameras is introduced into the stereoscopic pose estimation, so that only one pose parameter of the two images captured is optimized in the iterative process, and the two cameras are better bound as a camera, it can improve accuracy and efficiency while enhancing measurement reliability. The experimental results show that the proposed pose estimation algorithm can converge quickly, has high-precision and good robustness, and can tolerate different degrees of error uncertainty. So, it has useful practical application prospects.
... Instead of calculating the absolute positions of the vehicle and the target at the docking station, the estimated relative pose between them is input as feedback to the control system. This avoids the limitation [19], [20] of featurebased stereo vision, whose visual servoing technique is called image-based, 3D model-based recognition based on 3D-to-2D projection was applied in the proposed system. The 3D-to-2D projection has the merit of avoiding the ill-posed problem of the 2D-to-3D reconstruction approach that hinders the 3D pose estimation when trying to measure the pose from the decomposed 2D image information of dual-eye cameras. ...
This paper presents a stereo-vision-based approach for sea-bottom docking of autonomous underwater vehicles (AUVs) for battery recharging. According to the intended application, a unidirectional docking station was designed in which the AUV has to dock from a specific direction. Real-time relative pose (position and orientation) estimation was implemented utilizing three-dimensional model-based matching to the actual target and a real-time multi-step genetic algorithm. Using the proposed approach, we conducted the experiments in which an AUV docked to a simulated underwater battery recharging station in the sea near Wakayama City, Japan. The experimental results confirmed the functionality and potential of the proposed approach for sea-bottom docking of AUVs. Although similar sea trials were reported previously, detailed discussions and performance analyses were not presented, especially regarding the relations among pose estimation, output control voltage, and photographic records. The analyses confirmed that the successful docking was realized and that the method has tolerance against turbulence applied to a remotely operated vehicle near the docking station.
... Beside the surveillance systems, several applications utilize the proper camera placement such as: modeling indoor environments using an active range camera [7], estimating the camera parameters (i.e. calibration) [8], [9], achieving a visual coverage of a 3D object surface [10], 3D point reconstruction and estimating the object position based on stereo vision systems [11], [12]. ...
Optimizing the visual sensors coverage to perform security monitoring tasks has an undeniable impact on the performance as well as the cost of video surveillance systems. The placement arrangement is an NP-hard problem in which the main target is to seek an approximate solution. This paper addresses the placement of security cameras to maximize the total coverage of the camera network. The coverage of the visual sensor array is modeled descriptively using an enhanced pinhole camera model to obtain the layout of the field of view. The main contribution of the paper is to introduce a dynamic programming algorithm which defines an optimal policy for solving the visual sensor coverage problem. To validate the proposed algorithm, we compared the outcomes of the dynamic programming algorithm with existing benchmarking placement optimization techniques. The assessment reveals the effectiveness of the proposed dynamic programming solution to maximize the coverage for a set of predefined locations.
... However, the stereoscopic vision setup must be calibrated (we have to find the transformation matrix between R 1 and R w and the fundamental matrix witch allows to estimate the depth at scale factor. With the association to the CAD model based estimation of Z c , it is possible to affine the estimation of Z c ) and for three-dimensional reconstruction to be possible, the location and orientation of the cameras at different capture instants must be accurately known [23]. ...
Microassembly is an innovative alternative to the microfabrication process of MOEMS, which is quite complex. It usually implies the use of microrobots controlled by an operator. The reliability of this approach has been already confirmed for micro-optical technologies. However, the characterization of assemblies has shown that the operator is the main source of inaccuracies in the teleoperated microassembly. Therefore, there is great interest in automating the microassembly process. One of the constraints of automation in microscale is the lack of high precision sensors capable to provide the full information about the object position. Thus, the usage of visual-based feedback represents a very promising approach allowing to automate the microassembly process. The purpose of this article is to characterize the techniques of object position estimation based on the visual data, i.e., visual tracking techniques from the ViSP library. These algorithms enables a 3-D object pose using a single view of the scene and the CAD model of the object. The performance of three main types of model-based trackers is analyzed and quantified: edge-based, texture-based and hybrid tracker. The problems of visual tracking in microscale are discussed. The control of the micromanipulation station used in the framework of our project is performed using a new Simulink block set. Experimental results are shown and demonstrate the possibility to obtain the repeatability below 1 µm.
... Secondly, a second camera may be used. However, the stereoscopic vision setup must be calibrated and for threedimensional reconstruction to be possible, the location and orientation of the cameras at different capture instants must be accurately known [25]. Finally, the system may be completed by unidirectional sensors capable to estimate Z c coordinate such as laser sensors. ...
Micro assembly is an innovative alternative to the micro fabrication process of MOEMS which is quite complex. It usually implies the usage of micro robots controlled by an operator. The reliability of this approach has been already confirmed for the micro-optical technologies. However, the characterization of assemblies has shown that the operator is the main source of inaccuracies in the teleoperated micro assembly, so there is a great interest in automating the micro assembly process. One of the constraints of automation in micro scale is the lack of high precision sensors capable to provide the full information about the object position. Thus, the usage of visual based feedback represents a very promising approach allowing to automate the micro assembly process. The purpose of this paper is to characterize the techniques of object position estimation based on the visual data, i.e. Visual tracking techniques from the ViSP library. These algorithms allows to get the 3D object pose using a single view of the scene and the CAD model of the object. The performance of three main types of model based trackers is analyzed and quantified: edge-based, texture based and hybrid tracker. The problems of visual tracking in micro scale are discussed. The control of the micromanipulation station used in the framework of our project is performed using a new Simulink block set. Experimental results are shown and demonstrate the possibility to obtain the repeatability below 1 μm.
... Detection of low-dimensional features (edges, corners etc.) plays an important role in computer vision applications, such as image segmentation, 3D reconstruction, object recognition etc. [1][2][3]. Although the majority of images we collect currently are color images, current methods of detecting features only exploit gray information, such as Canny [4], Harris [5], Sift and so on. ...
Gray-based features are widely used in computer vision applications,
while image color is a very important source, which can provide more
feature information. To fully exploit color data, a color saturation
invariant based on dichromatic reflection model is first constructed.
The invariant is an object reflectance property independent of viewpoint
and illumination direction. The saturation invariant is then synthesized
with existing hue invariant to detect edge and corner features in color
image. Experiments show that the detection method proposed here can more
effectively tap into color information and achieve true target features
due to its lower sensitivity to shadow, shading and highlight. Moreover,
when comparing with many other existing edges and corners detecting
methods, experimental results demonstrate that the proposed method
performs better in detection accurate and effective.
