Figure 1 - uploaded by Antonio Agudo
Content may be subject to copyright.
A moving camera observes a non-rigid structure. The nodes where the external forces are acting and their magnitude are unknown. Boundary points undergo a rigid motion with respect to the camera. It is prior knowledge what nodes are boundary points.
Source publication
Bayesian EKF (Extended Kalman Filter) is cross-fertilized with Navier's equations solid deformation modeling to compute 3D non-rigid structure from monocular camera motion. The method operates with a projective camera and autonomously computes -for every sequence frame- both the geometry and the matches. The combination results in an sequential eff...
Context in source publication
Context 1
... estimate the structure and motion when observing a non-rigid scene by a hand-held monocular camera, eventu- ally in real time. We aim to process medical endoscope se- quences of body cavities such as the abdominal cavity. The cavity walls can be modeled as plates -thin solids-, and we assume a set of unknown magnitude forces acting on the surface Fig. 1. We propose to code 3D structure as a lin- ear elastic solid following the Navier's equations. Despite the proposed elastic model is a low cost one, only exact for small deformations, a eventually large scene deforma- tion can be accurately estimated, because the EKF is able to combine the available measurements of the deformed struc- ...
Similar publications
This paper proposes a pose-based algorithm to solve the full SLAM problem for an autonomous underwater vehicle (AUV), navigating in an unknown and possibly unstructured environment. The technique incorporate probabilistic scan matching with range scans gathered from a mechanical scanning imaging sonar (MSIS) and the robot dead-reckoning displacemen...
We present a visual odometry system for indoor navigation with a focus on long-term robustness and consistency. As our work is targeting mobile phones, we employ monocular SLAM to jointly estimate a local map and the device's trajectory. We specifically address the problem of estimating the scale factor of both, the map and the trajectory. State-of...
This paper presents an Invariant Information Local Sub-map Filter (IILSF) as a technique for consistent Simultaneous Localisation and Mapping (SLAM) in a large environment. It harnesses the benefits of sub-map technique to improve the consistency and efficiency of Extended Kalman Filter (EKF) based SLAM. The IILSF makes use of invariant information...
In this work the localization of a mobile robot
in an unknown environment is faced. A new version of the
Extended Kalman Filter (EKF) is presented. The proposed EKF
uses both measurements provided by robot on board and out of
board sensors in order to emphasize the qualities and overcome
the defects of such sensors. Moreover assuming a polynomial
m...
The mobile robot which accomplishes a work in explored region does not know location information of surroundings. Traditionally, simultaneous localization and mapping(SLAM) algorithms solve the localization and mapping problem in explored regions. Among the several SLAM algorithms, the EKF (Extended Kalman Filter) based SLAM is the scheme most wide...
Citations
... 2. Perform a standard Procrustes alignment. 3. Compute the mean shape of the current set of superimposed shapes. ...
... This metric is commonly used in the literature [5] or in mesh generator software such as gmsh. To run FE computations, it is often advised that the majority of elements has an AR superior to 0. 3. However, there is no theoretical back-up proving this threshold value. ...
... As mentioned in Paragraph 2.2. 3.5 the normality of the shape parameters has to be tested to validate the choice of the boundary shape model. In practice, when using the TPS-PR method only the 15 th liver mode and the first of the femur did not strictly follow a normal distribution (see Figures 2.2a and 2.2b). ...
This thesis investigates the use of model order reduction methods based on sparsity-related techniques for the development of real-time biophysical modeling. In particular, it focuses on the embedding of interactive biophysical simulation into patient-specific models of tissues and organs to enhance medical images and assist the clinician in the process of informed decision making. In this context, three fundamental bottlenecks arise. The first lies in the embedding of the shape parametrization into the parametric reduced order model to faithfully represent the patient’s anatomy. A non-intrusive approach relying on a sparse sampling of the space of anatomical features is introduced and validated. Then, we tackle the problem of data completion and image reconstruction from partial or incomplete datasets based on physical priors. The proposed solution has the potential to perform scene registration in the context of augmented reality for laparoscopy. Quasi-real-time computations are reached by using a new hyperreduction approach based on a sparsity promoting technique. Finally, the third challenge concerns the representation of biophysical systems under uncertainty of the underlying parameters. It is shown that traditional model order reduction approaches are not always successful in producing a low dimensional representation of a model, in particular in the case of electrosurgery simulation. An alternative is proposed using a metamodeling approach. To this end, we successfully extend the use of sparse regression methods to the case of systems with stochastic parameters.
... In recent literature of non-rigid reconstruction, Agudo et al. proposed the SLAM (Simultaneous Localization And Mapping) for elastic surface [10], where they have used the fixed position of the object boundary for image registration. They then later proposed a free boundary condition approach [11], [12] as an extension of their previous work. ...
... In this model, a noisy environment may create an accuracy error, besides the fact that errors may accumulate over the considered time frame. As such, these approaches [10]- [12] may not be able to exploit the mechanical constraints to cover a large deformation range. Additionally, rigid reconstruction techniques fail when applied directly to time-deforming objects. ...
... Combining the lateral strain equation (7) and force distribution equation (10), the relation between the force and change of radial pixel location can be represented in a general form for non normal axial loads, based on tilt angle and average load, as shown in (13) r (x,y) = r (x,y) ( ...
Breast lesion localization using tactile imaging is a new and developing direction in medical science. To achieve the goal, proper image reconstruction and image registration can be a valuable asset. In this paper, a new approach of the segmentation-based image surface reconstruction algorithm is used to reconstruct the surface of a breast phantom. In breast tissue, the sub-dermal vein network is used as a distinguishable pattern for reconstruction. The proposed image capturing device contacts the surface of the phantom, and surface deformation will occur due to applied force at the time of scanning. A novel force based surface rectification system is used to reconstruct a deformed surface image to its original structure. For the construction of the full surface from rectified images, advanced affine scale-invariant feature transform (A-SIFT) is proposed to reduce the affine effect in time when data capturing. Camera position based image stitching approach is applied to construct the final original non-rigid surface. The proposed model is validated in theoretical models and real scenarios, to demonstrate its advantages with respect to competing methods. The result of the proposed method, applied to path reconstruction, ends with a positioning accuracy of 99.7%.
... Deformation is described by stiffness parameters controlling behaviors of nodes sharing same edge. Some works have reported its effectiveness [21] [22]. No work demonstrates incrementally building finite element net and no complete implementations have presented the efficiency it solves dense deformable SLAM problem. ...
In this paper, we study the back-end of simultaneous localization and mapping (SLAM) problem in deforming environment, where robot localizes itself and tracks multiple non-rigid soft surface using its onboard sensor measurements. An elaborate analysis is conducted on conventional deformation modelling method, Embedded Deformation (ED) graph. We demonstrate and prove that the ED graph widely used in such scenarios is unobservable and leads to multiple solutions unless suitable priors are provided. Example as well as theoretical prove are provided to show the ambiguity of ED graph and camera pose. In modelling non-rigid scenario with ED graph, motion priors of the deforming environment is essential to separate robot pose and deforming environment. The conclusion can be extrapolated to any free form deformation formulation. In solving the observability, this research proposes a preliminary deformable SLAM approach to estimate robot pose in complex environments that exhibits regular motion. A strategy that approximates deformed shape using a linear combination of several previous shapes is proposed to avoid the ambiguity in robot movement and rigid and non-rigid motions of the environment. Fisher information matrix rank analysis with a base case is discussed to prove the effectiveness. Moreover, the proposed algorithm is validated extensively on Monte Carlo simulations and real experiments. It is demonstrated that the new algorithm significantly outperforms conventional rigid SLAM and ED based SLAM especially in scenarios where there is large deformation.
... In recent literature of non-rigid reconstruction, Agudo et al. proposed the SLAM (Simultaneous Localization And Mapping) for elastic surface [10], where they have used the fixed position of the object boundary for image registration. They then later proposed a free boundary condition approach [11], [12] as an extension of their previous work. ...
... In this model, a noisy environment may create an accuracy error, besides the fact that errors may accumulate over the considered time frame. As such, these approaches [10]- [12] may not be able to exploit the mechanical constraints to cover a large deformation range. Additionally, rigid reconstruction techniques fail when applied directly to time-deforming objects. ...
... Combining the lateral strain equation (7) and force distribution equation (10), the relation between the force and change of radial pixel location can be represented in a general form for non normal axial loads, based on tilt angle and average load, as shown in (13) r (x,y) = r (x,y) ( ...
Breast lesion localization using tactile imaging is a new and developing direction in medical science. To achieve the goal, proper image reconstruction and image registration can be a valuable asset. In this paper, a new approach of the segmentation-based image surface reconstruction algorithm is used to reconstruct the surface of a breast phantom. In breast tissue, the sub-dermal vein network is used as a distinguishable pattern for reconstruction. The proposed image capturing device contacts the surface of the phantom, and surface deformation will occur due to applied force at the time of scanning. A novel force based surface rectification system is used to reconstruct a deformed surface image to its original structure. For the construction of the full surface from rectified images, advanced affine scale-invariant feature transform (A-SIFT) is proposed to reduce the affine effect in time when data capturing. Camera position based image stitching approach is applied to construct the final original non-rigid surface. The proposed model is validated in theoretical models and real scenarios, to demonstrate its advantages with respect to competing methods. The result of the proposed method, applied to path reconstruction, ends with a positioning accuracy of 99.7%
... An analytical formulation was developed for isometric and conformal deformations using partial differential equations in [20], where the approach was developed for weak-perspective projection. A SLAM method for elastic surfaces with fixed boundary conditions was described in [17]. In [19], the authors formulate the reconstruction problem of a generic surface in terms of the minimization of the stretching energy and impose a set of fixed boundary 3D points to constrain the solution. ...
We deal with the reconstruction of surfaces that deform under a variety of conditions. The deformation can range from no extension to a certain degree of extensibility. The deformed surface is reconstructed from a single image, given a 3D reference shape. This shape corresponds to the undeformed state of the surface and can be computed using any appropriate technique. In particular, we use homographies defined from two views of the surface. To proceed with the 3D reconstruction of the deformed surface, we assume that the deformations are locally homogeneous and that the overall surface deformation can be obtained by combining the local homogeneous deformations. For this purpose, the surface is split into small patches. For each patch, a mapping between the undeformed and the deformed shapes is computed. The mapping is specified by using the quadratic deformation model Fayad et al. (Proceedings of British Machine Vision Conference (BMVC), 2004). As a result, given the undeformed shape, we define an optimization procedure whose goal is to estimate the 3D positions of deformed points in each image. The optimization is performed on each patch, independently of the others. The experimental results show that this approach allows precise reconstruction of a wide class of real deformations.
... Our sequential approaches assume that the shape at rest can be estimated similarly to batch [20,45] and sequential [1,40] approaches, using a rigid factorization algorithm [31] on a few initial frames. To this end, we consider that the observed sequence contains some nr initial frames where the object is mostly rigid and does not deform substantially. ...
This paper describes two sequential methods for recovering the camera pose together with the 3D shape of highly deformable surfaces from a monocular video. The nonrigid 3D shape is modeled as a linear combination of mode shapes with time-varying weights that define the shape at each frame and are estimated on-the-fly. The low-rank constraint is combined with standard smoothness priors to optimize the model parameters over a sliding window of image frames. We propose to obtain a physics-based shape basis using the initial frames on the video to code the time-varying shape along the sequence, reducing the problem from trilinear to bilinear. To this end, the 3D shape is discretized by means of a soup of elastic triangular finite elements where we apply a force balance equation. This equation is solved using modal analysis via a simple eigenvalue problem to obtain a shape basis that encodes the modes of deformation. Even though this strategy can be applied in a wide variety of scenarios, when the observations are denser, the solution can become prohibitive in terms of computational load. We avoid this limitation by proposing two efficient coarse-to-fine approaches that allow us to easily deal with dense 3D surfaces. This results in a scalable solution that estimates a small number of parameters per frame and could potentially run in real time. We show results on both synthetic and real videos with ground truth 3D data, while robustly dealing with artifacts such as noise and missing data.
... EKF-based approaches [12,16] were the first able to experimentally prove that visual SLAM in real-time is possible. More recently, the rigidity assumption was weakened in [1,4], allowing non-rigid deformations in the scene. To do this, the EKF estimator was combined with the Finite Element Method (FEM) to code the non-rigidity of the scene, resulting in a sfm system –denoted as EKF-FEM– able to deal with scenes containing both rigid and non-rigid map points. ...
... Our EKF-FEM-FRP algorithm allows us to estimate simultaneously both camera motion and deformable shape for each frame from a monocular video. Like in other nrsfm approaches [5,41,49], as we do not assume any point to be rigid, only the relative pose of the nonrigid scene with respect to the camera can be estimated, instead of the full 3D camera trajectory when several rigid points are used [1,4]. For this purpose, we propose a camera-centric EKF-based formulation coding shape relative to the last camera location [10]. ...
This paper describes a real-time sequential method to simultaneously recover the camera motion and the 3D shape of deformable objects from a calibrated monocular video. For this purpose, we consider the Navier-Cauchy equations used in 3D linear elasticity and solved by finite elements, to model the time-varying shape per frame. These equations are embedded in an extended Kalman filter, resulting in sequential Bayesian estimation approach. We represent the shape, with unknown material properties, as a combination of elastic elements whose nodal points correspond to salient points in the image. The global rigidity of the shape is encoded by a stiffness matrix, computed after assembling each of these elements. With this piecewise model, we can linearly relate the 3D displacements with the 3D acting forces that cause the object deformation, assumed to be normally distributed. While standard finite-element-method techniques require imposing boundary conditions to solve the resulting linear system, in this work we eliminate this requirement by modeling the compliance matrix with a generalized pseudoinverse that enforces a pre-fixed rank. Our framework also ensures surface continuity without the need for a post-processing step to stitch all the piecewise reconstructions into a global smooth shape. We present experimental results using both synthetic and real videos for different scenarios ranging from isometric to elastic deformations. We also show the consistency of the estimation with respect to 3D ground truth data, include several experiments assessing robustness against artifacts and finally, provide an experimental validation of our performance in real time at frame rate for small maps.
... This is a remarkable step-forward when compared to competing approaches, mostly constrained to inextensible surfaces or relying on relatively vast amounts of training data. A preliminary version of this work was presented in [1]. Here, we have pushed the limits of our model to show that it can cope with large elastic deformations. ...
... Finally, as said above, this paper is an extended version of [1]. Our recent work [2], also holds on [1], but only handles sets of non-rigid points, and not a combination of rigid and non-rigid point as we do here. ...
... Finally, as said above, this paper is an extended version of [1]. Our recent work [2], also holds on [1], but only handles sets of non-rigid points, and not a combination of rigid and non-rigid point as we do here. [2] results in a more general solution, but at the expense of lower estimation accuracy and most importantly, the inability to recover full camera trajectory. ...
We propose a new approach to simultaneously recover camera pose and 3D shape of non-rigid and potentially extensible surfaces from a monocular image sequence. For this purpose, we make use of the Extended Kalman Filter based Simultaneous Localization And Mapping (EKF-SLAM) formulation, a Bayesian optimization framework traditionally used in mobile robotics for estimating camera pose and reconstructing rigid scenarios. In order to extend the problem to a deformable domain we represent the object's surface mechanics by means of Navier's equations, which are solved using a Finite Element Method (FEM). With these main ingredients, we can further model the material's stretching, allowing us to go a step further than most of current techniques, typically constrained to surfaces undergoing isometric deformations. We extensively validate our approach in both real and synthetic experiments, and demonstrate its advantages with respect to competing methods. More specifically, we show that besides simultaneously retrieving camera pose and non-rigid shape, our approach is adequate for both isometric and extensible surfaces, does not require neither batch processing all the frames nor tracking points over the whole sequence and runs at several frames per second.
... In [14] a physical elastic solid model is coded by means of the boundary element method for 2D deformable object tracking, it is not necessary to identify boundary points in the image because 3 rank deficiency is enforced. Agudo et al. in [15] extend the EKF-SLAM method [16] to non-rigid scenes, they propose 2D thin-plate FEM model to embed Navier's equations within the EKF estimation. The method can combine both rigid and non-rigid points to estimate 3D reconstruction of deformable scenes and 3D camera trajectory using full perspective cameras. ...
... The thin-plate formulation is based on a 2D version of Eq. (1) to model forces and displacements within the plane (membrane effect), and on the Kirchhoff plate to model forces and displacements off the plane (bending effect). The discretized scene elements Ω e are planar triangles [15, 1]. In contrast, our 3D FEM formulation uses Eq. (1) directly to model the solid as a single layer 3D wedge element. ...
We present a new FEM (Finite Element Method) model for the 3D reconstruction of a deforming scene using as sole input a cal-ibrated video sequence. Our approach extends the recently proposed 2D thin-plate FEM+EKF (Extended Kalman Filter) combination. Thin-plate FEM is an approximation that models a deforming 3D thin solid as a surface, and then discretizes the surface as a mesh of planar trian-gles. In contrast, we propose a full-fledged 3D FEM formulation where the deforming 3D solid is discretized as a mesh of 3D wedge elements. The new 3D FEM formulation provides better conditioning for the rank analysis stage necessary to remove the rigid boundary points from the formulation. We show how the proposed formulation accurately estimates deformable scenes from real imagery even for strong deformations. Cru-cially we also show, for the first time to the best of our knowledge, NRSfM (Non-Rigid Structure from Motion) at 30Hz real-time over real imagery. Real-time can be achieved for our 3D FEM formulation combined with an EKF resulting in accurate estimates even for small size maps.
... Extended Kalman Filter (EKF) based approaches [6, 7] were the first computing SLAM in real-time, more recently keyframe based approaches [15] are yielding more accurate and efficient SLAM algorithms. In any case, Agudo et al. in [1] recently proved that EKF SLAM can successfully deal with scenes combining rigid and non-rigid points recovering both the camera trajectory and scene structure (Fig. 1(b)). The deformable scene is coded by means of Finite Element Method (FEM) modelling, it is assumed a set of Gaussian deforming forces acting on the non-rigid surface, resulting in the scene incremental displacement a due to the non-rigid displacement a nr . ...
... Greminger and Nelson propose in [12] coded an elastic solid by means of the Boundary Element Method (BEM) for 2D deformable object tracking; by enforcing a 3 rank deficiency, boundary conditions are removed from the formulation. Recently Agudo et al. in [1] propose linear FEM thinplate model to embed Navier's equations within the EKF estimation. They can deal with full perspective cameras and compute the data association, every single frame in the sequence is processed. ...
... The continuous displacement field is approximated by interpolation from the nodal displacements by means of the shape functions. In [1] it is proposed a thin-plate formulation to reduce the full 3D solid mechanic problem in a 2D problem, using the 2D plane stress model for membrane effect –displacement within the plane– and the Kirchhoff–Love plate model for bending effect –displacement off the plane. a i and forces f i in each node i are 6 d.o.f vectors defined as: ...
Navier's equations modelling linear elastic solid deformations are embedded within an Extended Kalman Filter (EKF) to compute a sequential Bayesian estimate for the Non-Rigid Structure from Motion problem. The algorithm processes every single frame of a sequence gathered with a full perspective camera. No prior data association is assumed because matches are computed within the EKF prediction-match-update cycle. Scene is coded as a Finite Element Method (FEM) elastic thin-plate solid, where the discretization nodes are the sparse set of scene points salient in the image. It is assumed a set of Gaussian forces acting on solid nodes to cause scene deformation. The EKF combines in a feedback loop an approximate FEM model and the frame rate measurements from the camera, resulting in an efficient method to embed Navier's equations without resorting to expensive non-linear FEM models. Classical FEM modelling has implied an interactive identification of boundary points to constrain the scene rigid motion, in this work this dissatisfying prior knowledge is no longer needed. The scene and camer rigid motion are combined in a unique pose vector and the estimation is coded relative to the camera. Additionally, the deforming effect of the Gaussian forces on the thin-plate is computed by means of the Moore-Penrose pseudoinverse of the FEM stiffness matrix. The proposed algorithm is validated with three real sequences gathered with hand-held camera observing isometric and non-isometric deformations. It is also shown the consistency of the EKF estimation with respect to ground truth computed from stereo.
