Fig 1 - uploaded by Jan Kybic
Content may be subject to copyright.
Lena test image with three rectangular test areas 1,2,3 (ROIs) with progressively decreasing level of detail.
Source publication
We address the problem of estimating the uncertainty of pixel based image registration algorithms, given just the two images to be registered, for cases when no ground truth data is available. Our novel method uses bootstrap resampling. It is very general, applicable to almost any registration method based on minimizing a pixel-based similarity cri...
Contexts in source publication
Context 1
... took the gray-scale 8-bit Lena image of size 512 512 pixels and selected three rectangular regions of interest (ROI) of size 61 61 containing high, medium, and low amount of texture and detail, respectively (Fig. 1). In each run, we have displaced the ROI with a randomly selected displacement uniformly distributed in the range pixels. We have perturbed both the original ROI and the displaced ROI with one of three types of noise: (i) uncorrelated zero-mean i.i.d. Gaussian (white) noise with varying standard deviation ; (ii) correlated Gaussian ...
Context 2
... second experiment studies the effect of varying the number of bootstrap resamples using the same setup as above (Section III-A) with uncorrelated additive Gaussian noise with standard deviation and region 1 (see Fig. 1). The graph in Fig. 3 shows the dependence of the coefficient of variation of the bootstrap estimate on the number of bootstrap re- samples for several different noise levels. We observe that the coefficient of variation decreases with but the decrease is slow and diminishes even further with increased noise level . This is in rough ...
Context 3
... error very well for low and medium SNR for all three criteria and for high SNR for the SAD and MI criteria. The difference for higher SNR for the correlation criterion only oc- curs at very high accuracy levels which are unlikely to appear in practice. Fig. 6. Geometrical RMSE in pixels as a function of the SNR in dB for position 1 in Fig. 1, as in Fig. 2. We compare the true error with the error " estimated by the bootstrap method for the (a) SAD, (b) normalized correlation, and (c) mutual information image similarity criteria. Each data point is an average of 100 ...
Citations
... This method is widely used in learning-based registration models, likely due to its straightforward implementation (Yang et al., 2016Madsen et al., 2020;Chen et al., 2022b;Xu et al., 2022). Bootstrap sampling, a traditional method, involves training the registration model multiple times on independent training sets to produce multiple inferences (Kybic, 2009). Snapshot sampling uses the cyclic learning rate in one training process for perturbing the model to converge to multiple different local minimums (Huang et al., 2017). ...
Over the past decade, deep learning technologies have greatly advanced the field of medical image registration. The initial developments, such as ResNet-based and U-Net-based networks, laid the groundwork for deep learning-driven image registration. Subsequent progress has been made in various aspects of deep learning-based registration, including similarity measures, deformation regularizations, and uncertainty estimation. These advancements have not only enriched the field of deformable image registration but have also facilitated its application in a wide range of tasks, including atlas construction, multi-atlas segmentation, motion estimation, and 2D-3D registration. In this paper, we present a comprehensive overview of the most recent advancements in deep learning-based image registration. We begin with a concise introduction to the core concepts of deep learning-based image registration. Then, we delve into innovative network architectures, loss functions specific to registration, and methods for estimating registration uncertainty. Additionally, this paper explores appropriate evaluation metrics for assessing the performance of deep learning models in registration tasks. Finally, we highlight the practical applications of these novel techniques in medical imaging and discuss the future prospects of deep learning-based image registration.
... Transformation uncertainty measures the local ambiguity of the spatial transformation (i.e., the deformation), whereas appearance uncertainty quantifies the uncertainty in the intensity values of registered voxels or the volumes of the registered organs. Transformation uncertainty estimates may be used for uncertainty-weighted registration (Simpson et al. 2011;Kybic 2009), surgical treatment planning, or directly visualized for qualitative evaluations (Yang et al. 2017b). Appearance uncertainty may be translated into dose uncertainties in cumulative dose for radiation or radiopharmaceutical therapy (Risholm et al. 2011;Vickress et al. 2017;Chetty and Rosu-Bubulac 2019;Gear et al. 2018). ...
In the last decade, convolutional neural networks (ConvNets) have been a major focus of research in medical image analysis. However, the performances of ConvNets may be limited by a lack of explicit consideration of the long-range spatial relationships in an image. Recently Vision Transformer architectures have been proposed to address the shortcomings of ConvNets and have produced state-of-the-art performances in many medical imaging applications. Transformers may be a strong candidate for image registration because their substantially larger receptive field enables a more precise comprehension of the spatial correspondence between moving and fixed images. Here, we present TransMorph, a hybrid Transformer-ConvNet model for volumetric medical image registration. This paper also presents diffeomorphic and Bayesian variants of TransMorph: the diffeomorphic variants ensure the topology-preserving deformations, and the Bayesian variant produces a well-calibrated registration uncertainty estimate. We extensively validated the proposed models using 3D medical images from three applications: inter-patient and atlas-to-patient brain MRI registration and phantom-to-CT registration. The proposed models are evaluated in comparison to a variety of existing registration methods and Transformer architectures. Qualitative and quantitative results demonstrate that the proposed Transformer-based model leads to a substantial performance improvement over the baseline methods, confirming the effectiveness of Transformers for medical image registration.
... This technique is corpora dependent since it relies on resampling observations from the corpora at hand (see Fig. 1 for an overview). Bootstrap resampling has been used for analyzing spatial point patterns (e.g., Solow, 1989), in astronomy (e.g., Barrow et al., 1984), image processing (e.g., Kybic, 2009), and genomics (e.g., Suzuki & Shimodaira, 2004) but has surprisingly been sparsely applied to educational research. While bootstrap resampling was recently employed to predict undergraduate engineering student performance (Taruna & Pandey, 2014), only a limited number of educational studies have empirically used this technique to examine variability in predictions of STEM classroom success. ...
Educators seek to develop accurate and timely prediction models to forecast student retention and attrition. Although prior studies have generated single point estimates to quantify predictive efficacy, much less education research has examined variability in student performance predictions using nonparametric bootstrap algorithms in data pipelines. In this study, bootstrapping was applied to examine performance variability among five data mining methods (DMMs) and four filter preprocessing feature selection techniques for forecasting course grades for 3225 students enrolled in an undergraduate biology class. While the median area under the curve (AUC) values obtained from bootstrapping were significantly lower than the AUC point estimates obtained without resampling, DMMs and feature selection techniques impacted variability in different ways. The ensemble technique elastic net regression (GLMNET) significantly outperformed all other DMMs and exhibited the least amount of variability in the AUC. However, all filter feature selection techniques significantly increased variability in student success predictions, compared to when this step was omitted from the data pipeline. We discuss the potential benefits and drawbacks of incorporating bootstrapping into prediction pipelines to track, monitor, and forecast classroom performance, as well as highlight the risks of only examining point estimates.
... utilisant des outils statistiques. Ici nous avons opté pour le bootstrap non paramétrique(Kybic, 2010) qui ne nécessite pas une hypothèse préalable par rapport à la distribution des données de base. Il est conseillé de générer un nombre important d'échantillons N (N ≥ n 2 , n étant la taille de l'échantillon de données). ...
Les changements environnemental et climatique ont conduit certains hydrosystèmes sahéliens de l’Afrique de l’Ouest à un comportement paradoxal : augmentation des coefficients d’écoulements malgré le déficit pluviométrique (1970-1990) et le reverdissement (1990 à nos jours). Le manque de données fiables de suivi de la dynamique environnementale en Afrique de l’Ouest a limité l’utilisation de modèles newtoniens pour appréhender ce comportement hydrologique paradoxal. L’objectif de cette thèse est donc de parvenir à une compréhension fine du comportement hydrologique des bassins versants du Sahel malgré le contexte d’insuffisance de données. La méthodologie adoptée a consisté premièrement en une analyse de la variabilité hydro-climatique et environnementale pour constater les changements subis par les bassins versants. Deuxièmement, l’application de modèle darwinien de type Budyko a permis de quantifier les contributions isolées et combinées des changements climatique et environnemental aux variations des écoulements, et d’établir la relation de coévolution climat-environnement à l’échelle de bassin versant. Enfin, cette relation de coévolution climat-environnement a été utilisée à travers le couplage d’un modèle de type Budyko au modèle hydrologique SWAT (Soil and Water Assessment Tool) pour prendre en compte la dynamique environnementale dans la simulation des écoulements. La partie sahélienne du bassin versant du fleuve Nakanbé (un affluent de la Volta au Burkina Faso) constituée de sept bassins emboîtés (38 – 21 178 km2) a été utilisée comme cadre d’application de la méthodologie développée sur la période 1965-2018. Les résultats des analyses de variabilité ont confirmé que les bassins emboîtés du Nakanbé à Wayen ont connu les deux paradoxes hydrologiques sahéliens et la période d’étude a été subdivisée en une période de base (1965-1977) et trois périodes d’impact : 1978-1994, 1995-2006 et 2007-2018. Les études d’impacts ont montré que la dégradation environnementale a été la principale cause du premier paradoxe hydrologique et que l’interaction climat-environnement est devenue prédominante au cours du deuxième paradoxe hydrologique sahélien. L’analyse de la coévolution a montré l’existence d’une relation de cause-effet entre le climat et l’environnement à l’échelle des bassins versants du Nakanbé et a indiqué que le paramètre du modèle Budyko de Fu est un indicateur de la variabilité spatio-temporelle de capacité de rétention en eau des sols. La prise en compte de l’évolution de la capacité de rétention en eau des sols dans la modélisation hydrologique au travers du couplage des modèles darwinen (Budyko de Fu) et newtonien (SWAT) a permis d’améliorer la simulation des écoulements des bassins versants du Nakanbé.
... Knowing about epistemic uncertainty helps determine if and to what degree the registration results can be trusted and whether the input data is appropriate for the neural network. The registration uncertainty estimates may be used for uncertainty-weighted registration (Simpson et al. 2011;Kybic 2009), surgical treatment planning, or directly visualized for qualitative evaluations (Yang et al. 2017b). Cui et al. (Cui et al. 2021) and Yang et al. (Yang et al. 2017b) incorporated MC dropout layers in the registration network design, which allows for uncertainty estimates by sampling multiple deformation field predictions from the network. ...
In the last decade, convolutional neural networks (ConvNets) have dominated the field of medical image analysis. However, it is found that the performances of ConvNets may still be limited by their inability to model long-range spatial relations between voxels in an image. Numerous vision Transformers have been proposed recently to address the shortcomings of ConvNets, demonstrating state-of-the-art performances in many medical imaging applications. Transformers may be a strong candidate for image registration because their self-attention mechanism enables a more precise comprehension of the spatial correspondence between moving and fixed images. In this paper, we present TransMorph, a hybrid Transformer-ConvNet model for volumetric medical image registration. We also introduce three variants of TransMorph, with two diffeomorphic variants ensuring the topology-preserving deformations and a Bayesian variant producing a well-calibrated registration uncertainty estimate. The proposed models are extensively validated against a variety of existing registration methods and Transformer architectures using volumetric medical images from two applications: inter-patient brain MRI registration and phantom-to-CT registration. Qualitative and quantitative results demonstrate that TransMorph and its variants lead to a substantial performance improvement over the baseline methods, demonstrating the effectiveness of Transformers for medical image registration.
... Hub et al. [10] proposed performing multiple registrations with perturbations in the B-spline grid ( [11]) as a measure of registration uncertainty. Kybic [12] proposed bootstrapping over pixels in the cost functions. Other approaches like block matching [13] and polynomial chaos expansions [14] are utilized in the context of detecting registration misalignment. ...
... Different from [25], the proposed method is capable of detecting relatively large registration misalignments. The inference time of the proposed method is approximately 2.8 seconds on a 3D patch of size 205 × 205 × 205, which is substantially faster than methods involving multiple registrations like [10,12,21]. ...
... We keep the misalignment labels of the last step equal to [0, 3), [3,6) and [6, ∞) mm by merging the auxiliary labels. Therefore, in the LSTM design with the 6-3-1 splitting approach, labels [0, 1), [1,3) are merged into a single label [0, 3), and in the LSTM design with the 12-6-3 splitting approach, labels [6,12), [12, ∞) are merged into a single label [6, ∞). The results are given in the bottom two rows in Table 1. ...
In this paper we propose a supervised method to predict registration misalignment using convolutional neural networks (CNNs). This task is casted to a classification problem with multiple classes of misalignment: “correct” 0-3 mm, “poor” 3-6 mm and “wrong” over 6 mm. Rather than a direct prediction, we propose a hierarchical approach, where the prediction is gradually refined from coarse to fine. Our solution is based on a convolutional Long Short-Term Memory (LSTM), using hierarchical misalignment predictions on three resolutions of the image pair, leveraging the intrinsic strengths of an LSTM for this problem. The convolutional LSTM is trained on a set of artificially generated image pairs obtained from artificial displacement vector fields (DVFs). Results on chest CT scans show that incorporating multi-resolution information, and the hierarchical use via an LSTM for this, leads to overall better F1 scores, with fewer misclassifications in a well-tuned registration setup. The final system yields an accuracy of 87.1%, and an average F1 score of 66.4% aggregated in two independent chest CT scan studies.
... In terms of popularity, bootstrapping has been longly used and studied by the robust statistics community [44,100,145], and became increasingly popular in the fields of machine learning [35,55,144] and computer vision [81,82,180]. More recently, the technique of bootstrapped ensembles has been applied to the uncertainty estimation of some classes of deep architectures as well, with applications in deep reinforcement learning [113,114] and computer vision [68,89]. ...
This work focuses on collaborative localization between a mobile camera and a static camera for video surveillance. In crowd scenes and sensitive events, surveillance involves locating the wearer of the camera (typically a security officer) and also the events observed in the images (e.g., to guide emergency services). However, the different points of view between the mobile camera (at ground level), and the video surveillance camera (located high up), along with repetitive patterns and occlusions make difficult the tasks of relative calibration and localization. We first studied how low-cost positioning and orientation sensors (GPS-IMU) could help refining the estimate of relative pose between cameras. We then proposed to locate the mobile camera using its epipole in the image of the static camera. To make this estimate robust with respect to outlier keypoint matches, we developed two algorithms: either based on a cumulative approach to derive an uncertainty map, or exploiting the belief function framework. Facing with the issue of a large number of elementary sources, some of which are incompatible, we provide a solution based on a belief clustering, in the perspective of further combination with other sources (such as pedestrian detectors and/or GPS data for our application). Finally, the individual location in the scene led us to the problem of data association between views. We proposed to use geometric descriptors/constraints, in addition to the usual appearance descriptors. We showed the relevance of this geometric information whether it is explicit, or learned using a neural network.
... and gradient of registration cost scores [15] as indicators of misalignment. Another approach is using bootstrap resampling to generate confidence measures [16]. Kybic proposed a faster algorithm that can achieve better performance than bootstrap resampling [17]. ...
Estimating registration error through independent directions, horizontal, vertical, and diagonal, is yet an unaccomplished task. If accurately done, this information can be used as feedback to further improve the registration quality. In this paper, we propose an algorithm for this purpose using a random forest regressor with features extracted using block matching. The proposed algorithm only requires two images as input and make predictions densely without requiring multiple registrations. The results on publicly available datasets show that the displacement of the best match after block matching provides strong cues about the registration error and the proposed algorithm is capable of estimating registration error through independent directions with high accuracy.
... However, such a mechanism is not readily available for registration strategies based on dense alignment, in which case previous works mostly resort to heuristics on the cost function (Gracias et al., 2004;Loewke et al., 2011;Peter et al., 2018;Sawhney et al., 1998) or to application-driven strategies such as the detection of the centerline in retinal imaging (Can et al., 2002;Yang and Stewart, 2004). The direct estimation of registration uncertainty given two registered images (Kybic, 2009), which can be seen as a generalisation of reliability assessment, is also a growing direction of research, especially in the context of medical image registration (Muenzing et al., 2012;Risholm et al., 2013;Sokooti et al., 2016). ...
Video mosaicking requires the registration of overlapping frames located at distant timepoints in the sequence to ensure global consistency of the reconstructed scene. However, fully automated registration of such long-range pairs is (i) challenging when the registration of images itself is difficult; and (ii) computationally expensive for long sequences due to the large number of candidate pairs for registration. In this paper, we introduce an efficient framework for the active annotation of long-range pairwise correspondences in a sequence. Our framework suggests pairs of images that are sought to be informative to an oracle agent (e.g., a human user, or a reliable matching algorithm) who provides visual correspondences on each suggested pair. Informative pairs are retrieved according to an iterative strategy based on a principled annotation reward coupled with two complementary and online adaptable models of frame overlap. In addition to the efficient construction of a mosaic, our framework provides, as a by-product, ground truth landmark correspondences which can be used for evaluation or learning purposes. We evaluate our approach in both automated and interactive scenarios via experiments on synthetic sequences, on a publicly available dataset for aerial imaging and on a clinical dataset for placenta mosaicking during fetal surgery.
... Its introduction by Efron in 1979, extending the famous Jackknife method [1], aimed to provide a new method for evaluating the estimator's performance. It served then in various applications, as examples, we can cite geophysics [2, 3], economics [4], signal processing [5] and especially image analysis [6,7]. Interested in accelerating statistical segmentation based on the family of Expectation-Maximization algorithms, the authors in [6] propose to resample the image by the Bootstrap with a reduced number of pixels to minimize on the one hand the computational time and on the other hand the correlation between the observations. ...
In this paper we intend to introduce a new representativeness criterion of the Bootstrap sample for images segmentation. Using the plug-in method in order to estimate probability density functions (pdf), we present a robust and stable criterion based on L 2 distance between the estimated probability density from the bootstrap sample and the empirical probability density of the image. This criterion is tested on satellite images.
