Recursive-iterative digital image correlation based on salient features

Full text

Recursive-iterative digital image
correlation based on salient features
Zhilong Su
Lei Lu
Xiaoyuan He
Fujun Yang
Dongsheng Zhang
Zhilong Su, Lei Lu, Xiaoyuan He, Fujun Yang, Dongsheng Zhang, “Recursive-iterative digital
image correlation based on salient features,”Opt. Eng. 59(3), 034111 (2020), doi: 10.1117/
1.OE.59.3.034111

Recursive-iterative digital image correlation
based on salient features
Zhilong Su,a,b Lei Lu,cXiaoyuan He,dFujun Yang,dand
Dongsheng Zhanga,b,*
aShanghai University, Shanghai Institute of Applied Mathematics and Mechanics,
School of Mechanics and Engineering Science, Shanghai, China
bShanghai Key Laboratory of Mechanics in Energy Engineering, Shanghai, China
cHenan University of Technology, College of Information Science and Engineering,
Zhengzhou, China
dSoutheast University, School of Civil Engineering, Nanjing, China
Abstract. Measuring surface deformation of objects with natural patterns using digital image
correlation (DIC) is difficult due to the challenges of the pattern quality and discriminative
pattern matching. Existing studies in DIC predominantly focus on the artificial speckle patterns
while seldom paying attention to the inevitable natural texture patterns. We propose a recursive-
iterative method based on salient features to measure the deformation of objects with natural
patterns. The method is proposed to select salient features according to the local intensity gra-
dient and then to compute their displacements by incorporating the inverse compositional
Gauss–Newton (IC-GN) algorithm into the classic image pyramidal computation. Compared
with the existing IC-GN-based DIC technology, the use of discriminative subsets allows avoid-
ance of displacement computation at pixels with poor spatial gradient distribution. Furthermore,
the recursive computation based on the image pyramid can estimate the displacements of the
features without the need for initial value estimation. This method remains effective even for
large displacement measurements. The results of simulation and experiment prove the method’s
feasibility, demonstrating that the method is effective in deformation measurement based on
natural texture patterns. ©2020 Society of Photo-Optical Instrumentation Engineers (SPIE) [DOI:
10.1117/1.OE.59.3.034111]
Keywords: digital image correlation; natural texture pattern; salient feature selection; image
pyramid representation.
Paper 191775 received Dec. 23, 2019; accepted for publication Mar. 11, 2020; published online
Mar. 25, 2020.
1 Introduction
Over recent years, there has been significant progress in estimating surface deformation with
optical measurement techniques, leading to large numbers of successful applications in industrial
and engineering fields. Among several optical deformation measurement techniques, digital
image correlation (DIC) is more attractive due to its merits, e.g., ease of operation and full-field
deformation measurement in a point-wise manner.1,2The accuracy of the state-of-the-art DIC
technique is up to 0.01 pixels when following well-controlled experimental conditions, such as
speckle pattern, parameter configuration, and illumination.3,4
In DIC, the robust subpixel displacement estimation is always the pursuit of researchers.
A plethora of research literature has been carried out in the research community.5–7Among
the existing methods, iterative registration of the local image intensities for estimating image
displacement viz the well-known subset matching has gained popularity. It originally appeared
in the Lucas–Kanade algorithm, which was proposed as a forward addition gradient descent
method8for image dense alignment. Based on this work, a classic iterative image matching
framework was established known as the forward additive DIC (FA-DIC) algorithm in
*Address all correspondence to Dongsheng Zhang, E-mail: donzhang@staff.shu.edu.cn
0091-3286/2020/$28.00 © 2020 SPIE
Optical Engineering 034111-1 March 2020 •Vol. 59(3)

deformation measurements.4,9Although many successful applications with FA-DIC can be
found, it comes with an expensive computational load because the Hessian matrix needs to
be continuously updated during the iteration process. To address the problem, a computationally
efficient variant of the FA-DIC algorithm was devised by authors,10 which they referred to as
the inverse compositional algorithm. In the field of optical deformation measurement, it was
improved by combining the zero-mean normalized sum of squared differences correlation metric
and Gaussian–Newton iterative strategy,11 establishing the well-known inverse compositional
Gauss–Newton (IC-GN) method. With the deepening of research on the shape or warp func-
tions,12 subpixel interpolation methods,13 efficiency and accuracy issues,14 etc., IC-GN has been
a popular approach for tackling high-precision subset matching problems in deformation meas-
urement. Therefore, the method presented in this paper advocates using this algorithm to search
the target points in the deformed images.
In order to obtain accurate displacement data, reliable initialization approaches should be
adopted in actual DIC calculations. A good initialization ensures the convergence and efficiency
in the subpixel matching. Initialization usually refers to a subset search at the integer-pixel level.
Several strategies have been reported to obtain the initial guess. Among them, the initial value
transfer scheme based on the seed point(s) is probably by far the most popular. With this strategy,
the quality and reliability guided search schemes for deformable image registration were
proposed.15,16 Although these methods help to suppress the error propagation caused by low-
quality computation points, the overall efficiency is limited because the estimation of displace-
ment at one point relies on its neighbors. For that, researchers have improved the initial
estimation in different ways. The cross correlation based on the fast Fourier transform is capable
of independently finding the initial value for each point of interest (POI), resulting in a path-
independent DIC that can be accelerated by massive parallelism.17 By combining particle
swarm optimization and block-based gradient descent search methods, the initial guess can
be estimated globally and simultaneously, thereby realizing real-time DIC technology based
on multithreading.18,19 In addition, the initialization based on scale-invariant features not only
has translation and rotation invariance, but also has the ability to perform large-scale parallel
calculations and it is also attractive and has been advocated to solve large deformation meas-
urement problems.20,21
Although great advances have been made in DIC techniques, most existing methods are
proposed based on artificial speckle patterns with good qualities. Therefore, several studies have
been reported from the perspective of improving the quality of speckle patterns, including the
optimization of speckle patterns22,23 and how to make the speckle patterns on the objects to be
measured.24 With the constant enlargement of the DIC application domain, the fabrication of
speckle patterns on the objects to be measured is difficult or limited in many cases, such as
in microscale or biomaterial measurements. In addition, in applications such as high-temperature
measurements, the quality of artificial speckle patterns may deteriorate. It is expected to measure
deformations based on the natural patterns or low-quality speckle patterns on the object surface.
Hence twin challenges, including whether the POIs can be reliably tracked and how to find their
position effectively in the deformed images, have yet to be solved. Despite these problems, in
practice, few studies have looked at the issue of obtaining reliable deformation measurement
results from natural or inferior patterns.
To fill this research gap, this paper proposes a recursive-iterative method to estimate displace-
ments at salient features in images. The method has a powerful potential to improve the accuracy
and stability of deformation measurements based on natural texture patterns. Since it is imple-
mented using image pyramidal computation, the proposed method is referred to as a salient
feature-based pyramid digital image correlation (SF-PyDIC). SF-PyDIC defines a salient feature
as the image subset that has a discriminative intensity distribution. Based on the iterative match-
ing algorithm IC-GN, a salient feature evaluation criterion is established for selecting POIs so
that they can be reliably tracked in the deformed images. This helps to circumvent pixels with
poor local textures. Subsequently, a recursive-iterative method based on image pyramid repre-
sentation and the IC-GN algorithm is introduced to search for salient features. By constructing
the image pyramid, the displacements of each feature are divided into sufficiently small areas
in each pyramid layer so that the IC-GN algorithm can be directly applied to estimate the dis-
placements recursively from the layer with the lowest resolution to the original. This process
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-2 March 2020 •Vol. 59(3)

does not require initial value estimation and can be used to solve problems with large
deformation.
The remainder of this paper is organized as follows: Sec. 2establishes some notations and
briefly describes the IC-GN algorithm in DIC. Section 3presents the SF-PyDIC method. The
salient feature selection strategy is introduced in Sec. 3.1. The displacement estimation strate-
gies, including image pyramid building and recursive salient feature matching, are described in
Sec. 3.2. Section 4gives experimental results. The accuracy and stability of the proposed SF-
PyDIC are investigated in Sec. 4.1, and the performance of the method in actual measurement is
studied in Sec. 4.2. Section 5concludes this paper.
2 Preliminaries
In deformation measurements with the DIC technique, the critical but challenging task is to
track a set of POIs in a deformable image sequence. The image corresponding to the initial
configuration of the object being measured often acts as the reference and is represented by
f. The image corresponding to a configuration after deformation is denoted by g. The quantities
fðxÞ¼fðx; yÞand gðxÞ¼gðx; yÞare then the intensity values of these images at the location
x¼½x; yT. Given any POI xin the reference image, the goal of DIC measurement is to estimate
its correspondence in the deformed image gand thus its displacement vector u¼½u; vT.
Among several DIC measurement techniques, estimating displacement by the local image
registration may be one of the most popular methods. This is implemented by mapping images
into the same coordinate system by finding the spatial correspondences between the reference
and deformed images. The IC-GN algorithm has been the state-of-the-art local image registration
framework in deformation measurements. Because of the high efficiency and robustness, it is
widely used to obtain subpixel displacement data nowadays.25 Considering the given POI xin
the reference f, IC-GN aims to find the correspondence xþuin the deformed image gby min-
imizing the dissimilarity between fðxÞand gðxþuÞin intensity appearance, where uis the
displacement vector to be estimated at point x. Because of the aperture problem, the intensity
dissimilarity is described by the sum of squared differences (SSD) correlation criterion. Let Mbe
the subset radius, then the displacement vector ucan be obtained by minimizing the following
SSD correlation function with the IC-GN algorithm:
EQ-TARGET;temp:intralink-;e001;116;355CðΔpÞ¼ X
xþM
η¼x−M
ff½xþWðη;ΔpÞ −g½xþWðη;pÞg2;(1)
where ηis the local coordinate relative to the point xin the subset, pdenotes the deformation
parameter vector containing the displacement uand its gradients, Δpis the incremental vector of
p, and Wð·Þis the shape function to characterize the deformation of the subset.26 The compo-
nents in pvary with the order of the shape function. It is worth mentioning that the zero-mean
normalized SSD criterion is recommended in the implementation to improve the robustness to
intensity variance to a certain degree.11
By applying the truncated first-order Taylor expansion to Eq. (1) with respect to the defor-
mation vector p, we obtain a linearized function of Δp:
EQ-TARGET;temp:intralink-;e002;116;208LðΔpÞ¼ X
xþM
η¼x−MfðxþηÞþ∇f∂W
∂pΔp−g½xþWðη;pÞ2
;(2)
where ∇f¼½
∂f
∂x;∂f
∂yis the gradient of image fat xþηand ∂W
∂pis the Jacobian of the shape
function. Taking the partial derivative of the linearized expression above with respect to Δp
and setting it to zero, we obtain the normal equations of the Gauss–Newton algorithm for solving
Δpas
EQ-TARGET;temp:intralink-;e003;116;105Hðx;ηÞΔp¼X
xþM
η¼x−M∇f∂W
∂pT
ϵðx;η;pÞ;(3)
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-3 March 2020 •Vol. 59(3)

where ϵðx;η;pÞ¼fðxþηÞ−g½xþWðη;pÞ is the point-wise intensity residual in the subset
and
EQ-TARGET;temp:intralink-;e004;116;711Hðx;ηÞ¼ X
xþM
η¼x−M∇f∂W
∂pT∇f∂W
∂p;(4)
is the Gauss–Newton approximation of the Hessian that is evaluated on the reference image.
Given an initial guess to the parameter vector pbeing estimated, the final deformation can
be estimated via solving Eq. (3) and updating parameters with the following inverse composi-
tional form:
EQ-TARGET;temp:intralink-;e005;116;608p←WðpÞ∘W−1ðΔpÞ;(5)
where ∘is the compositional operator and W−1is the inverse of the shape function. More details
about the inverse compositional parameter updating are given in Sec. 3.2.2.
3 Salient Feature and Displacement Estimation
In this section, we first introduce the criterion for selecting distinctive features in the reference
image. Then the recursive-iterative method, which estimates the displacements of salient features
by combining image pyramidal computation and the IC-GN algorithm, is established.
3.1 Salient Feature Selection
The fundamental concerns for the assessment of any displacement estimation algorithms are
accuracy and robustness, both of which are related to the quality of image patterns. However
in experiments with the natural surface patterns, the quality of the image is often hard to guar-
antee. To address the problem, an intuitive solution is to select pixels with discriminative local
intensity distributions in the reference image as the POIs, enhancing the stability of tracking from
the reference image to the frames after deformation. Shi and Tomasi27 have demonstrated that
features with good local texture can improve the robustness of matching, and a feature selection
criterion was established. By consideration of local subset deformation, we extend the feature
detection method to the field of optical deformation measurement and propose to define the
subsets that meet the discrimination criterion as salient features. For this point, we begin with
a revisit of the displacement estimation procedure in Sec. 2.
One can see that the critical step in estimating displacement using IC-GN is to compute the
deformation increment Δpaccording to Eq. (3). The condition is that the Hessian matrix H
should be well-conditioned to ensure the stability of the solution. Because the main purpose
is to compute the displacement components, the Hessian in Eq. (4) is further simplified to a 2×
2matrix corresponding to the displacement components as follows:
EQ-TARGET;temp:intralink-;e006;116;240H2×2¼X
xþM
η¼x−M
∇Tf∇f: (6)
Clearly, H2×2encodes the total intensity variation of the subset centered at xin both directions.
Let λmax and λmin be the largest and smallest eigenvalues of H2×2. Note that λmin should be greater
than zero because H2×2is often a positive definite symmetric matrix. Mathematically, it is
expected that the condition number λmax∕λmin of the matrix H2×2should be as small as possible
to ensure that the problem of displacement estimation is well-conditioned. In addition, the exist-
ence of image noise requires that λmin should be large enough to ensure H2×2is significantly
distinct from the noise level. If these conditions are satisfied, the subset surrounds xcan be stably
searched by the IC-GN optimizer. Thus, it is regarded as a salient feature. As the intensity varia-
tion in the image subset is limited by the grayscale range, the value of λmax is bounded in a finite
range rather than being arbitrarily large. This means that the salient features can be tailored by
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-4 March 2020 •Vol. 59(3)

the minimum eigenvalue value λmin. Therefore, the criterion for selection of salient features is
given by
EQ-TARGET;temp:intralink-;e007;116;565λmin >t; (7)
where tis a predefined positive threshold according to the quality of image patterns and the
desired density of salient features. A large threshold tusually produces features with high
saliency yet sparse distribution, whereas a small threshold tgenerates densely distributed fea-
tures but some have lower saliency. Despite tbeing determined heuristically, it could be deter-
mined according to the image noise level. In this study, we found that the determined features can
be tracked stably with the IC-GN optimizer when the threshold tis more than twice the noise
level. For images acquired by cameras, the noise level can be determined by the standard
deviation of Gaussian noise, which is often <0.5 in practice.
With the criterion in Eq. (7), a set of salient features can be selected as the POIs in the refer-
ence image by testing the minimal eigenvalue of the matrix in Eq. (6). Figure 1shows three
features selected from different image patterns. It can be seen that features corresponding to
the larger minimum eigenvalues often possess good pattern qualities, and thus could be more
suitable for displacement computation. The left feature is not suitable for matching because the
minimum eigenvalue value is even lower than the typical image noise level of 0.5. In contrast, the
feature on the right, which comes from a portion of speckle patterns, is more stable for comput-
ing as the minimum eigenvalue is very large (up to 37.38). (This seems to show why the speckle
patterns are the best choice for DIC measurement from another point of view.) In conclusion,
selecting the salient features as the POIs for the iterative IC-GN algorithm makes the displace-
ment computation more stable. In the following section, we present a recursive-iterative method
to estimate displacements of salient features without an initial guess.
3.2 Recursive-Iterative Displacement Computation
In displacement computation, a reliable search strategy for finding the corresponding spatial
position of each salient feature in the deformed image is required. Several existing methods,
including local subset matching16 and pyramidal feature tracking,28 are helpful to achieve this
goal. The former tracks deformed features by first obtaining an initial guess at the integer pixel
level and then performing an iterative optimization. However, a bad initial guess usually results
in failure on subpixel refinement, especially in large deformation problems. The latter provides
an alternative to solve this problem. The magnitude of displacements of a feature is reduced to a
limited range by shrinking the spatial resolution several times. This enables the displacements to
be estimated in recursion by directly using an iterative optimizer. If the IC-GN algorithm is used
as the optimizer, the pyramid tracking method can ensure inherent consistency from feature
selection to feature matching. Therefore, the recursive-iterative method is proposed to estimate
the displacements of the selected salient features by introducing the IC-GN algorithm into the
image pyramid calculation.
As a brief overview, the recursive-iterative displacement estimation consists of two steps:
(1) construction of image pyramid representation and (2) recursive displacement computation
using the IC-GN algorithm. Fine details are described as follows.
Fig. 1 Image feature samples with a size of 51 ×51 pixels and the minimum eigenvalues corre-
sponding to each feature sample.
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-5 March 2020 •Vol. 59(3)

3.2.1 Image pyramid representation
The pyramid representations of both reference and deformed images are constructed with a
downsampling scheme. To avoid high-frequency components of the signal alias into the low
frequencies, a low-pass filter produced by the outer product of a 1-D Gaussian kernel,
K¼½1;4;6;4;1, is adopted to generate low-resolution images. The kernel Kis a standard filter
used in the Laplacian pyramid,29 its outer product H¼KTKis a 5×5box filter that provides
adequate filtering at a low computational cost.
The image pyramid is finally built by stacking the original and downsampled images accord-
ing to the spatial resolution. The original image is on the bottom of the pyramid and each down-
sampled smaller image is stacked on top of the other. Let f¼f0be the original image in the
pyramid being built, the k’th layer image fkðk¼1;2;:::Þcan be generated from the layer fk−1
through the following strided convolution:
EQ-TARGET;temp:intralink-;e008;116;583fkðx; yÞ¼H⊛fk−1ð2x; 2yÞ:(8)
It is worth mentioning that the Gaussian kernel Kis normalized with a factor of 1
16 before the
convolution operation in practical implementation. Because the stride of the convolution above is
2, the size of the image fkis half of that of its upper neighbor fk−1, gaining a one-half reduction
in displacement magnitude. For example, the displacement vector of a point in an image with
a size of 1024 ×1024 pixels is ½15.0;15.0pixels. If the image size is compressed to
512 ×512 pixels, the displacement is thus reduced to ½7.5;7.5pixels. Following this point, the
iterative IC-GN algorithm could be used directly if the image is reduced enough. The maximum
value of kshould be configured heuristically according to the pre-evaluation of the maximum
displacement magnitude on the bottom layer and the acceptable minimum displacement on the
top layer. More details will be provided later.
3.2.2 Efficient recursive displacement estimation
The estimation of the displacement data from the reference and the deformed image pyramids is
implemented recursively. For this purpose, we start from the top of the image pyramids to find
the optimal correspondence of each salient feature layer by layer. Generally, the classic Lucas–
Kanade algorithm can be used in the process for a common feature matching task.28 However, it
does not work well for the displacement measurement tasks since the deformation of the subset is
not considered. Moreover, its forward additive iteration format increases the computational load
of the image pyramidal computation. Therefore, the IC-GN algorithm instead of the classic
Lucas–Kanade optimizer is adopted in this work in consideration of the subset deformation and
efficiency.
Without loss of generality, we introduce the recursive pyramidal displacement estimation on
a general pyramidal layer. Assume ffk↔gkgk¼0;1;:::;N to be an Nþ1tier pyramid pair with the
reference image fkand the deformed counterpart gk. For a given feature centered at xin the
original reference image f0, its counterpart in the downsampled image fkðk≥1Þis
xk¼x∕2kaccording to Eq. (8). The goal of pyramidal computation is to find the correspondence
of xkin the deformed image gk. Suppose the computation on layer kþ1ð≤NÞhas been finished,
obtaining a deformation vector pkþ1. Then the two displacement components in pkþ1can be
extracted to form a displacement vector ukþ1. According to the foregoing pyramid building proc-
ess, an initial matching position with xkin gkis derived as
EQ-TARGET;temp:intralink-;e009;116;175yk
0¼xkþ2ukþ1:(9)
With this initial position, the displacement vector is computed by finding the optimal correspon-
dence of xkwith the IC-GN algorithm. According to the normal equations in Eq. (3), the incre-
mental parameter vector Δpkis solved by
EQ-TARGET;temp:intralink-;e010;116;113Δpk¼H−1ðxk;ηÞX
xkþM
η¼xk−M
JTfkðxkþηÞ−H−1ðxk;ηÞX
xkþM
η¼xk−M
JTgk½xkþWðη;pkÞ;(10)
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-6 March 2020 •Vol. 59(3)

where J¼∇fk∂W
∂pkis the element-wise steepest descent contribution in the subset being used.
Because Jdoes not depend on the deformed image gk, the term JTand the inverse of Hessian
matrix H−1remain constant in iterations. Only the terms related to the deformed subset in gk
need to be updated [see Eq. (10)], meaning that all terms except for the deformed subset can be
precomputed to speed-up iterative computation process.
For the deformed subset gk½xkþWðη;pkÞ, the bicubic spline interpolation is employed to
compute the intensity values at subpixel positions. Considering that subpixel interpolation is
performed in multiple deformed image layers, a memory friendly yet efficient implementation
of the bicubic interpolation is recommended in the pyramidal computation.30 Suppose offsetting
of an integer pixel gðxÞto a subpixel position xþΔxwith 0≤Δx≤1and 0≤Δy≤1. The
intensity value gðxþΔxÞis computed by convolving the original image with the following
separable cubic kernel:
EQ-TARGET;temp:intralink-;e011;116;586h½Δx;gð∶;yÞ ¼ ½ 1ΔxΔx2Δx32
6
6
6
4
0100
−0.5 0 0.5 0
1−2.5 2 −0.5
−0.5 1.5 −1.5 0.5
3
7
7
7
5
2
6
6
6
4
gðx−1;yÞ
gðx; yÞ
gðxþ1;yÞ
gðxþ2;yÞ
3
7
7
7
5
;(11)
where gð∶;yÞ¼½gðx−1;yÞgðx; yÞgðxþ1;yÞgðxþ2;yÞ
T. Because the kernel is sepa-
rable, the interpolation is performed by applying Eq. (11) in the xdirection first to produce a
four-vector ^
g¼½hðΔx;y−1ÞhðΔx;yÞhðΔx;yþ1ÞhðΔx;yþ2Þ
T, and then in the y
direction to finally obtain gðxþΔxÞ¼hðΔy; ^
gÞ. For more details of the introduced interpola-
tion method, we refer to the original work.30
Once Δpkis obtained, the deformation parameters can be updated according to Eq. (5). For
the commonly used first-order shape function, the specific expression of the parameter updating
is given by
EQ-TARGET;temp:intralink-;e012;116;409WðpkÞ←WðpkÞ∘W−1ðΔpkÞ¼2
4
1þuk
xuk
yuk
vk
x1þvk
yvk
001
3
52
4
1þΔuk
xΔuk
yΔuk
Δvk
x1þΔvk
yΔvk
001
3
5
−1
;
(12)
where ∘is the composition operator10 and uk
x,vk
x,uk
y,vk
yare the displacement gradient compo-
nents in pk. The iteration procedure continues until the convergence condition kΔpkk≤τis
satisfied, where τis a predefined tolerance. The resulting displacement components in pkform
a new vector uktransferred to the next layer according to Eq. (9).
Letting Δuk
sbe the summation of the displacement vector increments obtained in each iter-
ation at the current layer, the initial guess of the displacement vector to be computed in the next
layer can be expressed as
EQ-TARGET;temp:intralink-;e013;116;248uk−1
0¼2ðuk
0þΔuk
sÞ;(13)
where uk
0¼ukþ1. A reasonable initial guess for the top layer can be uN
0¼0since the displace-
ment is often small enough after multiple downsampling operations. This shows that the dis-
placement estimation procedure based on the image pyramid is recursive. Supposing that the
maximum of Δuk
sat each layer is Δus, we can deduce that the recursive-iterative computation
can finally handle a displacement estimation on the bottom layer by up to
EQ-TARGET;temp:intralink-;e014;116;150umax ¼X
N
k¼0
2NΔus¼ð2Nþ1−1ÞΔus:(14)
This equation shows that the image pyramid representation is capable of computing large dis-
placement through maintaining a small overall displacement increment estimation at each layer.
Letting R¼kΔuskbe the convergence radius of the IC-GN algorithm, the relationship between
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-7 March 2020 •Vol. 59(3)

the displacement estimation capacity and the number of image pyramid layers can be obtained
from Eq. (14):
EQ-TARGET;temp:intralink-;e015;116;613N¼log2ðD∕Rþ1Þ−1;(15)
where Ddenotes the pre-estimated maximal displacement and Nis the maximal layer number.
With the recommended convergence radius R¼3 pixels in the study,31 Table 1briefly summa-
rizes several correspondences between Nand D.
Although the recursive-iterative computation method above yields good displacement esti-
mation, it comes with a high cost of computational time if the high-order shape functions are
used at all layers. A compromise is to perform displacement computations on the low-resolution
layers using the zero-order shape function, which characterizes point translation in the following
form:
EQ-TARGET;temp:intralink-;e016;116;486Wðx;pÞ¼10u
01v"x
y
1#;(16)
where the parameter vector p¼½u; vTcontains only displacement components. With the shape
function, the Hessian will be a 2×2matrix identical to that in Eq. (6), leading to a significant
improvement in iterative efficiency. Figure 2illustrates the overall pyramidal displacement esti-
mation process, where each pyramid is stacked by four image layers. One can see that the
pyramidal computation is divided into two phases. The zero-order IC-GN is applied to obtain
raw displacement data in the low-resolution layers, e.g., from layer 3 to layer 1. Then the first- or
second-order IC-GN is conducted on the original layer for refinement.
In conclusion, the improved pyramidal displacement estimation strategy inherits not only the
capacity of large displacement computation of pyramidal representation, but also the merits of
the IC-GN algorithm, including high efficiency and noise robustness.25 In addition, the zero-
mean normalized variant of the SSD criterion is recommended to further improve the robustness
to intensity variations.11
Table 1 Correspondences between the maximum displace-
ment and the maximum layer number.
N12 3 4 5 6
D(pixels) 9 21 45 93 189 381
Recursive
estimation
with zero-
order IC-GN
IC-GN
refinement
Downsampling Displacement transfer
Reference pyramid Deformed pyramid
Fig. 2 Schematic diagram of a four-layer pyramid representation and the recursive-iterative
displacement estimation.
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-8 March 2020 •Vol. 59(3)

4 Experiments
In this section, experiments were performed on simulated and real image data to verify the per-
formance of the proposed displacement estimation method. The natural surface patterns instead
of artificial speckle patterns were used in both experiments.
4.1 Simulation Results
In simulation, our goal was to explore the accuracy and stability of the proposed method. The
original image was captured from a specimen surface by a microscope camera with a resolution
of 1024 ×1024 pixels, as shown in Fig. 3(a). The specimen was made of Inconel 718 alloy with
a dimension of 20 ×20 ×2mm
3. A sequence of 20 images was then generated by translating the
original image with an incremental step of 0.05 pixels in the horizontal direction. The original
image was selected as the reference, and the displacements in the rest of the images were esti-
mated with the proposed SF-PyDIC method. For comparison, we also computed the displace-
ments with the classic pyramidal Lucas–Kanade (PyLK) method28 and the known reliability
guided DIC (RG-DIC),16 respectively.
In SF-PyDIC computing, about 2600 features were selected in the reference image according
to the criterion in Sec. 3.1. Some representatives are shown in Fig. 3(b). The subset size was
31 ×31 pixels and the threshold twas set to 3. One can see that the selected salient features are
distributed almost evenly in the image. Because the preassigned translation was small, a two-
layer image pyramid was built for each of the images in the simulation. The displacements of
each feature were computed from the image pyramids according to the proposed SF-PyDIC
method. The zero-order IC-GN was conducted to obtain raw displacements in the top layer and
the first-order IC-GN was conducted on the original layer for refinement. The PyLK method was
applied to the same features and the same image pyramids built in the SF-PyDIC computation.
For the use of RG-DIC, the computation step was set to 20 pixels to ensure the number of POIs
was consistent with the number of features used in the other methods. The subset size was the
same as that used in the SF-PyDIC.
To show the expected performance, we evaluated the displacement estimation errors of the
three methods, including the mean bias
EQ-TARGET;temp:intralink-;e017;116;367emean ¼1
nX
n
i¼1
^
ui−ureal;(17)
and the standard deviation error
EQ-TARGET;temp:intralink-;e018;116;312estd ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
1
n−1X
n
i¼1
ð^
ui−ureal −emeanÞ2
s;(18)
Fig. 3 (a) Natural pattern for generating image sequence and (b) part of salient features extracted
on the reference image.
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-9 March 2020 •Vol. 59(3)

where nis the number of features or POIs and ^
uis the measured displacement. Results are shown
in Fig. 4. It can be seen that both the mean and standard deviation of errors yielded by the
proposed SF-PyDIC are smaller than those produced by the PyLK and RG-DIC. Since the
SF-PyDIC and PyLK were conducted with the same computational configurations, the compari-
son of errors in Fig. 4demonstrates that the version in consideration of the local deformation has
higher accuracy and significantly improves the stability. Although the mean bias errors of the
RG-DIC are close to those of the SF-PyDIC except for the errors at displacement levels of 0.05
and 0.95, the standard deviation errors are much larger than those of the latter as indicated in
Fig. 4(b). The reason is that the regular grid nodes instead of salient features were adopted as the
POIs in the RG-DIC technique. The stability might be decayed due to the spatial gradient degen-
eration in the vicinities of those nodes with poor texture patterns. In contrast, the standard
deviation errors of the proposed SF-PyDIC are much lower and remain fairly consistent.
The comparison results clearly indicate that the proposed SF-PyDIC could obtain more accurate
and stable displacement measurement with the use of low-quality or natural surface patterns.
4.2 Real Experimental Results
This experiment was conducted to explore the performance of the proposed SF-PyDIC on a real
image sequence. The source images were collected in the process of fracture of a brittle material
using a scanning electron microscope (SEM) with a resolution of 2048 ×2048 pixels. For such a
microscopic experiment, it was difficult to fabricate high-quality speckle patterns on the surface
of the specimen. Therefore, the displacement data were computed according to the natural pat-
terns on the object surface.
Figure 5(a) shows an example of the natural surface patterns observed in the SEM. Clearly,
the pattern quality in the image is not as good as the man-made speckle patterns, such as the
speckles fabricated by the water-transferred technique.24 In order to compute the displacement
fields, a set of salient features, shown in Fig. 5(b), were extracted in the reference image by
applying the salient feature selection in Sec. 3.1. Subsequently, the displacements at every fea-
ture were computed with the proposed pyramidal estimation algorithm. Considering the fact that
the specimen underwent a large displacement due to the significant cracking, the maximum layer
number Nwas set to 4 in constructing the image pyramids. In stages of both feature selection
and displacement computation, the size of the subset was set to 31 ×31 pixels. The estimated
displacement fields in both directions are shown in Fig. 6. The results indicate that the upper and
lower parts of the specimen underwent significant rigid body motions, which was consistent with
the fracture phenomenon of brittle materials. The average horizontal displacements of the upper
and lower parts are up to about 24.8 pixels and 35.3 pixels, respectively; and the average vertical
displacements of the two parts are about −33.8 pixels and 19.1 pixels, respectively.
The displacement fields of the specimen were also computed with the RG-DIC technique for
comparison. To be successful in the process, the subset size was configured to a pretty big value,
61 ×61 pixels. Results are shown in Fig. 7. Clearly, there are conspicuous errors in the plotted
(a) (b)
–0.05
–0.03
0.00
0.03
0.05
0 0.2 0.4 0.6 0.8 1
Mean bias error (pixel)
Displacement (pixel)
SF-PyDIC PyLK RG-DIC
0.00
0.01
0.02
0.03
0.04
0 0.2 0.4 0.6 0.8 1
Standard deviation error (pixel)
Displacement (pixel)
SF-PyDIC PyLK RG-DIC
Fig. 4 Comparison of (a) mean bias errors and (b) standard deviation errors of horizontal displace-
ments estimated by SF-PyDIC, PyLK, and RG-DIC.
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-10 March 2020 •Vol. 59(3)

displacement fields. Compared with Fig. 6, it is obvious that the displacement fields obtained
by the proposed SF-PyDIC are much better and more reasonable. From the aspect of efficiency,
the seed points must be selected carefully for the RG-DIC until an acceptable initialization is
achieved. This point together with the correlation operation on the large subset resulted in the
400
800
1600
1200
20
24
28
32
36
40
2000
400 800 1200 1600 2000
(a)
u-displacement field (pixel)
400
800
1600
1200
–40
–30
0
20
30
2000
400 800 1200 1600 2000
(b)
–20
–10
10
v-displacement field (pixel)
Fig. 6 Displacement fields estimated by the proposed method: (a) u-displacement field and
(b) v-displacement field.
400
800
1600
1200
20
24
28
32
36
40
2000
400 800 1200 1600 2000
(a)
u-displacement field (pixel)
400
800
1600
1200
–40
–30
0
20
30
2000
400 800 1200 1600 2000
(b)
–20
–10
10
v-displacement field (pixel)
Fig. 7 Displacement fields estimated by the RG-DIC method: (a) u-displacement field and
(b) v-displacement field.
Fig. 5 (a) Reference image with natural patterns recorded in the SEM and (b) salient features
detected in the reference image.
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-11 March 2020 •Vol. 59(3)

long time elapsed before this task was finished with the RG-DIC. This experiment shows that the
proposed method is capable of computing good displacement data from low-quality speckle
patterns, even if the displacement magnitude is large.
5 Conclusion
A recursive-iterative method, referred to as SF-PyDIC, is introduced to estimate the displace-
ment fields of an object with natural textures or low-quality speckle patterns. This method is
a supplement to the existing DIC technology to enhance its ability of full-field measurement,
meeting the actual requirements of deformation measurement with natural texture patterns.
A salient feature selection criterion is established according to the iterative IC-GN algorithm,
so that the pixels with a salient spatial gradient distribution can be selected as POIs. By intro-
ducing the image pyramid representation, a recursive-iterative algorithm is proposed to estimate
the displacements of salient POIs using the IC-GN optimizer directly. We show that the displace-
ment estimation capability of the proposed SF-PyDIC method can be enhanced by appropriately
increasing the number of pyramid layers. Experiments based on simulated and real image data
show that the SF-PyDIC can achieve the expected performance. We expect that the SF-PyDIC
method can be potentially extended to full-field measurement areas where high-quality artificial
speckle patterns are limited.
Acknowledgments
This work was supported by the National Key R&D Program of China (Grant No.
2018YFF01014200) and the Natural National Science Foundation (NSFC) (Grant Nos.
11727804, 51732008, and 11672347) and Shanghai Postdoctoral Excellence Program (Grant
No. 2019192).
References
1. M. A. Sutton, J.-J. Orteu, and H. Schreier, Image Correlation for Shape, Motion and
Deformation Measurements: Basic Concepts, Theory and Applications, 1st ed., Springer,
New York (2009).
2. B. Pan, “Digital image correlation for surface deformation measurement: historical develop-
ments, recent advances and future goals,”Meas. Sci. Technol. 29(8), 082001 (2018).
3. M. Bornert et al., “Assessment of digital image correlation measurement errors: method-
ology and results,”Exp. Mech. 49, 353–370 (2009).
4. W. Tong, “Formulation of Lucas–Kanade digital image correlation algorithms for non-
contact deformation measurements: a review,”Strain 49(4), 313–334 (2013).
5. H. A. Bruck et al., “Digital image correlation using Newton–Raphson method of partial
differential correction,”Exp. Mech. 29, 261–267 (1989).
6. M. C. Pitter, C. W. See, and M. G. Somekh, “Subpixel microscopic deformation analysis
using correlation and artificial neural networks,”Opt. Express 8(6), 322–327 (2001).
7. H. Jin and H. Bruck, “Pointwise digital image correlation using genetic algorithms,”
Exp. Tech. 29(1), 36–39 (2005).
8. B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to
stereo vision,”in Proc. 7th Int. Joint Conf. Artif. Intell., Morgan Kaufmann Publishers Inc.,
San Francisco, California, Vol. 2, pp. 674–679 (1981).
9. P. Bing et al., “Performance of sub-pixel registration algorithms in digital image correla-
tion,”Meas. Sci. Technol. 17, 1615–1621 (2006).
10. S. Baker and I. Matthews, “Lucas–Kanade 20 years on: a unifying framework,”Int. J.
Comput. Vision 56, 221–255 (2004).
11. B. Pan, K. Li, and W. Tong, “Fast, robust and accurate digital image correlation calculation
without redundant computations,”Exp. Mech. 53, 1277–1289 (2013).
12. H. Lu and P. D. Cary, “Deformation measurements by digital image correlation: implemen-
tation of a second-order displacement gradient,”Exp. Mech. 40, 393–400 (2000).
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-12 March 2020 •Vol. 59(3)

13. Y. Su et al., “Noise-induced bias for convolution-based interpolation in digital image cor-
relation,”Opt. Express 24, 1175–1195 (2016).
14. Y. Gao et al., “High-efficiency and high-accuracy digital image correlation for three-dimen-
sional measurement,”Opt. Lasers Eng. 65,73–80 (2015).
15. L. Chen et al., “A quality-guided displacement tracking algorithm for ultrasonic elasticity
imaging,”Med. Image Anal. 13(2), 286–296 (2009).
16. B. Pan, “Reliability-guided digital image correlation for image deformation measurement,”
Appl. Opt. 48(8), 1535–1542 (2009).
17. Z. Jiang et al., “Path-independent digital image correlation with high accuracy, speed and
robustness,”Opt. Lasers Eng. 65,93–102 (2015).
18. R. Wu et al., “Real-time digital image correlation for dynamic strain measurement,”Exp.
Mech. 56, 833–843 (2016).
19. R. Wu et al., “Real-time three-dimensional digital image correlation for biomedical appli-
cations,”J. Biomed. Opt. 21(10), 107003 (2016).
20. Y. Zhang, L. Yan, and F. Liou, “Improved initial guess with semi-subpixel level accuracy in
digital image correlation by feature-based method,”Opt. Lasers Eng. 104, 149–158 (2018).
21. W. Li, Y. Li, and J. Liang, “Enhanced feature-based path-independent initial value estima-
tion for robust point-wise digital image correlation,”Opt. Lasers Eng. 121, 189–202 (2019).
22. Y. Su, Q. Zhang, and Z. Gao, “Statistical model for speckle pattern optimization,”Opt.
Express 25, 30259–30275 (2017).
23. Z. Chen et al., “Optimized digital speckle patterns for digital image correlation by consid-
eration of both accuracy and efficiency,”Appl. Opt. 57, 884–893 (2018).
24. Z. Chen et al., “A method to transfer speckle patterns for digital image correlation,”Meas.
Sci. Technol. 26, 095201 (2015).
25. X. Shao, X. Dai, and X. He, “Noise robustness and parallel computation of the inverse
compositional Gauss–Newton algorithm in digital image correlation,”Opt. Lasers Eng.
71,9–19 (2015).
26. X. Xu et al., “Effects of various shape functions and subset size in local deformation mea-
surements using DIC,”Exp. Mech. 55, 1575–1590 (2015).
27. J. Shi and C. Tomasi, “Good features to track[c],”in Proc. IEEE Conf. Comput. Vision and
Pattern Recognit., pp. 593–600 (1994).
28. J. Y. Bouguet, “Pyramidal implementation of the Lucas–Kanade feature tracker,”Intel
Corporation, Microprocessor Research Labs 4 (2000).
29. P. Burt and E. Adelson, “The Laplacian pyramid as a compact image code,”IEEE Trans.
Commun. 31, 532–540 (1983).
30. R. Keys, “Cubic convolution interpolation for digital image processing,”IEEE Trans.
Acoust. Speech Signal Process. 29(6), 1153–1160 (1981).
31. B. Pan, “An evaluation of convergence criteria for digital image correlation using inverse
compositional Gauss–Newton algorithm,”Strain 50(1), 48–56 (2014).
Zhilong Su received his PhD from the Department of Engineering Mechanics, Southeast
University, Nanjing, China, in 2019. Currently, he works at the School of Mechanics and
Engineering Science, Shanghai University, Shanghai, China. His research interests include opti-
cal measurement, visual deformation sensing, and numeric computation.
Xiaoyuan He received his BS degree from the Department of Applied Mechanics, Nanjing
University of Science and Technology, Nanjing, China, in 1982, his MS degree from the
Department of Mathematics and Mechanics, Southeast University, Nanjing, China, in 1987, and
his PhD from the Institute of Mechanics Southwest Jiaotong University, Chengdu, China, in
1994. Currently, he is a professor in the Department of Engineering Mechanics at the Southeast
University. His current research interest is photomechanics.
Dongsheng Zhang received his PhD from Tianjin University in 1993. He is now a full professor
at Shanghai Institute of Applied Mathematics and Mechanics of Shanghai University. His
research interests include advanced opto-mechanics and its applications.
Biographies of the other authors are not available.
Su et al.: Recursive-iterative digital image correlation based on salient features
Optical Engineering 034111-13 March 2020 •Vol. 59(3)