Image Completion: Survey and Comparative Study

Abstract

Image completion is an active and interesting research area in image processing and computer graphics. Restoration and retouching of damaged areas in an undetectable form is the objective of image completion techniques. Most of the recently developed video completion methods are extensions of image completion techniques to restore the damaged frames. With respect to video completion challenges and image completion future work, we survey existing methods and introduce a new classification. The methods in each category are described in detail. In the second part of the paper, we provide a comparison and evaluation study between the most recent image completion methods qualitatively as well as quantitatively. For a fair comparison, we introduced a new dataset and evaluated four available image completion methods on the same hardware. Experimental results are conducted to highlight the strengths and drawbacks of each image completion method.

Full text

Image Completion: Survey and Comparative Study
Sameh Zarif
*
,
†
,
§
, Ibrahima Faye
*
,
‡
,¶
and Dayang Rohaya
*
,
†
,||
*
Centre of Intelligent Signal and Imaging Research (CISIR)
Universiti Teknologi Petronas, Malaysia
†
Department of Computer and Information Sciences
Universiti Teknologi Petronas, Malaysia
‡
Department of Fundamental and Applied Sciences
Universiti Teknologi Petronas, Malaysia
§
sameh.shenoda@ci.menofia.edu.eg
¶
ibrahima_faye@petronas.com.my
||
roharam@petronas.com.my
Received 29 August 2014
Accepted 17 December 2014
Published 16 February 2015
Image completion is an active and interesting research area in image processing and computer
graphics. Restoration and retouching of damaged areas in an undetectable form is the objective
of image completion techniques. Most of the recently developed video completion methods are
extensions of image completion techniques to restore the damaged frames. With respect to video
completion challenges and image completion future work, we survey existing methods and
introduce a new classi¯cation. The methods in each category are described in detail. In the
second part of the paper, we provide a comparison and evaluation study between the most
recent image completion methods qualitatively as well as quantitatively. For a fair comparison,
we introduced a new dataset and evaluated four available image completion methods on the
same hardware. Experimental results are conducted to highlight the strengths and drawbacks of
each image completion method.
Keywords: Image inpainting; image completion; texture synthesis; object removal; image
retouching.
1. Introduction
Completion and reconstruction of damaged areas of digital images is an interesting
trend in computer vision and image processing since 2000. Recently, digitalization of
cultural and historical images has become an important step which has been ex-
tensively used in artwork restoration. Historical images can be degraded or damaged
due to removing unwanted object, error compression or transmission, bad environ-
ment, water damage, and some may have logos or stamps.
14,137,138
§
Corresponding author.
International Journal of Pattern Recognition
and Arti¯cial Intelligence
Vol. 29, No. 3 (2015) 1554001 (53 pages)
#
.
cWorld Scienti¯c Publishing Company
DOI: 10.1142/S0218001415540014
1554001-1
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Numerous terms, from di®erent applications, are used for image completion.
Error concealment is used in telecommunications application to represent the
completion of lost blocks during transmission. Image disocclusion is used in oc-
clusion handling applications to repair the occluded background object after re-
moving the occluding foreground object. Texture synthesis is used and is suitable
for repairing large missing regions especially if the damaged area needs to be ¯lled
with some texture. Image inpainting or retouching is one of the most common
term. It comes from art work restoration and is used to ¯ll in small damaged region
with structure information. Most of natural images usually contain both structure
and texture information. So, neither image inpainting nor texture synthesis is
enough for ¯ll in natural image. Finally, image completion is the popular term
nowadays that combines both texture synthesis and inpainting techniques to
complete large missing area on the image using both texture and structure
information.
98,113
The goal of image completion is to restore and complete the missing portion from
the surrounding structure and texture information in a nondetectable way for any-
one who is not aware of the original image. There are several applications of image
completion including recovering lost blocks in wireless image transmission, removal
of objects as special e®ects in image forgery, scratch removal in historical image
restoration, removal of occlusions such as text, subtitle, logos, and stamps. Other
applications include removing annotations like orientation and location from medi-
cal, aerial, and military images. Moreover it is one of the key approaches in cinema
post production. Post production ¯lms companies has signi¯cant budget allocated
for visual story modi¯cation which saves cost since it is more expensive to reshoot the
scene.
14,113,139
Traditionally, skilled artists have performed the restoration of image
manually as illustrated in Fig. 1. Now digital techniques are used for automatic
images restoration.
The region to be restored in image completion is a group of pixels that may or may
not be connected. The group of pixels is called artifact, gap, hole, scratch, missing
area, target region or unknown region and is commonly indicated by . Virtually all
inpainting cases require some kind of interaction by the user. Often, the user must
select the damaged region manually because the de¯nition of the region or distortion
in the image is largely perceptual. There is no method capable of detecting or
knowing automatically the region to be ¯lled.
Image inpainting, texture synthesis, and image completion are three di®erent but
related approaches. Many research papers used the terms inpainting, texture syn-
thesis, and completion with the same meaning and other papers used the term
inpainting as a general term for all cases. The di®erences between these related
techniques are the size of the region or hole to be restored and the type of data to ¯ll
in the region. The common requirement of all the related techniques is that the
region to be restored is manually selected by the user. To prevent the confusion, we
discuss the di®erence between the techniques in the next section.
S. Zarif, I. Faye & D. Rohaya
1554001-2
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

2. Related Work
In this section, we brie°y present an overview of the existing survey papers on image
completion followed by the organization of our work. Due to the rapid growths in the
recent years in the ¯eld of image completion and the ¯eld is recent, there are
shortages in the existing surveys and some of them are out of date. In Ref. 109, the
authors only present the structure-based approaches for repairing small damage
area. A part from the paper in Ref. 117 covers the structure and texture inpainting
and its relation to image-based rendering. This paper is out of date nowadays and
does not cover the new trends in the ¯eld. There are few recent survey papers
presented in Refs. 12,65,90 and 101. The authors introduced very short surveys on
image completion approaches without any comparative study. Theoretical analysis
of image inpainting methods and comparison study are presented in Ref. 68. Over-
view and recent advances in image inpainting is discussed in Ref. 55.
For more readable and organized survey, the ¯rst part of this paper reclassi¯es the
completion techniques into new categories as illustrated in Fig. 2. Based on the
number of source image, completion techniques are divided into single source image
methods and multiple source images methods. In all single source-based techniques,
the surrounding pixels on the image are used to complete the damaged region.
According to the size of the damaged region and the nature of the information
that is used for completion, the single-based approaches can be classi¯ed into
Fig. 1. Historical image inpainting examples, ¯rst row presents the original images, second row presents
the inpainted images.
Image Completion
1554001-3
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

structure-based (commonly called image inpainting), texture-based (commonly
called texture synthesis), and hybrid-based methods (commonly called image com-
pletion). The structure-based methods can be categorized into partial di®erential
equation (PDE), convolution as well as wavelet-based methods. The texture-based
approaches can be separated into statistical-, pixel-, and patch-based methods.
Hybrid-based methods can be divided into decomposition and exemplar-based
methods. In the following sections, the weaknesses and strengths for each category
are discussed.
With the advent of sparse representation, sparse prior has been intensively
studied for solving image inpainting and recovery problems. Few sparse coe±cients
can approximately represent the whole image or patch in a given basis (e.g. discrete
cosine transform (DCT), Fourier, wavelet, curvelets, contourlets, bandlets, wedge-
lets). The sparse vector of an image is basis dependent. The damaged pixels are
approximately completed from the generated sparse vector. Several methods in
di®erent categories have used sparse representation. These methods will be explained
in their corresponding categories.
55,98
In the second part of this paper, we introduce a comparative study based on the
same dataset as well as the same machine hardware. Initially, a new dataset con-
sisting of small and large damaged region images, texture, structure, and natural
images, is introduced. Then, four available methods are implemented on the dataset.
For a comprehensive and fair evaluation, the same dataset, hardware, and quality
measurements are used in the comparison.
In order to cover most of the previous works in each category and make the review
as easy as possible for the reader, we brie°y present the main idea of each method
combined with illustrations. For more mathematical details, we encourage the in-
terested reader to look into the given references.
The remaining part of this paper is organized as follows. In Sec. 3, we present
more details about single source image-based approaches. An overview of multiples
source images-based approaches is introduced in Sec. 4. In Sec. 5, a comparative
CompleƟon
Approaches
Single Source
Image
Structure
Based
PDE and
VariaƟonal
Based
ConvoluƟon
Based
Wavelet
Based
Texture
Based
StaƟsƟcal
Based
Pixel
Based
Patch
Based
Hybrid
Based
DecomposiƟon
Based
Exemplar
Based
MulƟple source
Images
Fig. 2. Image completion classi¯cation block diagram.
S. Zarif, I. Faye & D. Rohaya
1554001-4
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

study and the experimental results are presented. Finally, the conclusion is drawn
in Sec. 6.
3. Single Source Image-Based Approaches
Most of the completion algorithms use the surrounding known information in the
damaged image to repair the hole. Some of them, in the texture-based category, use
only a sample seed texture. Based on the size of the hole and the type of image, the
methods are classi¯ed into structure-, texture-, and hybrid-based methods. In the
following subsections, we review the available methods in each category.
3.1. Structure-based methods
The continuity of geometrical structure information is the main interest of the
structure-based methods. Most of them handle only nontextured image with small
damaged region. Isophotes (line of equal gray values) is the conceptualized term of
image structure. The basic idea of structure-based methods is to smoothly connect
the isophotes and level curves inside the hole in a proper way. The structure-based
methods can be classi¯ed into three categories as shown in Fig. 2. In the next sub-
section, each category is described in detail.
3.1.1. PDE- and variational-based methods
One of the ¯rst e®orts that preserve image structure is made by Masnou and
Morel.
95
It solves the completion problem as a disocclusion problem. This method
tried to connect the line of equal gray value located at the boundary region based
on geodesic curve. The geodesic curve connects the lines that are having the same
color and orientation based on shortest possible path between two points. The
method assumes that the connecting isophotes never intersect each other. Theo-
retical justi¯cation and more details are given in Ref. 94. There are many lim-
itations including quite trivial method for connecting isophotes, angles of
connection are not preserved, and the methods handle only very small damaged
region.
Bertalmio et al.
14
pioneered a digital image inpainting algorithm based on higher
order PDE. This method is considered as an extension of the level lines-based dis-
occlusion method in Ref. 95. Based on the same idea, the methods iteratively
propagate information from the outside of the area along the level lines isophotes.
The direction of the largest spatial change is used to maintain the angles of con-
nection. A discretized gradient vector is used to obtain the angle direction for each
pixel located at the region boundary. Anisotropic di®usion is used instead of geodesic
curve to connect the isophotes as in Eq. (1).
unþ1ði;jÞ¼unði;jÞþtun
tði;jÞ;8ði;jÞ2;ð1Þ
ut¼rðuÞr?u;ð2Þ
Image Completion
1554001-5
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

where nrepresents the iteration number, inpainting rate of change is represented by
t,un
tði;jÞis the updated factor of image unði;jÞ. The Laplacian operation is used to
smooth the update factor as in Eq. (2) where r?urepresents the isophotes direction
and rðuÞis the gradient Laplacian smoothness. This method works well for small
and structure damaged region. It takes more time to inpaint small regions. In ad-
dition to that, it produces blurring results due to the smoothness process. Figure 3
illustrates the di®erence between the isophotes connection of the methods in Refs. 14
and 95.
Bertalmio et al.
11
improved their method in Ref. 14 by introducing Navier–Stokes
for °uid dynamic equations as illustrated in Eq. (3), where vand prepresent the
velocity and pressure, respectively. This method uses a nonlinear PDE to connect the
isophotes. Stream function for a 2D incompressible °ow is used to represent the in-
tensity of the image as in Eq. (4), where the 2D vorticity is represented by !¼rv.
The vector produced from the stream function is used to propagate outside information
into inside the hole by solving a vorticity transport equation as in Eq. (5), where the
anisotropic di®usion of the smoothness !is represented by the function g.
vtþvrv¼rpþv;rv¼0;ð3Þ
!tþvr!¼!; ð4Þ
@!=@tþvr!¼rðgðjr!jÞr!Þ:ð5Þ
This improvement increases the stability and speed of the previous method. The
method still produces blurred results and works only for small structured damaged
regions. Moreover it requires from the user to tune the parameters.
Chan and Shen
26,27
proposed two image inpainting algorithms based on total
variation (TV) and on the curvature-driven di®usion (CDD). The TV is a PDE-
based method that uses an Euler–Lagrange equation and an anisotropic di®usion to
prorogate the information from outside to inside the damaged region based on the
Fig. 3. Isophotes connection, left image presents incomplete isophotes, middle image presents completed
isophotes by method in Ref. 95, right image presents the completion by method in Ref. 14.
S. Zarif, I. Faye & D. Rohaya
1554001-6
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

contrast of the isophotes as in Eq. (6).
@u
@t¼r ru
jruj

þeðuþu0Þ:ð6Þ
In the above equation, e¼ð1xDÞ; represents the TV denoising scheme and
xD is the inpainting mask. This method was designed for only inpainting small
regions. It fails to connect broken edges such as embedded lines in a uniform
background.
The CDD method extended the TV method to take into account the geometric
information of isophotes. It modi¯es the coe±cient of conductivity to be stronger
when the isophotes are having large curvature as shown in Eq. (7), where gis the
annihilator function of large curvatures and stabilizer of small ones. The known
information of the image is represented by u0.
@u
@t¼r gðjkjÞ
jrujru

;X2D;
u¼u0;X2Dc;
8
>
<
>
:ð7Þ
k¼r ru
jruj

:ð8Þ
The curvature kat a pixel Xis given by Eq. (8). The method allows the inpainting
to proceed over larger areas. CDD can connect some broken edges, but the inpainted
results usually look blurred. In a later paper,
25
the authors tried to study and
combine the properties of the method in Ref. 95 with the methods in Refs. 14,27
and 49 to solve the connectivity problem.
Esedoglu and Shen
49
proposed two nontexture inpainting methods to produce
more natural visual e®ect. The ¯rst method extends the TV method in Ref. 26 by
using Mumford–Shah image model to decrease the order of approximation and
computation. In this method, -convergence approximation is applied to accelerate
the numerical convergence and makes it simple. Equation (9) illustrates the com-
bination of Mumford–Shah and -convergence to solve the inpainting problem.
J½u;zju0;D¼ 1
2Z
DðxÞðuu0Þ2dx þ
2Z
z2jruj2dx
þZ
jrzj2þð1zÞ2
4

dx:ð9Þ
This method prefers straight edges, so that it produces arti¯cial corners. Moreover it
violates the connectivity principle. The second method improves the former one by
using high-order correction of the Euler's elastic curve model instead of straight line
curve as shown in Eq. (10).
eðÞ¼Zðþk2Þds ¼lengthðÞþZ
k2ds;ð10Þ
Image Completion
1554001-7
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

where krefers to the curvature, ,are positive tunable weights, and ds represents
the element length. Convergence, energy, and stability are the main problems of
these methods.
Jiang and Moloney
67
presented a PDE-based method especially for transmission
error concealment. The completion of missing blocks is based on modi¯ed orientation
di®usion model stage and intensity di®usion stage. Smoothness and orientation
constraints have been introduced for the optimization problem. Combination of
intensities and orientations stages is used to maximize the smoothness around the
reconstructed region as shown in Eq. (11).
uxxcos 2^
þuyysin 2^
ðuxy þuyx Þcos ^
sin ^
¼0;ð11Þ
where ^
is the estimated orientation, cos ^
and sin ^
are the gradient direction in-
tensities. Finally, numerical discretization based on linear estimation is presented to
implement the method without iterative operations. This method gives satisfactory
results for small lost blocks that have not manifested singularities orientation.
Gu et al.
54
pioneered Monte Carlo di®usion inpainting method based on Euler
energy. This method extends the approaches in Refs. 25 and 94 by improving the
Mumford–Shah image model as illustrated in Eq. (12). It interpolates the smooth
area of the damaged region by using random simulation of integral boundary as
shown in the ¯rst term of Eq. (12). Snake elastic of the active contour is used in the
second term to connect the broken edges around the boundary.
E½u;¼
2Znjruj2dx þZðjX0ðsÞj2þjX00ðsÞj2Þds:ð12Þ
The smoothing region is presented by n, where denotes the edges, and s
represents the arc-length. and have been set to 0.3 and 0.7, respectively. The
method is faster than previous methods but fails to ¯t long narrow areas.
Telea
118
proposed a fast and simple marching algorithm that can be looked as a
PDE-based approach without the computational overheads. The algorithm repairs
the damaged pixels as a level sets by propagating image estimator along image
gradient to simplify the computation of °ow. The algorithm used the known pixels
neighborhood to calculate smoothness of the image as a weighted average to inpaint
the region as illustrated in Eq. (13).
uðpÞ¼Pq2B"ðpÞwðp;qÞ½IðqÞþrIðqÞðpqÞ
Pq2B"ðpÞwðp;qÞ;ð13Þ
where pand qpresent the damaged and known pixel, respectively, wðp;qÞis the
normalized weight function, and B"ðpÞis a known neighborhood of the pixel p. The
limitation of this method is the apparition of blur in the result when the region to be
inpainted is thicker than 10 pixels.
Wang et al.
127
presented an image inpainting method based on isophotes prop-
agation for handling thick region. This method replaces the 2D smoothness
S. Zarif, I. Faye & D. Rohaya
1554001-8
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

operations like anisotropic di®usion with 1D smoothness along the isophotes to re-
duce the over-smoothing e®ect. It is based on an optimization function to ¯nd the
direction of isophotes that minimizes the spatial intensity change for each line
passing the damaged pixel as presented in Eq. (14).
p0¼argmin jþðpÞðqpÞj
jjqpjj þ
ffiffiffiffiffiffiffiffiffiffiffiffi
þðpÞ
p;pB"ðpÞ;
(ð14Þ
uðqÞ¼
2uðpmÞIðp0Þ;pm2 and jjujj <";
uðpmÞ;pm2 and jjujj  ";
uðp0Þ;pm62 :
8
<
:ð15Þ
The symbols are the same like previous. The ¯rst part of this function computes
the angle. The weighted reciprocal of the gradient magnitude appears in the second
part. The line that has maximum gradient magnitude around the damaged pixels
is selected to ¯ll the hole using Eq. (15), where pm:¼1=2ðqþp0Þ. This method
improves the quality and processing time, but still has some artifacts in the
smoothness of isophotes.
Second-order anisotropic di®usion smoothing method for multi-value image
inpainting is proposed by Tschumperle.
122
This method is based on comparable
tensor-driven PDE and image regularization. It combines the normal PDE di®usion
and line integral convolutions proposed in Ref. 21 to speed and robust the numerical
di®usion process. Equation (16) illustrates the iterative update of the damaged pixel
according to a ¯nite di®erence approximation.
@ui
@t¼traceðTHiÞþ2
ruT
iZ
¼0
Jffiffiffiffiffiffi
Ta
pffiffiffiffiffiffiffi
Ta
qd; 8i¼1;...;n;ð16Þ
where Tis the tensor ¯eld, Hipresents the Hessian of ui, and Jffiffiffiffiffiffi
Ta
pis the Jacobian
of the vector ¯eld !ffiffiffiffiffiffiffi
Ta
p. This method is faster than previous PDE methods, but
still takes minutes to restore small damaged region.
Auroux and Masmoudi
7
proposed an inpainting approach based on crack locali-
zation and topological asymptotic analysis. The inpainting problem is considered as
crack localization problem, which can be solved by Dirichlet and Neumann method.
The inverse conductivity problem is solved by using PDE approaches to detect
cracks. Crack detection based on topological gradient is used to identify the edges of
the damaged region as shown in Eq. (17).
gðx;nÞ ¼ ½ðruDðxÞnÞðrvDðxÞnÞþðruNðxÞnÞðrvNðxÞnÞ;ð17Þ
where gðx;nÞis the topological gradient, uDand uNare the Dirichlet and Neumann
conditions on the domain's boundary, vDand vNpresent the corresponding adjoin
solutions states, and nis the unit vector normal to the crack. A minimization of a
cost function based on topological asymptotic analysis is performed to provide the
best localization for the damaged edges to ¯ll in the missing area.
Image Completion
1554001-9
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Bornemann and Marz
18
proposed a noniterative image inpainting method based
on coherence transport equation. It extended some anisotropic di®usion steps which
are neglected in Ref. 14 to interleave the stabilization. Moreover, it tuned the
transport equation in Ref. 118 to remove the numerical iteration and keep the faster
processing. Structure tensor is used to estimate the coherence direction of the image.
Based on the strength of the coherence value, the method modi¯es the propagation
from directional transport or normal di®usion. This method works well around the
boundaries and structure part but fails in the texture area. It produces visual arti-
facts when the damaged region is large. Moreover, it depends on parameters tuning.
The second author
93
extends the method to solve the problem of boundary seriali-
zation by adapting three other distance functions. It adds a new parameter for the
user to select the appropriate distance function. The method produces more satis-
factory results than the previous one. It requires more interaction from the user.
There are numerous slight improvements for the previous explained methods. The
authors in Refs. 16,81,83,92,114 and 133 tried to reduce the computational
complexity of PDE and solved the connectivity problem. Tai et al.
114
extended the
method in Ref. 11 by using nonlinear TV-Stokes equation to obtain the smallest
TV-norm. The method tried to reconstruct large region by imposing zero divergence
condition on the repaired directions. The stability of the method depends on the
initial value. The authors in Refs. 81,83 and 133 extended the CDD method in
Ref. 27. Quick Curvature-Driven Di®usions (QCDD) is proposed by Xu et al.
133
to
reduce the computational complexity of the CDD method. QCDD method is based
on curvature and di®usion instead of gradient. Average of neighbor pixels is used to
di®use the information. This method fails to preserve edges. Moreover it produces
blurred results. Another improvement was done by Li and Wang.
81
P-Laplace op-
erator is used to repair the damaged region in two directions instead of one. Li and
Yu
83
proposed a nonlocal di®erential CDD that uses similar structure pixels instead
of neighborhood pixels. Lu et al.
92
extend the TV-based model in Ref. 26 by dividing
the missing region into layers. The inpainting is done layer by layer based on the
priority of the edge pixels. The priority is calculated based on the known neigh-
borhood pixels around the damaged pixel. Layers with higher priority are inpainted
¯rst. This method is faster than the TV method in Ref. 26 but is still slow and
produces blurred results. Biradar and Kohir
16
tried to address the limitation of PDE
method in Ref. 14 by introducing a simple method based on nonlinear median ¯lter
to di®use median value from exterior to interior instead of using Eq. (1). The method
works well only for thin damaged region like scratches or text removal. It produces
blur and artifact when the missing area is large.
Finally PDE- and variational-based methods are widely used for repairing thin
damaged region and smoothing the boundaries. There are common drawbacks in
most of these works including the computational complexity of the di®usion process,
tedious implementation, and fail to inpaint areas containing ¯ne texture details.
Moreover, they produce noticeable blurring artifacts and fail to reconnect the linear
S. Zarif, I. Faye & D. Rohaya
1554001-10
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

structures when the damaged area is large. In addition to that, some of them are
numerically unstable.
3.1.2. Convolution-based methods
In parallel to PDE- and variational-based methods, there are other di®usion
approaches that are based on convolution ¯lter operation.
Oliveira et al.
99
proposed the ¯rst convolution-based image inpainting method to
reduce the processing time. This method starts by clearing the color information of
the damaged region. Then, it propagates the known information by iteratively
convolving prede¯ned di®usion mask with the missing region. The proposed di®usion
masks look equivalent to weighted averaging from the neighborhood pixels but with
zero weight at the center as illustrated in Fig. 4.
Di®usion barrier is used to stop the iteration based on the contrast changes. The
algorithm is very fast and simple compared to PDE methods. It produces satisfactory
results for only small damaged regions that do not have high contrast edges.
Hadhoud et al.
58
proposed a noniterative method that improves and extends the
convolution-based approach in Ref. 99. This method modi¯es the di®usion masks
from zero weight at the center to zero weight at the bottom right corner. This
modi¯cation removes the need to iterate the convolution many times, which causes
more blur in the result. It uses the modi¯ed masks only once by convolving it with the
damaged region to propagate the information.
Figure 5illustrates the di®erence between the convolution from the center of the
mask and the bottom right corner of the mask. This method is faster than Ref. 99 but
still produces blurred results.
Noori et al.
96,97
proposed an adaptive convolution-based method. The method in
Ref. 96 improved the method in Ref. 99 by using adaptive mask instead of ¯xed mask
as shown in Fig. 4. The coe±cients of the convolving mask are computed from the
Fig. 4. The two di®usion masks used in the method.
Fig. 5. The positions of the di®usion masks, left presents the method in Ref. 99, right presents the method
in Ref. 58.
Image Completion
1554001-11
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

gradient of known neighboring pixels as illustrated in the following equation.
FðXÞ¼
1X


2
if jXj
2;
X
1

2
if 
2jXj;
0if jXj:
8
>
>
>
>
>
>
<
>
>
>
>
>
>
:
ð18Þ
In the above equation, FðXÞis the coe±cient function, Xis a pixel gradient, and is
used to control the propagation softness. This method is better than the method in
Ref. 99 but still produces blurred results due to the iterative convolution operation.
The authors in Ref. 97 enhanced their work by using bilateral ¯lter to preserve edges.
The convolution mask coe±cients are based on space and range domain neighbor-
hood. Local variance is used to adjust the number of convolution iterations. This
improvement is e±cient for removing noise. It needs large number of iterations to
restore edges.
Finally convolution-based techniques are considered as the most rapid methods
compared to PDE- and variational-based methods. They e®ectively work well for
removing noise. These methods su®er from the blurred results due to the convolution
iteration. Moreover, they fail when the damage region is large. In addition, the
quality of the results depends on the number of iterations.
3.1.3. Wavelet-based methods
Wavelet transform is widely used for dealing with signal and image due to its e±-
ciency to collect and maintain sharp features using few wavelet coe±cients after the
release of JPEG 2000. Combination of TV minimization and wavelet family is
commonly used in many earlier image restoration works.
33,45,78,79,125,126
Therefore,
there are many techniques that addressed the problem of damaged structure infor-
mation and lost wavelet coe±cients especially during error transmission, compres-
sion, and storage using wavelet-based inpainting.
Chan et al.
29
proposed a variational wavelet inpainting model for only repairing
lost wavelet coe±cients during the transmission process. Combination of the TV
minimization and wavelet transform is used to solve the geometrical and degradation
problem. This method ¯lls the damaged coe±cients in wavelet domain by using a TV
minimization model in order to maintain the geometrical image features and remove
noise. The authors presented two related models, one of them for free noise channel
while the other for noisy channel. Equations (19) and (20) illustrate free noise and
noisy channel models, respectively.
min
j;k:ðj;kÞ2Fðu;u0Þ¼ZR2jrxuð; xÞjdx ¼TVðuð; xÞÞ;ð19Þ
S. Zarif, I. Faye & D. Rohaya
1554001-12
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

min
j;k
Fðu;u0Þ¼ZR2jrxuð; xÞjdx þX
ðj;kÞ
j;kðj;kj;kÞ2:ð20Þ
In the above equations, uð; xÞrepresents the wavelet transform, is the dam-
aged region, j;kand j;kare the wavelet coe±cients where j;k¼j;k8ðj;kÞ 62 , and
j;kis zero inside the damaged region and constant otherwise. In Ref. 30, the same
authors improved the ¯tting term for the noisy channel model by using multi-level
parameters. This improvement is mainly used to e®ectively remove the noise. Both
methods maintain the geometrical information such as sharp edges but still consume
much time and are not suitable with ¯ne structures. Guo and Qiao
56
pioneered an
image inpainting method based on spatial and wavelet domain to ¯ll in damaged
coe±cients. This method reduces the computational time of Ref. 56 by dealing with
low and high frequencies separately. Wavelet transform is used to repair low fre-
quencies. Then, spatial domain is used for high frequencies. It is fast and simpler than
the method in Ref. 29 but it works well only on long and narrow damaged region.
Zhang and Chan
140
extended the local variational wavelet method in Ref. 33 to be
nonlocal TV regularization in order to restore the texture and structure information
around the damaged region. The authors in Refs. 31 and 129 proposed a new opti-
mization transfer method based on bivariate functional to e±ciently solve the
minimization problem in Ref. 29. It iteratively solves the wavelet inpainting as
denoising problem by using Chambolle's nonlinear projection method in Ref. 24.In
Ref. 129, the same authors enhanced their method in Ref. 31 by removing the inner
TV solver iterations to faster the convergence. In this method, a primal-dual iterative
method is applied to solve the inpainting problem. Both methods are faster and more
e±cient than the method in Ref. 29.
Dobrosotskaya and Bertozzi
43
presented a combination of variational PDE
methods with nonlocality of wavelet transform for blind deconvolution and
inpainting. In the wavelet part, this method modi¯es the energy function of Ginz-
burg–Landau to be nondi®erential function using wavelet-based semi-norm in order
to eliminate the blurry edges. It constructs \wavelet Laplacian" operator which
combines the scale proportional from Laplacian and Eigen functions from wavelet
transform as shown in Eq. (21).
!u:¼X
1
j¼1
22jhu;
j;ki jk:ð21Þ
In the above equation, jk is the orthonormal wavelet basis. In the variational
PDE part, it extends and simpli¯es the PDE method in Ref. 15 based on second-
order Allen Cahn equation. This method is computationally less than PDE methods,
mainly because it does not require PDE solver.
Yau et al.
136
proposed a wavelet inpainting method based on L0-Norm and TV
minimization. A sparse representation based on L0-Norm minimization in the
wavelet domain is used to optimize the wavelet coe±cients. Then, TV minimization
Image Completion
1554001-13
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

based on graph cut is performed to ¯ll the damaged information. The minimization
problem is divided into two sub-minimization problems by introducing one more
¯tting term to faster the processing as illustrated below.
min
^
ujjð^
u^
gÞjj0þ1
jj^
u^
fjj2
2;ð22Þ
min
fjjufjj2
2þTVðfÞ:ð23Þ
In the above equations, is a regularization parameter, TVðfÞpresents the TV of
image f,fis an approximation of u, and 1
jj^
u^
fjj2
2presents the ¯tting function. The
wavelet representation of uis represented by ^
u.is the wavelet coe±cient. A 3D
graph cut is used to solve the discrete form of the minimization problems. It produces
better results compared to the method in Ref. 29.
Zhang et al.
141
presented a fractional order variational inpainting method in space
and wavelet domains. This method combines the noisy and noiseless TV models in
Ref. 29 with fractional order p-Laplace to avoid its limitations. This combination is
used to inpaint the damaged image in both space and wavelet domains.
min F½u¼1
pZE[jrujpdxdy;2Rþ;p2½1;2;ð24Þ
min F½u¼1
pZE[jrujpdxdy þ
2ZEjuu0j2dxdy;2Rþ;p2½1;2;ð25Þ
min
j;k:ðj;kÞ2Fðu;u0Þ¼1
pZR2jr
xuð; xÞjpdx;2Rþ;p2½1;2;ð26Þ
min
j;k
Fðu;u0Þ¼1
pZR2jr
xuð; xÞjpdx þX
ðj;kÞ
j;kðj;kj;kÞ2;
2Rþ;p2½1;2:ð27Þ
Equations (24) and (25) present the combination for noiseless and noisy images in
the space domain, respectively. Equations (26) and (27) introduce both models in the
wavelet domain. All the equations parameters are de¯ned as in Eqs. (19)–(23).
Fractional order gradient and p-Laplace are used instead of normal TV term and
integral gradient for better restoration ability. The four models of this method
produce better results compared to the normal TV model in Ref. 29 but consume
more time in fractional order gradient.
Li et al.
80
proposed an adaptive optimization wavelet inpainting model based on
weighted nondecimated DCT-II. The objective function is a smoothed ‘1 norm of the
DCT-Haar multiresolution analysis coe±cients as illustrated in Eq. (28).
min
f;d
1
2jjWu djj2
2þjjdjj1:PDg

;ð28Þ
S. Zarif, I. Faye & D. Rohaya
1554001-14
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

where Wu is the DCT-II redundant system for image u,dpresents the auxiliary
vector, and is a non-negative diagonal matrix. For each iteration, the adaptive
Beck and Teboulle's algorithm
10
is used to solve the inpainting model in (28). This
method is suitable only for removing text and impulsive noise.
Liu et al.
89
introduced an integration and optimization of TV norm
29
and ‘2 norm
in wavelet domain to e®ectively solve lower di®usion speed and staircase e®ect in the
smoothed region. It combines the advantage of ‘2 norm for regularization in the
smoothed area with the high quality edges of TV norm.
min
j;k
Fðu;u0Þ¼ZR2
jruð; xÞj þ 1
2ð1Þjruð; xÞj2dx
þ1
2X
ðj;kÞ
j;kðj;kj;kÞ2:ð29Þ
The parameter is used to balance the TV and harmonic model where 2½0;1.It
achieves better results than the TV model in case of high wavelet coe±cients. The
other parameters are de¯ned as in Eqs. (19)to(23).
In parallel to wavelet-based inpainting, there are few techniques which address
the inpainting problem by using framelet instead of wavelet
22,23,28,40
and other
methods combine wavelet transform and Fourier transform to recover the damaged
image as in Ref. 32.
Finally wavelet-based inpainting methods are widely used to recover the lost
wavelet coe±cients due to transmission, compression, and storage. The methods are
very e®ective for noise removal. The common drawbacks of the wavelet-based
methods are the computational complexity, they cannot represent diagonal curves
well, and they are not shift invariant. In addition to that, they are not suitable with
¯ne details and large damaged region.
3.1.4. Discussion
All categories of structure-based methods are suitable and widely used in inpainting
small and nontexture regions. In terms of the size of the damaged regions and the
quality of results, PDE- and variational-based methods are better than the other
categories. From the processing time point of view, the convolution-based methods
are faster than the others. Wavelet-based methods are commonly used compared to
others in case of noise removal and recovering lost blocks during transmission and
compression. However, there are common drawbacks in all categories including
blurred results due to di®usion process, failing to recover ¯ne details and large
regions. In addition to that, PDE- and variational-based methods as well as wavelet-
based methods are very time consuming. Table 1summarizes most of the published
works on structured-based approaches and lists the strength and drawbacks of each
work. Rows 1–14 list the PDE- and variational-based methods. Convolution-based
methods are summarized in rows 15 to 17. The remaining part of Table 1represents
the wavelet-based methods.
Image Completion
1554001-15
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Table 1. Summary of structure-based approaches, its strengths, and drawbacks.
No. Reference Main Idea Optimization Strengths Drawbacks
1 Masnou and
Morel
95
It uses geodesic curve to connect the lines of
same color and orientation based on shortest
path.
No Easy to implement. Quite trivial, angles are not pre-
served, handle only very small
region.
2 Bertalmio et al.
14
It extended the method in Ref. 95 to maintain
the connection angles by using discretized
gradient and anisotropic di®usion.
No Works well for small region. It takes more time, produces blur-
red results, and numerically
unstable.
3 Bertalmio et al.
11
It improved the work in Ref. 14 by using
Navier–Stokes °uid dynamic equation.
No Stability and speed are improved. Requires user interactions, pro-
duces blurred results.
4 Chan and Shen
26
It uses TV based on Euler–Lagrange equation
and anisotropic di®usion to prorogate the
information.
No It works well in noisy image and
text removal.
Fails to connect broken edges and
time consuming.
5 Chan and Shen
27
It extended the method in Ref. 26 by CDD to
handle the geometric information.
No It connects some broken edges, and
support larger area.
Results look blurred and time con-
suming.
6 Esedogl and
Shen
49
It improved the method in Ref. 26 by using
Mumford–Shah model and -convergence.
No Approximation and computation
are decreased.
Produces arti¯cial corner, violates
connectivity, not stable.
7 Jiang and
Moloney
67
It based on modi¯ed orientation di®usion model
and intensity di®usion.
Yes It is not iterative method. It used only for error concealment.
8Guet al.
54
Monte Carlo di®usion method based on im-
proved Mumford–Shah model and snake
elastic.
No It connects the broken edges
around the boundary.
It fails to ¯t long narrow areas,
time consuming.
9 Telea
118
Marching method based on image gradient and
pixel neighborhood.
No Fast and simple. Produces blurred result when the
region is thicker than 10 pixels.
10 Wang et al.
127
It replaces the 2D smoothness anisotropic dif-
fusion with 1D smoothness to reduce the
over-smoothing e®ect.
Yes Quality and processing time are
improved
Some artifacts in the smoothness of
isophotes.
11 Tschumperle
122
Second-order di®usion method based on com-
parable tensor-driven PDE and image reg-
ularization.
No It improved the speed and robust of
the numerical di®usion process.
It takes minutes to restore small
damaged region. Results look
blur when the region is large.
12 Auroux and
Masmo
7
This approach based on crack localization and
topological asymptotic analysis.
Yes Preserves damaged edges. It produces blurred results and is
time consuming.
S. Zarif, I. Faye & D. Rohaya
1554001-16
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Table 1. (Continued )
No. Reference Main Idea Optimization Strengths Drawbacks
13 Bornemann and
Marz
18
It based on coherence transport equation and
improved anisotropic di®usion. Structure
tensor is used to estimate the coherence
direction.
No Noniterative, fast, produces good
results around boundary.
Produces artifacts when the region
is large or texture, it depends on
parameters tuning.
14 The authors in
Refs. 16,81,
83,92,114
and 133
Small extension to the previous methods in
Refs, 11,14,26 and 27 to address the
limitation of PDE.
No Little bit enhancement in compu-
tational complexity and con-
nectivity problem
They produce blurred and artifacts
when the missing area is large.
15 Oliveira et al.
99
It iteratively convolves prede¯ned di®usion
weighted averaging mask with the missing
region.
No Fast and simple. Used only for very small region
with low contrast edge, itera-
tive, produces blurred results.
16 Hadhoud et al.
58
It extended and improved the method in Ref. 99
by modifying the di®usion masks to zero
weight at the bottom right corner
No Faster than Ref. 99, simple and
noniterative.
Used only for scratches and text
removal, and produces blurred
results.
17 Noori et al.
96,97
It extended the method in Ref. 99 by using
adaptive convolution mask based on gradi-
ent of known neighborhoods. Bilateral ¯lter
is used to preserve edges.
No They Produce better results than
Refs. 58 and 99 and e±cient for
noise removal.
Iterative, produce blurred results,
and require large number of
iterations to restore edges.
18 Chan et al.
29,30
It combines TV minimization and wavelet
transform to ¯ll the lost wavelet coe±cients.
Yes Support noisy and noiseless images Consume much time and are not
suitable with ¯ne structures.
Only for lost blocks.
19 Guo and Qiao
56
It extended the method in Ref. 29 by using
spatial and wavelet domain to ¯ll in dam-
aged coe±cients.
Yes Faster and simpler than Ref. 29 It works well only on long and
narrow damaged region.
20 The papers in
Refs. 31
and 129
Optimization based on bivariate functional in
Ref. 31 to solve the minimization problem in
method in Ref. 29. Primal-dual iterative
method is applied in Ref. 129 to enhance
Ref. 31.
Yes Faster and e±cient convergence
than the method in Ref. 29.
Only for lost blocks during trans-
mission and compression.
21 Dobrosotskaya
and
Bertozzi
43
It combines PDE method with nonlocality of
wavelet and modi¯es the energy function to
be nondi®erential function.
No Computationally less than PDE
methods. Not requires PDE
solver.
For small corrupted data, support
binary and gray images only.
Image Completion
1554001-17
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Table 1. (Continued )
No. Reference Main Idea Optimization Strengths Drawbacks
22 Yau et al.
136
Sparse representation based on L0-norm and
TV minimization based on graph cut to ¯ll
the damaged information.
Yes It produces better results compared
with the method in Ref. 29.
It used only for noise removal and
damaged coe±cients.
23 Zhang et al.
141
It combines TV models in Ref. 29 with frac-
tional order p-Laplace in space and wavelet
domains to avoid its limitations.
Yes It produces better results than
normal TV model in Ref. 29.
It consumes more time in fractional
order gradient.
24 Li et al.
80
Adaptive optimization wavelet inpainting
model based on weighted nondecimated
DCT-II.
Yes It support larger damaged region
compared to wavelet category.
It is suitable only for removing text
and impulsive noise.
25 Liu et al.
89
Integration and optimization of TV norm
29
and
‘2norm in wavelet domain to e®ectively
solve lower di®usion speed.
Yes It achieves better results than
Ref. 29 in case of high wavelet
coe±cients.
Handles only the damaged coe±-
cients and time consuming.
26 The papers in
Refs. 22,28
and 40
They tried to address the inpainting problem by
using framelet-based instead of wavelet.
Yes Better convergence. Still for damaged blocks and noise
removal, and time consuming.
S. Zarif, I. Faye & D. Rohaya
1554001-18
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

3.2. Texture-based methods
Texture is an essential part. It is obvious that structure-based methods fail to ¯ll in
the damaged texture. In parallel to structure-based methods, texture synthesis is
used to inpaint texture images. It is used to recover damaged texture by using
surrounding texture information or from input sample. We brie°y discuss some of the
common methods that are based on input sample to produce more texture from the
input sample, which are used later in the hybrid-based methods. For more details
about texture synthesis techniques that are based on input sample, we recommend
the interested reader to read the survey.
128
In this review, we concentrate on texture
synthesis methods that repair the damaged texture-based on image neighborhood.
Texture-based methods can be classi¯ed into statistical-, pixel-, and patch-based
approaches as shown in Fig. 2. In the next subsection we describe each category in
detail.
3.2.1. Statistical-based methods
Statistical-based texture synthesis or inpainting approaches extract texture features
from the surrounding known texture or from an input sample texture using statistical
models. The extracted features are used to construct the damaged texture or gen-
erate new texture that have the same visual appearance as the available texture.
Heeger and Bergen
61
presented a texture synthesis method that match the
characteristics texture of a given input sample based on ¯rst-order statistics. In this
method, Laplacian and steerable pyramids transform are used to decompose images
into information at multiple scales and orientations in order to extract the features of
interest. The textures are synthesized by matching histograms of input sample with
the noisy image. The noisy image is modi¯ed to look like the input texture based on
histogram matching and image pyramids. The operations involved in the method are
histogram matching, pyramid generation, and texture matching. This method works
well only for homogeneous input texture. It fails when the input sample is quasi-
periodic or random mosaic texture.
Bonet
17
proposed a texture synthesis method from input texture sample based on
multiresolution image pyramid and histogram of ¯lter response. Probability density
estimator based on joint occurrence of features is used to analyze the input texture.
Sampling of spatial frequency band is used to synthesize the new texture from the
input texture based on similarity of probability density. The method is unable to
handle texture images with complex structures.
Portilla and Simoncelli
104,110
proposed a parametric texture synthesis model
based on joint statistics of wavelet coe±cients in Ref. 110 and extended it in Ref. 104.
A Markov statistical descriptor based on multi-scale wavelet coe±cients is used to
match between input and synthesizing textures. It provides impressive synthesis
results for stochastic and repeated input texture. The synthesis of high frequency
component and the complexity are the main drawbacks of these methods.
Image Completion
1554001-19
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Levin et al.
77
presented a textural inpainting method based on global image
neighborhood statistics. Exponential family distribution based on histogram of local
features is used to model the image statistics where the probability of an image is
given by Eq. (30).
Pðu;figÞ ¼ 1
ZeP
iP
x;y
iðfiðx;yÞÞ:ð30Þ
The value of feature iat location ðx;yÞin image uis given by fiðx;yÞ. The
normalization factor is presented by Z.iðfÞis the exponential family distribution.
The probable features from the exponential family distribution are used to ¯ll in the
damaged region. This method uses gradient magnitude and pairwise gradient angle
features only for all images and approximation of maximum likelihood to simplify the
exponential family distribution. Loopy belief propagation is used to optimize the
probability to ¯nd the most probable ¯ll in. The method produces poor results for
high detailed texture due to lack of features. Moreover, the inpainted region is not
sharp due to the optimization scheme.
3.2.2. Pixel-based methods
Pixel-based textural inpainting techniques can be addressed by a nonparametric
sampling based on neighborhood search. These techniques take a given input texture
sample and produce a new texture by synthesizing pixel by pixel instead of applying
¯lters. The value of the output synthesis pixel is determined by comparing the
neighbor of a synthesis pixel with all neighbors of the texture sample. The pixel that
has the best match is assigned to the output pixel. The performance of these algo-
rithms is based on the neighborhood size. The neighbor widow size should be large
enough to model the local texture information and ¯nd the best match.
Efros and Leung
47
pioneered a nonparametric approach for producing texture
from an initial seed instead of applying ¯lters to the sample texture. The model
copied the pixels from the sample image itself in synthesis. Then, the texture is
modeled as a Markov random ¯eld (MRF). This model produces the new synthesis
pixels from a square window around that pixel by indicating the probability distri-
bution of brightness values for a pixel given the brightness values of its spatial
neighborhood. This method has few problems such as slipping in the wrong part of
search space and growing garbage.
Wey and Levoy
131
extended the work of Efros and Leung in Ref. 47 by using
multi-resolution neighborhood search to reduce the computational time. This algo-
rithm is based on a tree structured vector quantization. The texture is synthesized in
a coarse to ¯ne manner by using search for the best neighborhood match and multi-
level resolution pyramid. The smooth L2 norm is used to measure the similarity
between neighborhoods. This method reduces the quality of the result but decreases
the computational time.
Ashikhmin
5
extended the work of Wey and Levoy
131
to address its drawbacks in
handling natural textures and blurred results. The algorithm takes into account the
S. Zarif, I. Faye & D. Rohaya
1554001-20
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

positions of the assigned pixels in its search for the current pixel instead of starting a
new search at each pixel. The input sample pixels assumed to be appropriately
forward shifted with respect to the synthesis pixels are well suited to ¯ll in the
current pixel. The suitable candidate pixel is chosen to ¯ll the current pixel only if it
is completely inside the image. A user control to guide the ¯lling process is added.
The user can specify certain high level texture that will be used in the search algo-
rithm. The user provides an image that shows how the resulting texture should look.
This method is faster than Ref. 131 but fails to extract structure information.
Hertzmann et al.
62
presented a combination of smooth L2 norm from Wey and
Levoy
131
and the search technique of Ashikhmin
5
to synthesis the new texture. An
approximation of nearest neighbor and coherence search in Ref. 5is used to ¯nd the
best texture. This method produces higher results without edge discontinuities
compared to the method in Ref. 5, but it is slower. Undesirable artifacts results are
produced in some cases due to bad similarity metrics.
Tang
115
proposed a nonparametric technique based on a MRF model for handling
homogeneous texture. The nonparametric sampling procedure for ¯lling the image
utilizes a series of binary mask Gaussian pyramid level texture to synthesize the
texture in a coarse to ¯ne order. The pixel from the initial synthesis guess level is
randomly and uniquely chosen in the sampling process. This algorithm has good
results for homogenous texture. The algorithm gives a poor result in a texture that
includes di®erent texture regions and the hole spans di®erent regions. In addition to
that, it is very time consuming.
Lefebvre and Hoppe
75,76
presented parallel controllable and appearance space
texture synthesis methods respectively. The parallel controllable method in Ref. 75
combines Gaussian stack, coordinate upsampling, and sub-pass correction together
to analyze texture structure and coherence. Multiresolution jittering is used to
achieve texture variation. This method support parallel evaluation using several
optimizations. In addition to that, it is more °exible and intuitive control. The
appearance space method in Ref. 76 extended their method in Ref. 75 to address the
neighborhood-based per pixel problem that produces poor results in semantic
structures. It creates appearance vector for each pixel including neighborhood in-
formation, feature information, and radiance transfer instead of only color. This
method replaces the point wise colors comparison with appearance vectors for tex-
ture matching. Principal component analysis (PCA) is used to reduce the dimension
of the appearance vectors before synthesis. The texture synthesis stage is based on the
generated appearance space image instead of the input sample. It achieves better
results than Ref. 75 due to the fact that appearance vectors are richer than pixel colors.
3.2.3. Patch-based methods
The patch-based methods addressed the complexity of pixel-based method by syn-
thesizing patch by patch as opposed to pixel by pixel. Synthesizing patch by patch is
preserving the global structure of the input texture.
Image Completion
1554001-21
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Efros and Freeman
46
proposed a simple and e®ective patch-based texture syn-
thesis algorithm to treat the output image as a grid of block. The algorithm syn-
thesizes the new local and global structures texture by stitching blocks of existing
sample texture together in a consistent way. The patches are selected from the input
texture to the output image in raster scan order with 4 to 8 pixels patches overlap.
Minimum error boundary cut is used to reduce the artifacts between the overlapped
regions. This method produces good and stable results for both structured and
stochastic textures in reasonable time compared to the pixel-based methods. It
cannot address the constrained texture synthesis or hole ¯lling problem.
Liang et al.
85
introduced nonparametric patch-based algorithm that samples the
texture patches based on local conditional MRF density to avoid patch boundary
mismatching. A combination of quad tree pyramid, approximate nearest neighbors
search, and PCA is used to improve and accelerate the patch search. It provides
implicit constraints to avoid the synthesis garbage in Ref. 47. This method is real
time and handles constrained texture synthesis. Moreover, it handles a wide variety
of textures. It fails if the texture image is not frontal parallel. Also, it cannot handle
high frequency features around the boundaries. Tonietto et al.
120
extended the work
in Ref. 85 by using wavelet coe±cients similarity metric instead of color to ¯nd the
best patches. Wavelet coe±cient metric improves the structure of the texture but
requires more computational time.
Kwatra et al.
71,72
presented a texture synthesis method for image and video based
on graph cut.
72
The main idea of this method is to calculate and copy only the
optimal part from the selected patch into the output instead of the whole patch by
using graph cut. In Ref. 71, a controllability optimization-based texture synthesis
using MRF similarity metrics is presented by the same authors. The Expectation
Maximization method is used to formulate the energy function. Joint optimization is
used to re¯ne the entire texture instead of region growing to model large neighbor
interactions. This method produces blurring and misalignment output due to local
minima stuck.
Tang et al.
116
proposed a texture synthesis method for completing the region after
object removal based on coherence local search. This method is based on the sur-
rounding texture to ¯ll in the damaged area instead of input sample. It adopts the
patch search to be limited on local neighborhood. Coherence con¯dence is used to
select the best patch matching to reduce the accumulation error. Finally, graph cut
optimization is used to merge and integrate the selected patch into the surrounding
texture. This method does not preserve the structure information.
3.2.4. Discussion
Texture-based approaches were introduced to be suitable for producing large texture
image from input sample textures. Texture-based methods could be used to restore
digitized images especially if the damaged area needs to be ¯lled with some texture.
However they usually fail if the area to be reconstructed contains an additional color
S. Zarif, I. Faye & D. Rohaya
1554001-22
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

or intensity gradient. Most of statistical-based techniques work well in reproducing
stochastic or irregular textures, but they usually fail to synthesis structured or
regular textures. The synthesized textures from the pixel-based methods are better
than statistical-based method especially in regular texture even so they usually fail to
synthesis large structured texture. Moreover, they su®er from narrow neighborhood
and lack of global optimization. In addition to that, they take more time compared to
statistical-based methods due to the search process. Finally, patch-based texture
synthesis methods produce better quality than pixel-based techniques in terms of
global structure texture. They are usually faster and cover a wide variety of textures.
However, they produce visual artifacts around object boundaries. Table 2sum-
marizes the published work on texture-based approaches and lists the strength and
drawbacks of each work. Rows 1–4 list the statistical-based methods. Pixel-based
methods are summarized in rows 5 to 11. The remaining part of Table 2represents
the patch-based methods.
3.3. Hybrid-based methods
Most images are neither pure texture nor pure structure. Normally, images usually
contain both structure and texture information. The structure represents edges,
corners, etc. The texture represents the image regions with feature patterns. Pure
structure-based or texture-based techniques are not able to reconstruct the damaged
area that includes composite textures and structures. Hybrid-based methods com-
bine both texture- and structure-based techniques to reconstruct images with large
unknown areas, using both texture and structure information. Hybrid-based tech-
niques can be classi¯ed into decomposition- and exemplar-based methods as shown
in Fig. 2. In the next subsection we describe each category in detail.
3.3.1. Decomposition-based methods
Decomposition-based methods are considered the ¯rst e®orts to restore the damaged
texture and structure information. The main idea of the decomposition-based
methods is separating the input image into structure and texture layers, then
inpainting each layer separately by using structure- and texture-based methods. The
¯nal results can be then obtained by combining both reconstructed layers. The main
objective of the decomposition-based methods is to combine the advantages and
overcome the weakness of structure- and texture-based methods.
Bertalmio et al.
13
presented a combination of structure- and texture-based
techniques to achieve high quality results. A decomposition technique is used to
divide the input image into its structure and texture components. The PDE-based
method in Ref. 14 is used to repair the structure component. Then, the texture-based
method in Ref. 47 is used to restore the textural component. Finally, the recon-
structed structure and texture components are combined to obtain the ¯nal restored
result. This method overcomes the smooth e®ect of the methods based on PDE but
still have limitations for reconstruction of large areas.
Image Completion
1554001-23
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Table 2. Summary of texture-based approaches, its strengths and drawbacks.
No. Reference Main Idea Optimization Strengths Drawbacks
1 Heeger and
Bergen
61
Laplacian and steerable pyramids are used to
extract the texture features. Histogram
matching is used to synthesize texture.
No Works well in homogeneous
input texture.
It fails when the input sample is quasi-periodic
or random mosaic texture.
2 Bonet
17
It based on multiresolution image pyramid
and histogram of ¯lter response.
No Works well in stochastic or
irregular textures.
It cannot handle texture images with complex
structures.
3 Portilla and
Simon-
celli
104,110
Parametric texture synthesis model based on
joint statistics of wavelet coe±cients in
Ref. 110 and extended in Ref. 104.
No Impressive results for sto-
chastic and repeated
texture.
It fails in high frequency component, and high
computational complexity.
4 Levin et al.
77
Exponential family distribution based on
histogram is used to model the global
image neighborhood statistics.
Yes No need for input sample. It produces poor results for high detailed
texture, and the inpainted region is not
sharp.
5 Efros and
Leung
47
Nonparametric approach based on MRF to
synthesize texture by using probability
distribution.
No The quality of the results
are better in a wide va-
riety of textures.
Consumes more time, slipping in the wrong
part of search space, and growing garbage.
6 Wey and
Levoy
131
It extended the method in Ref. 47 by using
multiresolution neighborhood search and
tree structured vector quantization.
No It is faster than the method
in Ref. 47.
The quality of the result is decreased com-
pared to Ref. 47. It cannot handle natural
textures.
7 Ashikhmin
5
It extended the method in Ref. 131 by taking
into account the assigned pixels positions
in its search criteria.
No This method is faster and
better quality than
Ref. 131.
It fails to extract structure information.
Requires user interaction.
8 Hertzmann
et al.
62
It combines smooth L2 norm from Ref. 131
and the search technique of Ref. 5to
synthesis the new texture.
No It produces higher results
without edge dis-
continuities than meth-
od in Ref. 5.
It is slower than Ref. 5, undesirable artifacts
results are produced in some cases.
9 Tang
115
Gaussian pyramid level texture is used to
synthesize the texture in a coarse to ¯ne
order.
No It has good results for ho-
mogenous texture.
It fails in a texture that spans di®erent tex-
ture regions. It is very time consuming.
10 Lefebvre and
Hoppe
75
It combines Gaussian stack, coordinate
upsampling, and sub-pass correction to-
gether to analyze texture structure and
coherence.
Yes It support parallel evalua-
tion, it is more °exible
and intuitive control.
It produces poorly results in semantic struc-
tures.
S. Zarif, I. Faye & D. Rohaya
1554001-24
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Table 2. (Continued )
No. Reference Main Idea Optimization Strengths Drawbacks
11 Lefebvre and
Hoppe
76
It extended the method in Ref. 75 to address
the neighborhood based per pixel problem
by using appearance vector.
Yes It achieves better results
than Ref. 75.
It fails to synthesis large structured texture.
12 Efros and
Free-
man
46
It treats the output image as a grid of block
and synthesizes the new local and global
structures texture by stitching blocks.
Yes It produces good and stable
results for both struc-
tured and stochastic
textures.
It cannot address the hole-¯lling problem.
13 Liang et al.
85
It combines quadtree pyramid, approximate
nearest neighbors search, and PCA to
improve and accelerate the patch search.
Yes Real time and handles a
wide variety of textures.
It fails if the texture image is not frontal
parallel.
14 Tonietto
et al.
120
It extended the work in Ref. 85 by using
wavelet coe±cients similarity metric in-
stead of color to ¯nd the best patches.
Yes It improves the structure of
the texture.
It requires more computational time.
15 Kwatra
et al.
72
It calculates and copies optimal part from the
selected patch into the output instead of
the whole patch by using graph cut.
Yes It can synthesize regular,
random, and natural
images and videos.
It fails in case of dynamic texture.
16 Kwatra
et al.
71
Controllability optimization-based texture
synthesis using MRF similarity metrics to
extend Ref. 72.
Yes It handles larger class of
structured and stochas-
tic textures.
It produces blurring and misalignment output
due to local minima stuck.
17 Tang et al.
116
It based on coherence local search to complete
the damaged region. It adopts the patch
search to be limited on local neighbor-
hood.
Yes It reduces the accumulation
error. No need for input
sample.
It does not preserve the structure information.
Image Completion
1554001-25
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Yamauchi et al.
135
introduced a structure and texture combination method based
on frequency decomposition. The input image is decomposed into low and high
frequencies components by using DCT. The convolution-based method in Ref. 99 is
applied to inpaint the low frequency part. Then, Gaussian pyramid is used to divide
the high frequency component into multiresolution levels. The multiresolution tex-
ture-based method in Ref. 131 is used to synthesize the texture starting from highest
level. The ¯nal result comes from the summation of inpainted low frequency com-
ponent and synthesized high frequency component.
Grossauer
52
proposed a hybrid method similar to Ref. 13. Nonlinear di®usion
¯lter in Ref. 103 is used to decompose the image into a geometrical part and a texture
part. Ginzburg–Landau PDE-based method in Ref. 53 is used to reconstruct the
geometrical part. Then, coherence enhancement ¯lter is applied on the geometrical
inpainted part to smooth the kinks and edge corners. After that, a gradient con-
trolled-based region growing method is used to segment the inpainted geometrical
part into multiple connected segments. The texture-based method in Ref. 131 is
employed to repair each segment from its local available texture. Finally, the syn-
thesized segments are combined to the geometrical part in order to construct the
¯nal output. The main drawback of this method is the dependence of the quality of
the results on the parameters. Moreover, it produces unsatisfactory results when the
object to be removed covers multiple segments with di®erent textures.
Shao et al.
108
suggested a decomposition-based technique that uses a total vari-
ational de-noising method to separate the damaged image into its structure and
texture images. The exemplar-based method in Ref. 36 is directly incorporated for
the texture image. In structure image, a Laplacian operator is ¯rstly employed to
enhance the structure information. Then a combination of exemplar-based and
Poisson equation is applied to reconstruct the structure part. The reconstructed
components are added back to get the ¯nal output. The Poisson step improves the
boundaries of the copied blocks to be less noticeable.
Jia and Tang
66
pioneered similar idea of decomposition that are based on texture
segmentation. The input image is texture segmented into di®erent regions by
adopting the texture-based segmentation method in Ref. 42. Adaptive scaling
N-dimensional tensor voting based on MRF is applied to recover the texture of each
region from its N-size neighborhood. Tensor voting and B-splines are used to connect
the region curves. This method performs well in most cases, but fails in case of
complex background and irregular shapes. Moreover, the texture segmentation is not
accurate.
Elad et al.
48
extended their sparse representation decomposition method in
Ref. 111 based on morphological component analysis to adopt linear layers separa-
tion. This method decomposes the input image into texture and structure layers by
using two mutually incoherent dictionaries. A combination of the two dictionaries is
applied to obtain the sparse representation. Minimization of sparse vectors of the two
layers is used instead of inpainting each component separately. Pursuit basis fusion
S. Zarif, I. Faye & D. Rohaya
1554001-26
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

based on TV regularization is used to complete the missing layer with composite of
structure and texture information. The quality of the results looks blurring when the
damaged area is large.
Bugeau and Bertalmio
19
presented an improvement of the second author's pre-
vious work in Ref. 13. In this paper, the original image is inpainted ¯rst before the
decomposition stage by using the structure-based method in Ref. 122 in order to
avoid the decomposition problems in the structure part. Then, the decomposition
method in Ref. 123 is applied to separate the structure and texture components. The
structure part is used to improve the exemplar-based method in Ref. 36. The im-
proved version of Ref. 36 is then employed to repair the texture part. Finally, the
original structure part and the repaired texture part are combined to obtain the
result. The quality of the results is improved compared to the previous work in
Ref. 13 but it is still parameters dependent.
Hesabi et al.
63
presented the closest method to Ref. 13 that combines two powerful
approaches to recover texture and structure information. The input image is
decomposed into structure and texture components by using the decomposition
method in Ref. 6. The reconstruction of each part is achieved separately. The image
inpainting method described in Ref. 107 is applied to repair the missing structural
information in the structure component, while the damaged information in the
textural component is reconstructed using the texture synthesis method in Ref. 36.
The ¯nal result is obtained by combining the two reconstructed components. The
restored images look plausible in general.
Chen and Xu
34
introduced structure and heuristic texture completion method
based on primal sketch decomposition model. Sparse coding and MRF models are
used to represent the structure and texture respectively in the primal sketch model in
Ref. 57. 3-degree-junctions based on Elastica cost function is applied to complete the
missing structure. Con¯dence map for ¯lling order and texture synthesis based on
graph cut method in Ref. 72 are used to ¯ll in the texture component. In Ref. 35, the
authors presented a two-stage completion method. Euler spiral is used to detect the
salient structure and complete the damaged curves in the ¯rst stage. Textures in-
formation around the repaired structure is synthesized based on belief propagation
method. The texture stage is based on multiscale patch guided by reconstructed
structure stage to inpaint the remaining texture. A global optimization based on
combination of simulated annealing and Monte Carlo method is used to optimize the
solution. Both methods in Refs. 34 and 35 achieve satisfactory results only when the
image salient structures are not complex.
3.3.2. Exemplar-based methods
In parallel to the decomposition-based methods, exemplar-based methods are used to
¯nd the optimal and similar patch from the surrounding known information to repair
the damaged region. These methods use other strategies to complete the natural
images that have structure and texture missing pixels. A combination of di®erent
Image Completion
1554001-27
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

terms from di®erent structure and texture methods in one objective function is
commonly used instead of inpainting each layer separately. These methods complete
the damaged region patch by patch. The quality of the results is based on many
factors such as patch size, patch matching, similarity metric, patch ¯lling priority,
optimization, and coherence.
Drori et al.
44
proposed a fragment-based algorithm for image completion that
could preserve both structure and texture. The method iteratively ¯lls the missing
region from the remaining image using the principle of self-similarity. A con¯dence
map is used to determine which pixels have more surrounding information. The
completion starts from the most con¯dent pixels, and proceeds in a multi scale
fashion from coarse to ¯ne. In each step, a similar image fragment is found and copied
to the current unknown location, until the image is completed. Most of the results
seem good in quality. However, it fails to handle the object located on the boundary
and ambiguities between intersected objects. Moreover, this method is very slow due
to the extensive search in the whole known region.
Criminisi et al.
36
presented an exemplar-based image completion method to ¯ll in
the damaged or target region from the surrounding known or source region. Let be
the target region in image Iand its region boundary. The source region is indi-
cated by ¼I. Square patch pis centered on each boundary pixel pas
illustrated in Fig. 6.
The patch with the higher priority is selected to compete it ¯rst. The selected
patch is completed by the most similar patch from the source region . The entire
similar patch is copied to the patch pinstead of single pixel to faster the processing.
The patch priority is based on con¯dence term CðpÞand data term DðpÞas in
Eqs. (31)–(33) where jpjis the patch size, is the normalization factor, rI?
pis the
direction of isophotes at pixel p, and np is the normal at pixel p.
PðpÞ¼CðpÞDðpÞ;ð31Þ
CðpÞ¼Pq"p\CðqÞ
jpj;ð32Þ
DðpÞ¼jrI?
pnpj
:ð33Þ
The con¯dence term CðpÞis considered as the number of reliable or known data
surrounding the pixel p. The data term DðpÞis considered as a function of the
Fig. 6. Exemplar-based structure propagation method (from Ref. 36).
S. Zarif, I. Faye & D. Rohaya
1554001-28
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

strength of isophotes hitting the boundary, which is similar to PDE method in
Ref. 14. This method iteratively calculates the priorities of all patches that located on
boundary pixels. Then, select the patch that has higher priority. After that, sum of
squared di®erences is used to search for the best match in the source region and copy
its content to the damaged patch. Finally, update the con¯dence term and repeat
again until completing the damaged region. One of the strong impacts of this method
is the patch priority. Inpainting based on higher priority produces high quality
results around object boundaries as well as large damaged region. It is considered
as one of the state-of-the-art image completion techniques. On the other side, it
still has some limitations including, the method ignores the local similarity in the
natural image and searches in the whole image to ¯nd the best patch, which is time
consuming.
Due to the e±ciency of the method in Ref. 36, numerous extensions have been
proposed. We brie°y discuss some of them. The authors in Refs. 37,38,51,64,73,74,
91,106,119,124,132,134,142 and 143 suggested a modi¯cation of the search space,
¯lling priority or patch matching in order to reduce the time and increase the e±-
ciency. Wu and Ruan
132
modi¯ed the data term based on the cross isophotes dif-
fusion method in Ref. 27 to preserve linear structure. A hybrid similarity distance is
used by Liu et al.
91
instead of sum of squared di®erences to increase the accuracy of
the patch matching. An adaptive scheme to determine the size of template window
and local search strategy is introduced in Ref. 64 to faster the patch search. An
optimization approach for minimizing the objective function is applied in Ref. 143 to
formulate the problem of patch size determination. In Ref. 142, the authors used an
optimization problem to maximize the local consistency to select the optimal patch.
Kwok et al.
73
presented a fast query method based on DCT. It selects the most
signi¯cant coe±cients from the frequency coe±cients after decomposing the exem-
plars. The estimation of the best matching is based on search array data structure. A
local gradient method is then used to complete the missing area. Xu and Sun
134
modeled the patch representation and priority based on patch sparsity. The con¯-
dence of the patch structure is measured by using sparseness of nonzero similarity to
neighbor patches. Higher priority is assigned to the patch of larger sparsity. After
that, a linear sparse combination of exemplars is used to repair the selected patch.
Sharp edge results are produced due to the sparse combination. The method is
unable to restore the damaged hidden structure without any cue. Florinabel et al.
51
combined spatial and frequency domains to recover the linear structures. The pri-
ority of patch completion is based on a combination of DCT coe±cients term and
edge term. Structure-based similarity and pixels interpolations are considered for
patch matching. This method improves the structure inpainting but it requires extra
time for interpolation. Thangamani et al.
119
proposed a hybrid inpainting method
that combines and extends the exemplar-based methods in Ref. 36 and edge-based
restoration in Ref. 106 for handling indoor virtual 3D models. It simpli¯ed the ex-
tensive patch search in Ref. 36 by using hash tables. Super pixel constraints and
Image Completion
1554001-29
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

spline curves are used to overcome the limitation of edge-based inpainting in Ref. 106
for handling any type of curves. This method fails to repair the images with trans-
parent surfaces. Le Meur et al.
74
suggested a hierarchical super resolution. A coarse
version of the input image is used to generate multiple inpainted images with dif-
ferent parameter settings. A combination of the inpainted images is used to produce
the ¯nal completed image. It uses a super resolution method to recover the full
resolution of the ¯nal result. This method takes more time due to the super resolution
process. Wang et al.
124
presented a robust and improved version of the exemplar-
based method by modifying the priority and matching terms. It adjusts the ¯lling
priority by introducing regularized factor to control the con¯dence smoothness.
Combination of normalized cross correlation and modi¯ed version of sum of squared
di®erences is used to ¯nd the best patch. The quality of the results is more robust
compared to Ref. 36 but the processing time is increased due to the second round of
patch matching. Dang et al.
37,38
modi¯ed the exemplar-based method by using a
window-based priority in Ref. 38. In Ref. 37, the authors presented a hierarchical
optimization approach based on pyramidal representation. This method combines
the idea of exemplar-based and global optimization-based methods in hierarchical
scheme. The modi¯ed exemplar method in Ref. 38 is applied to restore the damaged
region on a coarse resolution that are obtained from Gaussian pyramid. The spatial
information of the inpainted coarse version (o®set map) is used in the second step to
construct the hole on its original resolution. A multi label graph cut is used to model
and optimize the inpainting problem on the higher resolution. Both methods are
faster than Ref. 36 and slightly better in term of linear structure.
Sun et al.
112
proposed a global optimization structure propagation approach based
on dynamic programing and belief propagation to solve the salient structure. In this
method the most signi¯cant damaged structure is manually selected by connecting
the curves and lines from outside to inside the damaged region. The selected salient
structure curves are completed ¯rst by using a structure propagation based on the
known patches around it. The remaining sub-regions are ¯lled patch by patch using
the patch-based texture synthesis methods in Refs. 5and 62. The order of ¯lling is
based on con¯dence map as in Refs. 36 and 44. The quality of the results is similar to
the method in Ref. 44. It fails to handle depth ambiguity and complex missing curves.
The quality of the inpainting process is user dependent and it takes more time when
restoring larger image containing many curves. Li and Zhao
84
presented similar
method to Ref. 112 but suggested to avoid the manual selection of salient structure by
introducing an automatic detection of damaged salient structure based on wavelet
transform. A curve ¯tting and extension are then used to complete the detected
structure. The method of Ref. 36 with modi¯ed ¯lling priority is applied to propagate
the texture to the remaining damaged texture. It produces results similar to Ref. 112
but without user interaction. The other limitations are the same as per Ref. 112.
Komodakis and Tziritas
69
formulated a new exemplar-based method as a discrete
global optimization problem with a well-de¯ned objective function. The optimal
S. Zarif, I. Faye & D. Rohaya
1554001-30
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

solution is found using By Priority-BP algorithm which carries two major
improvements over the standard belief propagation. The method starts with \label
pruning" which accelerates the process by allowing less number of source locations
that can be copied to more con¯dent points. It then uses a \priority-based message
scheduling" which also speeds up by giving high priority to belief propagation from
more con¯dent points. This method gives better results compared to the previous
exemplar-based methods. Nevertheless, these are no evidences that the inpainting
process leads to a global minima and need user guidance. Moreover it takes slightly
more time than Ref. 36.
Wexler et al.
130
proposed a nonparametric space time video completion for sta-
tionary camera as a global optimization problem based on the ideas of Refs. 5and 47.
This method ¯nds the optimal patch from the source region based on global cost
function (coherent measure). It takes into account the spatial and temporal
dimensions. It solves the inpainting problem by sampling a set of spatial-temporal
patches from other frames to ¯ll in the missing data. It optimizes the patch search
process at di®erent resolution levels using spatio-temporal pyramids and nearest
neighbor algorithms. The author produces an image completion version from this
method. The executable ¯le from this version is available on the author website.
Exhaustive searching strategy for ¯nding appropriate patches leads to high
computational load. Moreover, it is sensitive to the initialization parameters. Kopf
et al.
70
extended the nonparametric work of Ref. 130 to formulate a data-driven
image completion method for quality prediction. It produces more satisfactory
results and improves the synthesized texture by using automatic derived search
space constraints. This work is more useful and related to stitched panorama. It lacks
object level understanding. Moreover, the ¯tting prediction function is not perfect
due to mismatching.
Fang and Lien
50
presented a multiresolution exemplar-based directional and
nondirectional method to speed up the convergence process. In training stage, a
patch-based Eigen space is created based on multiresolution by using a down-
sampling method. In the completion stage, a Hessian matrix decision value is used to
compute the ¯ll in priority based on the gradient direction of the Hessian. The
directional and nondirectional completion based on upsampling method is used to
repair and improve the geometrical and local details of the damaged region. A tex-
ture re¯nement is then applied to maintain and optimize the consistency of the
completed region. The quality of results and the processing time are better compared
to Ref. 36. However, it still consumes more time in the training stage.
Pritch et al.
105
introduced a general image editing approach based on shift-map
and optimal graph labeling for reshu²ing, retargeting, and inpainting. The optimal
shift-map can be calculated based on data and smoothness terms. Data term is used
to handle the object size, saliency map, and rearrangement constraints. Minimization
of the output image discontinuities is handled by the smoothness term. Hierarchical
graph cut is performed to solve the graph labeling. This method produces impressive
Image Completion
1554001-31
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

results for the mentioned applications but fails to preserve the structure variations in
inpainting.
Barnes et al.
8,9
pioneered a fast PatchMatch and generalized PatchMatch tech-
niques respectively that accelerate the family of patch-based methods which are
computationally expensive. Randomized cooperative climbing scheme is used in the
PatchMatch method to ¯nd a single nearest neighbor image patch at the same
rotation and scale. This method has been extended to a generalized PatchMatch, to
¯nd K-nearest neighbor instead of one and the search process cover di®erent scales,
rotations, and translation. Both methods are an order of magnitude faster than
previous patch-based family but are still not real time.
Adobe Photoshop is one of the most famous and common image processing
commercialized applications. Content aware ¯ll
1
is a new added feature in adobe
Photoshop CS5 version. This feature is used to automatically ¯ll in the missing area
from the surrounding known information. Content aware ¯ll combines the advan-
tages of the space time method in Ref. 130 and generalized Patch Match method in
Ref. 8to produce faster and high quality results. The faster patch search and match
in Ref. 8adds more contribution and solves the limitations of the space time method
in Ref. 130. Therefore, content aware ¯ll is considered the current image completion
state-of-the-art approach in terms of processing time and quality of results.
Most of the previous hybrid methods discussed before are based on combination of
texture and structure energy terms. Bugeau et al.
20
introduced a variational
framework that combines structure, texture, and coherence energy terms of existing
approaches into one energy function as illustrated in Eq. (34).
"ð’Þ:¼1"1ð’Þþ2"2ð’Þþ3"3ð’Þ;ð34Þ
where 1;
2;
30 represent the weights, ’is the corresponding map, and
"1ð’Þ;"
2ð’Þ, and "3ð’Þare the texture, di®usion, and coherence energy terms, re-
spectively. The texture term is inspired from Ref. 47. The Discrete Laplacian dif-
fusion is used in the second term. Coherence transport di®usion in Ref. 18 is applied
in the third term. Finally, it tried to ¯nd the global minimum of the energy function
based on iterative way. This method consumes more time and produces poor results
in case of not enough patches in the image.
Arias et al.
2–4
introduced nonlocal variational frameworks for inpainting and their
analysis. In Ref. 4, the image redundancy is encoded as a function of nonlocal weight.
Combination of criterions from di®erent methods such as NL-means, NL-median,
NL-Poisson, and NL-gradient medians is used for patch similarity. Exemplar-based
inpainting in Ref. 36 and PDE di®usion methods are combined based on gradient to
smooth the information propagation around the boundary and inside the damaged
region. In Ref. 3, the authors extended the framework in Ref. 4to be more general
and allowed di®erent derivation of inpainting models. Theoretical analysis of both
frameworks is discussed in Ref. 2.
Zarif et al.
137
proposed an image completion method free of extensive search
process by using only the neighbor pixels. The idea of this method is based on
S. Zarif, I. Faye & D. Rohaya
1554001-32
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

detecting the local similarity of natural image. Fourier transform is used to calculate
the axis symmetrical similarity. Two-side hole ¯lling is used to ¯ll in the damaged
region based on the detected direction. This method uses both known left and right
sides around the missing area to maintain the horizontal similarity direction. The top
and bottom source region is used when the detected direction is vertical. Figure 7
illustrates the replacement process that is used to ¯ll in the hole. The method is
considered one of the fastest methods, it is mainly due to the removal of the search
process. It works well only when the dominant color directions are horizontal or
vertical.
Darabi et al.
39
introduced an image melding method that synthesizing a smooth
transition area between di®erent images to reduce the inconsistent changes. Com-
bination of patch-based synthesis, interpolation, and gradient blending are used to
overcome image cloning, morphing, and stitching challenges. It can be used for single
image tasks like inpainting by using its components. The limitations of this method
are the linearity of search with the number of damaged pixels and unwanted dis-
tortions in some cases.
He and Sun
60
addressed the problem of image completion using statistics of patch
o®sets and optimization. This method consists of three stages. The ¯rst stage is to
match the similar patches in order to ¯nd their o®sets based on SSD. A 2D histogram
is used in the second stage to compute o®sets statistics. In the third stage stack of
combining shifted images is used to complete the damaged structured and texture
information by optimizing a MRFs energy function. Equations (35)–(37) illustrate
the three stages, respectively.
SðXÞ¼argmin
sjjPðXþSÞPðXÞjj2;jSj>thre:; ð35Þ
hði;jÞ¼X
X
ðSðXÞ¼ði;jÞÞ;ð36Þ
EðLÞ¼X
X2
EdðLðXÞÞ þ X
ðX;X0ÞjX2;X02
ESðLðXÞ;LðX0ÞÞ:ð37Þ
In above equations, PðXÞrepresents a patch centered at position X¼ðx;yÞ,
SðXÞis the o®set with 2D coordinates ði;jÞ. The histogram of all o®sets is
Fig. 7. Local similarity replacement ¯lling method (from Ref. 137).
Image Completion
1554001-33
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

represented by hði;jÞwhere ðÞ is one when the argument is true or 0 otherwise. The
data and smoothness terms are represented by Edand Es, respectively. Four con-
nected neighbors and labels are represented by ðX;X0Þand L, respectively. This
method produces results and processing time similar to content aware ¯ll in Pho-
toshop.
1
It is faster and better than other state-of-the-art methods. It fails when the
selected patch o®sets do not represent the right dominant statistics.
Liu and Caselles
86
reformulated the exemplar-based completion in Ref. 41 as a
global minimization optimization problem based on o®set map. Adapted combina-
tion of data and smoothness terms by using multi-scale graph cuts is implemented to
ensure the coherence and continuity inside the damaged region. Feature represen-
tation from the original resolution is applied to propagate the missing information at
low resolution to enhance the correspondences precision and ambiguity. It produces
more plausible results compared to the methods in Refs. 8,20,36,69 and 105 but the
processing time is still high.
3.3.3. Discussion
The decomposition-based methods select the best structure and texture-based method
to separately recover the structure as well as texture information. These methods
produce better results compared to structure- and texture-based method. The absent
of accurate mathematical model to separate the input image into texture and structure
components is the main drawback of these methods. Moreover, the processing time of
these methods is very high due to the decomposition and combination processes. In the
other hand, the main concept of exemplar-based methods is how to complete the
damaged area from the surrounding known image part with the best composition of
structure and texture information. These methods are faster and more e±cient com-
pared to the decomposition-based methods. However, there are numerous challenges
for ¯nding the best patch including patch search, matching of patches, patch size,
priority of ¯lling, optimization, and coherence around the reconstructed patches. In
addition to that, the hidden salient structure and depth ambiguity are the main
drawbacks of most of the methods and are still far from being solved. The summary of
published works on hybrid-based approaches with advantages and drawbacks is
presented in Table 3. The decomposition-based methods are summarized in rows 1
to 9. The remaining part of Table 3represents the exemplar-based methods.
4. Multiple Source Images
Single source image approaches that are explained above are mainly using the known
information around the damaged area for the completion process. For complex
hidden structure and depth ambiguity, it is very di±cult to get perfect results from
single source image-based approaches. Multiple source images approaches are used to
repair the damaged area based on similar images from a database. These methods are
considered to be close solutions in complex situations. Hays and Efros
59
presented a
S. Zarif, I. Faye & D. Rohaya
1554001-34
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Table 3. Summary of hybrid-based approaches, its strengths and drawbacks.
No. Reference Main Idea Optimization Strengths Drawbacks
1 Bertalmio et al.
13
It decomposes the image into structure
and texture components and uses the
methods in Refs. 14 and 47 to repair
components separately.
No It overcomes the smooth e®ect
of the PDE methods.
Still have limitations for reconstruc-
tion of large areas. Decomposition
is not accurate.
2 Yamauchi
et al.
135
Decomposes the image into low and high
frequencies components. Methods in
Refs. 99 and 131 are used to restore low
and high components, respectively.
No It is faster than the method in
Ref. 13.
It produces blurred results due to
convolution process.
3 Grossauer
52
Decomposes the image into geometrical
and texture parts. The geometrical is
restored by Ref. 53. Method in Ref. 131
is used to repair the texture segment
by segment.
No The kinks and edge corners are
preserved.
Quality is parameters dependent, pro-
duces unsatisfactory results when
the object covers multiple segments
with di®erent textures.
4 Shao et al.
108
Laplacian operator and exemplar-based
method in Ref. 36 are performed to
inpaint the structure and texture parts
separately.
No Poisson step improves the
boundaries of copied blocks
to be less noticeable.
Texture segmentation is not accurate.
It is time consuming.
5 Jia and Tang
66
The image is segmented into di®erent
regions by adopting texture segmen-
tation method in Ref. 42. Tensor vot-
ing based on MRF and B-splines are
applied to recover the texture and
curves of each region.
No This method performs well in
most cases.
It fails in case of complex background
and irregular shapes.
6 Elad et al.
48
It separates the image based on sparse
representation using two mutually in-
coherent dictionaries. Pursuit fusion
based on TV is used to complete the
missing layer.
Yes Performs well in case of noise. Results look blurring when the dam-
aged area is large.
7 Bugeau and
Bertalmio
19
It extends the method in Ref. 13 by
inpainting the original image before
decomposition. Improved version of
Ref. 36 based on the structure part is
used for the texture part.
No The quality of the results is
improved compared with
the method in Ref. 13.
It is parameter dependent. Decompo-
sition is not accurate.
Image Completion
1554001-35
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Table 3. (Continued )
No. Reference Main Idea Optimization Strengths Drawbacks
8 Hesabi et al.
63
Same idea of Ref. 13 by using di®erent
methods. The methods in Refs. 6,36
and 107 are used for decomposition,
structure, and texture inpainting.
No The restored images look plau-
sible in general.
It is time consuming and decomposi-
tion is not accurate.
9 Chen and Xu
34,35
Sparse coding and MRF models are used
to represent the structure and texture,
respectively. 3D-junctions function is
used for structure and method in
Ref. 72 for texture. In Ref. 35, Euler
spiral and belief propagation are used
for layers inpainting.
Yes Both methods achieve satisfac-
tory results
Both methods fail when the image sa-
lient structures are complex.
10 Drori et al.
44
It ¯lls the missing region using the prin-
ciple of self-similarity in a multi scale
fashion based on con¯dence map.
No Most of the results seem good in
quality.
It fails to handle the boundary object
and ambiguities of intersected
objects, and very slow.
11 Criminisi et al.
36
It completes patch by patch based on
patch priority. The entire similar
patch is copied instead of single pixel
based on SSD.
No It produces high quality result
around object boundary
and large damaged region.
It ignores the local similarity, searches
in the whole image, and time con-
suming.
12 The papers in
Refs. 37,38,
51,64,73,74,
91,106,119,
124,132,134,
142 and 143
The authors tried to reduce the time and
increase the e±ciency by modifying the
search space, ¯lling priority or patch
matching of the method in Ref. 36.
Some of
them yes
Quality of results has been im-
proved, and some of them
reduce the processing time.
Linear and hidden structures are the
main drawbacks.
13 Sun et al.
112
Signi¯cant damaged structure is manually
selected and completed by using
structure propagation. The remaining
are ¯lled based on patch-based meth-
ods in Refs. 5and 62.
No It produces high quality results. It fails to handle depth ambiguity and
complex missing curves. It requires
user interaction. It takes more time
in large region.
S. Zarif, I. Faye & D. Rohaya
1554001-36
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Table 3. (Continued )
No. Reference Main Idea Optimization Strengths Drawbacks
14 Li and Zhao
84
It used wavelets to avoid the user inter-
action in Ref. 112. Curve ¯tting and
modi¯ed version of Ref. 120 is used in
completion process.
No It produces similar results like
Ref. 112 but without user
interaction.
Depth ambiguity and complex missing
curves are still challenges.
15 Komodakis and
Tziritas
69
Formulated the inpainting as a discrete
global optimization problem based on
Priority belief propagation.
Yes It gives better results compared
to the previous exemplar
based methods.
It takes more time than [91], requires
user guidance, and no evidences
that it leads to a global minima
16 Wexler et al.
130
It optimizes the patch search at di®erent
resolution levels using spatio-temporal
pyramids and nearest neighbor algo-
rithms.
Yes It can handle image and video. Exhaustive searching leads to high
computational load. It is sensitive
to the initialization parameters
17 Kopf et al.
70
It extended the method in Ref. 130 to
formulate a data-driven image com-
pletion method for quality prediction.
Yes It produces satisfactory results
and improved the synthe-
sized texture.
It lacks object level understanding, the
¯tting prediction function is not
perfect due to mismatching.
18 Fang and Lien
50
Presented a multiresolution exemplar
based directional and nondirectional
method to speed up the convergence.
Yes The quality of results and the
processing time are better
compared to Ref. 36.
It consumes more time in the training
stage.
19 Pritch et al.
105
It presented image editing method based
on shift-map and optimal graph
labeling.
Yes It produces impressive results
for reshu²ing and retarget-
ing.
It fails to preserve the structure var-
iations in inpainting.
20 Barnes et al.
8,9
Pioneered fast PatchMatch techniques
that accelerate the family of patch
based methods.
Yes Both of them are an order of
magnitude faster than pre-
vious patch-based methods.
It is not real time.
21 Content aware
¯ll
1
It combines and enhances the method in
Ref. 130 and Patch Match method in
Ref. 8to produce faster and high
quality results.
Yes It considered state-of-the-art
approach in terms of pro-
cessing time and quality.
It is not real time in case of large
images.
22 Bugeau et al.
20
It combines structure, texture and coher-
ence energy terms of existing approa-
ches into one energy function.
Yes The results are more smooth
and coherence.
This method consumes more time.
Quality of results is based on patch
size.
Image Completion
1554001-37
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Table 3. (Continued )
No. Reference Main Idea Optimization Strengths Drawbacks
23 Arias et al.
2–4
Nonlocal variational frameworks for
inpainting and their analysis are dis-
cussed
Yes They provide stability and
convergence in fewer itera-
tions.
These methods consumes more time.
Quality is based on patch size.
24 Zarif et al.
137
It is based on Fourier transform to detect
the axis symmetrical similarity and
two-side hole to ¯ll in the damaged
region based on the detected direction.
No It is very fast. It works well only when the directions
of nature image colors are hori-
zontal or vertical.
25 Darabi et al.
39
Combination of patch based synthesis,
interpolation, and gradient blending
are used to overcome image cloning,
morphing, and stitching challenges.
Yes It can handle multiple applica-
tions like image cloning,
stitching, harmonization,
and morphing.
Consumes more time due to the lin-
earity of search with the number of
damaged pixels, and produces un-
wanted distortions in some cases.
26 He and Sun
60
It uses statistical of patch o®sets and op-
timization for completion. It consists of
patch matching, o®sets statistics, and
stack shift image stages.
Yes It produces similar results and
processing time to content
aware ¯ll in Ref. 1.
It fails when the selected patch o®sets
do not represent the right domi-
nant statistics.
27 Liu and Case-
lles
86
Adaptive combination of data and
smoothness terms by using multi-scale
graph cuts is used to ensure the co-
herence and continuity inside the
damaged region.
Yes It produces more plausible
results compared to the
methods in Refs. 8,20,36,
69 and 105.
Processing time is high.
S. Zarif, I. Faye & D. Rohaya
1554001-38
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

new image completion direction that is based on several million images database to
restore the missing region. The method searches for similar areas from the database
using gist descriptor in Ref. 121 to reduce the search space and get the closest images.
Graph cuts texture method in Ref. 72 and Poisson blending in Ref. 102 are used to
merge the new region into the damaged area. It produces several results and the user
can select the best. This method is time consuming, it requires more than an hour to
restore one image on a single CPU. Moreover, it fails when the image matching did
not get the best match. Liu et al.
87,88
proposed another idea for completing the
damaged area and occluded objects based on large displacement view images by
moving the camera away from obstacles. Li et al.
82
introduced multiple source images
method based on scene and color transform. This method tried to get the best color,
texture, and structure features from image database that are most similar to the
characteristics of damaged region. Cost function is used to adjust the imputed
sideline. Clustering-based color transform is used to smooth the repaired area with its
surrounding. This method fails to solve the border defects and does not produce
satisfactory results when the number of images in the database is limited.
Finally, multiple source images-based approaches are considered an alternative
solution to the single source image approach in case of complex situations. The main
drawbacks of these approaches are that they require huge number of images in the
database, time consuming, and require more coherence process to smooth the
background when the matched texture is not the same.
5. Comparison Study
We start our comparison study by comparing the ¯lling categories together and
listing its applications. Table 4illustrates the di®erences between the structure,
texture, hybrid, and multiple source images-based categories. Six features are used in
the comparison. The hybrid-based category is considered as the most general and the
Table 4. Structure-, texture-, hybrid-, and multiple source images-based comparison.
Features
Structure-Based
Category
Texture-Based
Category
Hybrid-Based
Category
Multiple Images-
Based Category
Hole size Small Large Large Large
Filling data Geometrical Texture Composite of structure
and texture
Composite of
structure and
texture
Quality Low Medium High High
Required images One Two One Database of images
Processing time Most of them high Most of them me-
dium
Most of them low Very high
Applications Noise and text re-
moval, scrat-
ches, restora-
tion of lost
blocks
Generating large
texture, surface
covering, and
object removal
Object removal, resto-
ration of large
damaged area, and
image editing
Object removal,
restoration of
large damaged
area, and image
editing
Image Completion
1554001-39
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

most successful category to handle large damaged region with composite of structure
and texture information. Moreover, it is based on a single image and most of the
methods produce high quality results in reasonable time compared to other cate-
gories. In addition to that, hybrid-based category supports wide variety of applica-
tions. To illustrate and ensure the e®ectiveness of the hybrid-based methods
compared to other categories, we selected some of the most famous and successful
methods in each category and implemented them on the same image. The methods in
Refs. 18 and 122 are selected from the structure-based category. The methods in
Refs. 1,19,20,36,74,130 and 137 are selected from the hybrid-based category.
Finally the method in Ref. 59 is selected from multiple source images category.
Figure 8presents the results for a visual comparison between the selected methods on
(a) (b) (c) (d)
(e) (f) (g) (h)
(i) (j) (k) (l)
Fig. 8. Di®erent categories based comparison for challenge image: (a) Original image, (b) masked image,
(c) result of method in Ref. 122, (d) result of method in Ref. 18, (e) result of method in Ref. 19, (f) result of
method in Ref. 36, (g) result of method in Ref. 130, (h) result of method in Ref. 1, (i) result of method
in Ref. 20, (j) result of method in Ref. 137, (k) result of method in Ref. 74 and (l) result of method
in Ref. 59.
S. Zarif, I. Faye & D. Rohaya
1554001-40
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

one of the challenge images. The results from the structure-based category in
Figs. 8(c) and 8(d) look blur and the artifacts are visually clear as well as the result
from the decomposition-based method in Fig. 8(e). In the other side, most of the
results from exemplar-based methods and multiple source images category look good.
The main problems of multiple source images category are the processing time and
the requirement of huge number of images in the database as mentioned before.
Therefore, our comparison concentrates more on the hybrid-based methods.
There are numerous challenges to compare the state-of-the-art methods. The ¯rst
challenge is the lack of general dataset to measure and compare the quality of hybrid-
based techniques in the same dataset. The second challenge is the unavailability of
the source codes to compare the processing time in the same hardware. Processing
time is one of the biggest challenges that face completion in video. The third chal-
lenge is the absence of mathematical metrics to evaluate the quality of the results.
Therefore, our comparative study consists of the following:
.Quantitative-based comparison.
.Qualitative-based comparison.
5.1. Quantitative-based comparison
In this section, we present a peak signal to noise ratio (PSNR) and processing time
comparisons between four exemplar-based methods. The selection of the four
methods is based on the availability of source code implementation of these methods.
The available four methods are:
.Exemplar-Based Method 2004.
36
.Space Time Method 2007.
130
.Content Aware Fill 2010.
1
.Local Similarity Method 2012.
137
For comprehensive and fair processing time evaluation, the four methods have
been tested on a PC with Pentium Dual Core 2.8 GHz CPU and one GB of RAM as
well as same images. In parallel to the processing time, a peak signal to noise ratio is
used to measure the quality of the results. PSNR cannot used to measure the quality
after object removal due to the di®erence between the original (with object) and
completed (without object). Therefore, we created arti¯cial damaged region in tex-
ture images. Then, we applied the four methods to restore the original images. After
that, we calculated the PSNR between the original images and the inpainted images.
If Iði;jÞis the original image of size mby nand Kði;jÞis the completed image with
the same size, the mean square error (MSE) is given by:
MSE ¼1
mn X
m1
i¼0X
n1
j¼0½Iði;jÞKði;jÞ2;ð38Þ
Image Completion
1554001-41
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

PSNR ¼10 log
10
255 255
MSE

:ð39Þ
Figure 9illustrates the results of the four methods after restoring the arti¯cially
damaged images. Table 5shows PSNR comparisons for the images in Fig. 9. Table 6
presents processing time comparisons between the four methods for the images in
(a) (b) (c) (d)
Fig. 9. Example of restoration of original images from arti¯cially damaged images. Row 1 represents the
original images, row 2 represents the arti¯cial damaged images, row 3 represents the results of method in
Ref. 36, row 4 represents the results of method in Ref. 130, row 5 represents the results of method in Ref. 1,
and row 6 represents the results of method in Ref. 137.
S. Zarif, I. Faye & D. Rohaya
1554001-42
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Figs. 8–11. It is obvious that, the methods in Refs. 1and 137 produce high quality
results for the texture images in Fig. 9and faster processing time compared to others.
5.2. Qualitative-based comparison
According to the changes that happened to the completed image in both structure
and color information after removing unwanted object, it is very di±cult to evaluate
the quality by traditional objective evaluation such as PSNR. In Ref. 100, some
other quantitative metrics have been proposed to validate the inpainting results.
These metrics are still not perfect and the performance of them is image dependent.
So, the quality depends on the human visual perception system (HVPS) rather
than mathematical measures.
82,137,138
The human visual evaluation is commonly
accepted.
Subjective study based on HVPS is considered in our qualitative comparison to
compare the e±ciency of the four methods in Refs. 1,36,130 and 137. We create a
0
10
20
30
40
50
60
Method in
Ref. 36
Method in
Ref. 130
Method in
Ref. 1
Method in
Ref. 137
Average # of preferred
images
Fig. 10. Average preferred images from 30 observers on 60 images dataset.
Table 6. Processing time comparisons between the four methods in Refs. 1,36,130 and 137 (in s).
Image Size Method in Ref. 36 Method in Ref. 130 Method in Ref. 1Method in Ref. 137
Fig. 8800*600 520 730 1.02 0.052
Fig. 9(a) 300*240 118 432 0.467 0.021
Fig. 9(b) 620*620 301 2300 0.853 0.095
Fig. 9(c) 250*345 114 530 0.488 0.026
Fig. 9(d) 250*250 71 300 0.520 0.020
Fig. 10 320*250 140 470 0.478 0.025
Fig. 11 320*250 122 447 0.475 0.024
Table 5. PSNR comparisons between the four methods in Refs. 1,36,130 and 137.
Image Size Method in Ref. 36 Method in Ref. 130 Method in Ref. 1Method in Ref. 137
Fig. 9(a) 300*240 19.97 21.16 21.79 21.61
Fig. 9(b) 620*620 52.33 54.47 52.78 62.27
Fig. 9(c) 250*345 22.69 23.17 24.18 24.39
Fig. 9(d) 250*250 28.95 35.60 36.09 29.46
Image Completion
1554001-43
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

new dataset consist of a variety of texture, structure, and natural images as well as
deferent damaged region size and test each of them on it. The dataset consists of 60
original images and 60 inpainted versions for each method. We applied the com-
puterized subjective study on 30 observers. All the observers are postgraduate stu-
dents and working in di®erent image processing topics. For more accurate results, the
subjective study was done without time limitation.
For each observer, the subjective study presented the original image and the four
inpainted versions from the methods in Refs. 1,36,130 and 137 at the same time.
The positions of the four inpainted versions are randomly set. For each presented
image, the observer is asked to specify which result is better. Each observer evaluates
all the 60 images in the dataset. Figure 10 illustrates the average voting of the 30
observers. Most observers prefer the results of the method in Ref. 1followed by the
method in Ref. 137.
For other methods that are not available, a variety of experiments consisting of
completely damaged regions after removing unwanted objects from natural images
are conducted. From the previous literature reviews, we selected two challenging
images in our human visual comparison. It is very di±cult to compare all the
(a) (b) (c) (d)
(e) (f) (g) (h)
(i) (j) (k) (l)
Fig. 11. Visual-based comparison for challenge image: (a) Original image, (b) masked image, (c) result of
method in Ref. 122, (d) result of method in Ref. 18, (e) result of method in Ref. 19, (f) result of method in
Ref. 36, (g) result of method in Ref. 69, (h) result of method in Ref. 130, (i) result of method in Ref. 1, (j)
result of method in Ref. 20, (k) result of method in Ref. 60 and (l) result of method in Ref. 74.
S. Zarif, I. Faye & D. Rohaya
1554001-44
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

methods that were discussed on these images. We selected the most recent and the
state-of-art methods. Figures 11 and 12 present a visual comparison between 10 and
11 recent methods respectively on the same image. Both the structure-based methods
in Refs. 18 and 122 fail to restore the salient structure due to the large size of the
damaged region. Few methods in Refs. 1,20,37 and 60 from the exemplar-based
category successfully produce satisfactory results in both ¯gures.
Finally, in Fig. 13 we illustrate one of the open problems that are still far from
being solved. Most of the successful methods in the previous ¯gures fail to solve the
depth ambiguity problem. The authors of Ref. 36 mentioned that the depth ambi-
guity is one of the limitations of their method as represented in Fig. 13(b). The
method in Ref. 112 treated the problem as a planar graph and did not take into
account the occlusion. Figure 13(c) shows its results with the help of user interaction
to select the salient structure. The results of space time method in Ref. 130 and
content aware ¯ll in adobe photoshop
1
are presented in Figs. 13(d) and 13(e), re-
spectively. Local similarity method in Ref. 137 fails to handle this problem by using
the local information as illustrated in Fig. 13(f). Statistical of patch o®set method in
(a) (b) (c) (d)
(e) (f) (g) (h)
(i) (j) (k) (l)
Fig. 12. Visual-based comparison for hidden structure: (a) Original image, (b) result of method in
Ref. 18, (c) result of method in Ref. 36, (d) result of method in Ref. 112, (e) result of method in Ref. 69, (f)
result of method in Ref. 130, (g) result of method in Ref. 105, (h) result of method in Ref. 1, (i) result of
method in Ref. 60, (j) result of method in Ref. 137, (k) result of method in Ref. 74 and (l) result of method
in Ref. 37.
Image Completion
1554001-45
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

Ref. 60 as well cannot solve the problem by introducing the o®sets manually as
shown in Fig. 13(g). The result in Fig. 13(h) introduces the failure of the hierarchical
super resolution method in Ref. 74 to handle the depth ambiguity.
6. Conclusion
Fill in damaged parts in an image and video is an active and interesting topic in the
past few years. There are numerous techniques that have been published to handle
variety of applications such as disocclusion, restoration, object removal, and texture
synthesis. In the ¯rst part of this work, we introduce a new classi¯cation for these
techniques and assign each of them to the related category. Technical details for each
method in each category are provided to illustrate the strengths and drawbacks. In
the second part of this work, qualitative and quantitative comparative studies be-
tween the most successful and recent methods are provided. To be fair as much as
possible the four available image completion methods are compared using the same
dataset and the same device hardware. The experimental results illustrate the
strengths and drawbacks of each method. We can conclude from the comparative
study that the exemplar-based category is the most global and successful category
that can handle large damaged region with composition of structure and texture
information. The exemplar-based methods in Refs. 1,60,74 and 137 have the best
performances in terms of processing time and quality. Therefore, we recommend
them for all researchers to be applied in their video completion future work.
Depth ambiguity, complex damaged structure curves, processing time, and
quality assessment are still open problems in this ¯eld.
(a) (b) (c) (d)
(e) (f) (g) (h)
Fig. 13. Example to one of the most open problems in image completion where no exemplar-based
method seems to work.
S. Zarif, I. Faye & D. Rohaya
1554001-46
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

References
1. ADOBE, Photoshop CS5, Content-aware ¯ll (2010), available at http://www.adobe.
com/technology/projects/content-aware-¯ll.html.
2. P. Arias, V. Caselles and G. Facciolo, Analysis of a variational framework for exemplar-
based image inpainting, Multiscale Model. Simulat. 10(2) (2012) 473–514.
3. P. Arias, V. Caselles, G. Facciolo, V. Lazcano and R. Sadek, Nonlocal variational models
for inpainting and interpolation, Math. Models Methods Appl. Sci. 22 (2012) 1–65.
4. P. Arias, G. Facciolo, V. Caselles and G. Sapiro, A variational framework for exemplar-
based image inpainting, Int. J. Comput. Vis. 93 (2011) 319–347.
5. M. Ashikhmin, Synthesizing natural textures, ACM Symp. Interactive 3D Graphics
(2001), pp. 217–226.
6. J. Aujol and S. Kang, Color image decomposition and restoration, J. Vis. Commun.
Image Represent. 17(4) (2006) 916–928.
7. D. Auroux and M. Masmoudi, A one-shot inpainting algorithm based on the topological
asymptotic analysis, Comput. Appl. Math. 25 (2006) 251–267.
8. C. Barnes, E. Shechtman, A. Finkelstein and D. B. Goldman, PatchMatch: A ran-
domized correspondence algorithm for structural image editing, ACM Trans. Graph. 28
(2009) 1–11.
9. C. Barnes, E. Shechtman, D. B. Goldman and A. Finkelstein, The generalized Patch-
Match correspondence algorithm, ECCV (2010), pp. 29–43.
10. A. Beck and M. Teboulle, A fast iterative shrinkage-thresholding algorithm for linear
inverse problems, SIAM J. Image Sci. 2(1) (2009) 183–202.
11. M. Bertalmio, A. Bertozzi and G. Sapiro, Navier-Stokes, °uid dynamics, and image and
video inpainting, IEEE Computer Vision and Pattern Recognition (CVPR'01) (2001),
pp. 355–362.
12. M. Bertalmío, V. Caselles, S. Masnou and G. Sapiro, Inpainting, in Encyclopedia of
Computer Vision (Springer, 2011).
13. M. Bertalmio, G. Sapiro and V. Osher, Simultaneous structure and texture images
inpainting, IEEE Trans. Image Process. 12(8) (2003) 882–889.
14. M. Bertalmio, G. Sapiro, V. Caselles and C, Ballester, Image inpainting, in Proc.
SIGGRAPH (2000), pp. 417–424.
15. A. Bertozzi, S. Esedoglu and A. Gillette, Inpainting of binary images using the Cahn–
Hilliard equation, IEEE Trans. Image Process. 16(1) (2007) 285–291.
16. R. Biradar and V. Kohir, A novel image inpainting technique based on median di®usion,
Sadhana 38(4) (2013) 621–644.
17. J. Bonet, Multiresolution sampling procedure for analysis and synthesis of texture
images, in Proc. SIGGRAPH (1997), pp. 361–368.
18. F. Bornemann and T. Marz, Fast image inpainting based on coherance transport, J.
Math. Imaging Vis. 28(3) (2007) 259–278.
19. A. Bugeau and M. Bertalmío, Combining texture synthesis and di®usion for image
inpainting, Int. Conf. Computer Vision Theory and Applications (2009), pp. 26–33.
20. A. Bugeau, M. Bertalmio, V. Caselles and G. Sapiro, A comprehensive framework for
image inpainting, IEEE Trans. Image Process. 19(10) (2010) 2634–2645.
21. B. Cabral and L. C. Leedom, Imaging vector ¯elds using line integral convolution, in
Proc. SIGGRAPH, Vol. 27 (1993), pp. 263–272.
22. J. Cai, R. Chan and Z. Shen, A framelet-based image inpainting algorithm, Appl.
Comput. Harmon. Anal. 24(2) (2008) 131–149.
23. J. Cai, R. Chan, L. Shen and Z. Shen, Convergence analysis of tight framelet approach
for missing data recovery, Adv. Comput. Math. 31 (2009) 87–113.
Image Completion
1554001-47
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

24. A. Chambolle, An algorithm for total variation minimization and applications, J. Math.
Imaging Vis. 20(1/2) (2004) 89–97.
25. T. Chan, S. Kang and J. Shen, Euler's elastica and curvature based inpaintings, SIAM J.
Appl. Math. 63(2) (2002) 564–592.
26. T. Chan and J. Shen, Mathematical models for local non-texture inpainting, SIAM J.
Appl. Math. 62(3) (2001) 1019–1043.
27. T. Chan and J. Shen, Non texture inpainting by curvature driven di®usions, J. Vis.
Commun. Image Represent. 12(4) (2001) 436–449.
28. R. Chan, L. Shen and Z. Shen, A Framelet-based approach for image inpainting, Tech.
Rep. 2005-4, The Chinese Univ. Hong Kong, Hong Kong (2005).
29. T. Chan, J. Shen and H. Zhou, Total variation wavelet inpainting, J. Math. Imaging Vis.
25(1) (2006) 107–125.
30. T. Chan, J. Shen and H. Zhou, A total variation wavelet inpainting model with mul-
tilevel ¯tting parameters, in Proc. SPIE, Vol. 6313, Art. no. 63130C (2006).
31. R. Chan, Y. Wen and A. Yip, A fast optimization transfer algorithm for image
inpainting in wavelet domains, IEEE Trans. Image Process. 18(7) (2009) 1467–1476.
32. R. Chan, J. Yang and X. Yuan, Alternating direction method for image inpainting in
wavelet domains, SIAM J. Imaging Sci. 4(3) (2011) 807–826.
33. T. Chan and H. Zhou, Optimal constructions of wavelet coe±cients using total variation
regularization in image compression, CAM Report, No. 00–27, UCLA, Applied Math-
ematics (2000).
34. X. Chen and F. Xu, Automatic image inpainting by heuristic texture and structure
completion, Advances in Multimedia Modeling, LNCS, Vol. 5916 (Springer Berlin Hei-
delberg, 2010), pp. 110–119.
35. X. Chen, B. Zhou, Y. Guo, F. Xu and Q. Zhao, Structure guided texture inpainting
through multi-scale patches and global optimization for image completion, Sci. China
Inf. Sci. 57(1) (2014) 1–16.
36. A. Criminisi, P. Perez and K. Toyama, Region ¯lling and object removal by examplar-
based inpainting, IEEE Trans. Image Process. 13(9) (2004), 1200–1212.
37. T. Dang, A. Beghdadi and M. Larabi, A perceptual image completion approach based on
a hierarchical optimization scheme, Signal Process 103 (2014) 127–141.
38. T. Dang, M. Larabi and A. Beghdadi, Multi-resolution patch and window-based priority
for digital image inpainting problem, Int. Conf. Image Proc. Theory, Tools and Appl.
(2012), pp. 280–284.
39. S. Darabi, E. Shechtman, C. Barnes, D. Goldman and P. Sen, Image melding: Com-
bining inconsistent images using patch-based synthesis, ACM Trans. Graph. 31(4)
(2012) 1–10.
40. I. Daubechies, B. Han, A. Ron and Z. Shen, Framelets: MRA-based constructions of
wavelet frames, Appl. Comput. Harmon. Anal. 14 (2003) 1–46.
41. L. Demanet, B. Song and T. Chan, Image inpainting by correspondence maps: A de-
terministic approach, Proc. VLSM (2003), pp. 1–8.
42. Y. Deng and B. Manjunath, Unsupervised segmentation of color-texture regions in
images and video, IEEE Trans. Pattern Anal. Mach. Intell. 8(23) (2001) 800–810.
43. J. Dobrosotskaya and A. Bertozzi, A wavelet-Laplace variational technique for image
deconvolution and inpainting, IEEE Trans. Image Process. 17(5) (2008) 657–663.
44. I. Drori, D. Cohen and H. Yeshurun, Fragment-based image completion, in Proc.
SIGGRAPH, Vol. 22 (2003), pp. 303–312.
45. S. Durand and J. Froment, Artifact free signal denoising with wavelets, in Proc.
ICASSP, Vol. 6 (2001), pp. 3685–3688.
S. Zarif, I. Faye & D. Rohaya
1554001-48
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

46. A. Efros and W. Freeman, Image quilting for texture synthesis and transfer, in Proc.
SIGGRAPH (2001), pp. 341–346.
47. A. Efros and T. Leung, Texture synthesis by non-parametric sampling, IEEE 7th Int.
Conf. Computer Vision, Vol. 2 (1999), pp. 1033–1038.
48. M. Elad, J. Stark, P. Querre and D. Donoho, Simultaneous cartoon and texture image
inpainting using morphological component analysis (MCA), Appl. Comput. Harmon.
Anal. 19 (2005) 340–358.
49. S. Esedoglu and J. Shen, Digital image inpainting by the Mumford-Shah Euler image
model, Eur J. Appl. Math. 13 (2002) 353–370.
50. C. Fang and J. Lien, Rapid image completion system using multiresolution patch-based
directional and nondirectional approaches, IEEE Trans. Image Process. 18(12) (2009)
2769–2779.
51. D. Florinabel, S. Juliet and V. Sadasivam, Combined frequency and spatial
domain-based patch propagation for image completion, Comput Graph 35 (2011) 1051–
1062.
52. H. Grossauer, A combined PDE and texture synthesis approach to inpainting, in
ECCV, LNCS, Vol. 3022 (2004), pp. 214–224.
53. H. Grossauer and O. Scherzer, Using the complex Ginzburg–Landau equation for digital
inpainting in 2D and 3D, Scale Space Methods in Computer Vision, LNCS, Vol. 2695
(Springer Berlin Heidelberg, 2003), pp. 225–236.
54. J. Gu, S. Peng and X. Wang, Digital image inpainting using Monte Carlo method, IEEE
Int. Conf. Image Processing, Vol. 2 (2004), pp. 961–964.
55. C. Guillemot and O. Le Meur, Image inpainting: Overview and recent advances, IEEE
Signal Process. Mag. 31(1) (2013) 127–144.
56. W. Guo and L. Qiao, Inpainting based on total variation, IEEE Int. Conf. Wavelet
Analysis and Pattern Recognition (2007), pp. 939–943.
57. C. Guo, S. Zhu, Guo, S. Zhu and Y. Wu, Primal sketch: Integrating structure and
texture, Comput. Vis. Image Understand. 106(1) (2007) 5–19.
58. M. M. Hadhoud, K. A. Moustafa and S. Z. Shendoa, Digital images inpainting using
modi¯ed convolution based method, Int. J. Signal Process. Image Proces. Pattern
Recogn. 1(1) (2008) 1–10.
59. J. Hays and A. Efros, Scene completion using millions of photographs, ACM Trans.
Graph 26(3) (2007) 1–8.
60. K. He and J. Sun, Statistics of patch o®sets for image completion, ECCV (2012),
pp. 16–29.
61. D. Heeger and J. Bergen, Pyramid based texture analysis/synthesis, in Proc. SIG-
GRAPH (1995), pp. 229–238.
62. A. Hertzmann, C. Jacobs, N. Oliver, B. Curless and D. Salesin, image analogies, in Proc.
SIGGRAPH (2001), pp. 327–340.
63. S. Hesabi, M. Jamzad and N. Mahdavi, Structure and texture image inpainting, IEEE
Int. Conf. Signal and Image Processing (2010), pp. 119–124.
64. Z. Hongying, J. Yuhong and W. Yadong, Image completion by a fast and adaptive
exemplar-based image inpainting, IEEE Int. Conf. Computer Application and System
Modeling (2010), pp. 115–119.
65. R. Jadav, S. Patel and D. Joshi, A comparative study of di®erent techniques in image
inpainting, Indian J. Eng. 3(7) (2013) 41–44.
66. J. Jia and C. Tang, Image repairing: Robust image synthesis by adaptive ND tensor
voting, in Proc. IEEE CVPR (2003), pp. 643–650.
67. H. Jiang and C. Moloney, Error concealment using a di®usion based method, IEEE
Inter. Conf. Image Processing, Vol. 1 (2002), pp. 832–835.
Image Completion
1554001-49
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

68. E. King, G. Kutyniok and W. Lim, Image inpainting: Theoretical analysis and com-
parison of algorithms, in Proc. SPIE, Vol. 8858, id. 885802 (2013), pp. 1–11.
69. N. Komodakis and G. Tziritas, Image completion using e±cient belief propagation via
priority sechaduling and dynamic prunning, IEEE Trans. Image Process. 16(11) (2007)
2649–2661.
70. J. Kopf, W. Kienzle, S. Drucker and S. Kang, Quality prediction for image completion,
ACM Trans. Graph. 31(6) (2012) 1–8.
71. V. Kwatra, I. Essa, A. Bobick and N. Kwatra, Texture optimization for exemplar based
synthesis, in Proc. SIGGRAPH (2005), pp. 795–802.
72. V. Kwatra, A. Schodl, I. Essa, G. Turk and A. Bobick, Graphcut textures: Image and
video synthesis using graph cuts, in Proc. SIGGRAPH (2003), pp. 277–286.
73. T. Kwok, H. Sheung and C. Wang, Fast query for exemplar-based image completion,
IEEE Trans. Image Process. 19(12) (2010) 3106–3115.
74. O. Le Meur, M. Ebdelli and C. Guillemot, Hierarchical super-resolution based inpaint-
ing, IEEE Trans. Image Process. 22(10) (2013) 3779–3790.
75. S. Lefebvre and H. Hoppe, Parallel controllable texture synthesis, in Proc. SIGGRAPH
(2005), pp. 777–786.
76. S. Lefebvre and H. Hoppe, Appearance-space texture synthesis, in Proc. SIGGRAPH
(2006), pp. 541–548.
77. A. Levin, A. Zomet and Y. Weiss, Learning how to inpaint from global image statistics,
in Proc. IEEE ICCV (2003), pp. 305–313.
78. Y. Li, D. Dai and L. Shen, Multiframe super-resolution reconstruction using
sparse directional regularization, IEEE Trans. Circuits Syst. Video Technol. 20(7)
(2010) 945–956.
79. Y. Li, L. Shen, D. Dai and B. Suter, Framelet algorithms for deblurring images corrupted
by impulse plus Gaussian noise, IEEE Trans. Image Process. 20(7) (2011) 1822–1837.
80. Y. Li, L. Shen and B. Suter, Adaptive inpainting algorithm based on DCT induced
wavelet regularization, IEEE Trans. Image Process. 22(2) (2013) 752–763.
81. S. Li and H. Wang, Image inpainting using curvature-driven di®usions based on
P-Laplace operator, in Proc. Int. Conf. Innovation Computing, Information, and
Control (2009), pp. 323–325.
82. H. Li, S. Wang, W. Zhang and M. Wu, Image inpainting based on scene transform and
color transfer, Pattern Recogn. Lett. 31 (2010) 582–592.
83. L. Li and H. Yu, Nonlocal curvature-driven di®usion model for image inpainting, IEEE
Int. Conf. Information Assurance and Security (2009), pp. 513–516.
84. S. Li and M. Zhao, Image inpainting with salient structure completion and texture
propagation, Pattern Recogn. Lett. 32 (2011) 1256–1266.
85. L. Liang, C. Liu, Q. Xu, B. Guo and H. Shum, Real-time texture synthesis by patch
based sampling, ACM Trans. Graph. 20(3) (2001) 127–150.
86. Y. Liu and V. Caselles, Exemplar-based image inpainting using multiscale graph cuts,
IEEE Trans. Image Process. 22(5) (2013) 1699–1711.
87. C. Liu, Y. Guo, L. Pan, Q. Peng and F. Zhang, Image completion based on views of large
displacement, Vis. Comput. 23(9) (2007) 833–841.
88. C. Liu, Y. Guo, Q. Peng, J. Wang and W. Chen, Distortion optimization based
image completion from a large displacement view, Comput. Graph. Forum 27(7) (2008)
1755–1764.
89. Z. Liu, Z Li and W. Hu, An improved algorithm for wavelet image inpainting, J.
Comput. Inf. Syst. 9(11) (2013) 4415–4423.
90. C. Liu, Q. Peng and X. Wang, Recent development in image completion techniques,
IEEE Int. Conf. Computer Science and Automation Engineering (2011), pp. 756–760.
S. Zarif, I. Faye & D. Rohaya
1554001-50
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

91. J. Liu, S. Zhang, W. Yang and H. Li, A fast image inpainting method based on hybrid
simiarty-distance, IEEE Int. Conf. Pattern Recognition (2010), pp. 4432–4435.
92. X. Lu, W. Wang and D. Zhuoma, A fast image inpainting algorithm based on TV model,
in Proc. Int. Multiconf. Engineers and Computer Scientists (2010), pp. 1457–1460.
93. T. Marz, Image inpainting based on coherence transport with adapted distance func-
tions, SIAM J. Imaging Sci. 4(4) (2010) 981–1000.
94. S. Masnou, Disocclusion: A variational approach using level lines, IEEE Trans. Image
Process. 11(2) (2002) 68–76.
95. S. Masnou and J. M. Morel, Level line based disocclusion, IEEE 5th Int. Conf. Image
Processing (1998), pp. 259–263.
96. H. Noori, S. Saryazdi and H. Nezamabadi, A convolution based image inpainting, Int.
Conf. Comunication Engineering (2010), pp. 130–134.
97. H. Noori, S. Saryazdi and H. Nezamabadi, A bilateral image inpainting, IJST, Trans.
Electr. Eng. 35(E2) (2011) 95–108.
98. T. Ogawa and M. Haseyama, Image inpainting based on sparse representations with a
perceptual metric, EURASIP J. Adv. Signal Process. (1) (2013) 1–26.
99. M. Oliveira, B. Bowen, R. Mckenna and Y. S. Chang, Fast digital image inpainting, in
Proc. VIIP (2001), pp. 261–266.
100. A. Oncu, F. Deger and J. Hardeberg, Evaluation of digital inpainting quality in the
context of artwork restoration, 12th European Conf. Computer Vision (ECCV) (2012),
pp. 561–570.
101. P. Patel, A. Prajapati and S. Mishra, Review of di®erent inpainting algorithms, Int. J.
Comput. Appl. 59(18) (2012) 30–34.
102. P. Perez, M. Gangnet and A. Blake, Poisson image editing, ACM Trans. Graph. 22(3)
(2003) 313–318.
103. P. Perona and J. Malik, Scale–space and edge detection using anisotropic di®usion,
IEEE Trans. Pattern Anal. Mach. Intell. 12(7) (1990) 629–639.
104. J. Portilla and E. Simoncelli, A parametric texture model based on joint statistics of
complex wavelet coe±cients, Int. J. Comput. Vis. 40(1) (2000) 49–71.
105. Y. Pritch, E. Kav-Venaki and S. Peleg, Shift-map image editing, ICCV (2009),
pp. 151–158.
106. A. Rares, M. Reinders and J. Biemond, Edge based image restoration, IEEE Trans.
Image Process. 14 (2005) 1454–1468.
107. S. Roth, and M. Black, Fields of experts, Int. J. Comput. Vis. 82(2) (2009) 205–229.
108. X. Shao, Z. Liu and H. Li, An image inpainting approach based on the Poisson equation,
IEEE Int. Conf. Document Image Analysis for Libraries (2006), pp. 368–372.
109. T. Shih and R. Chang, Digital inpainting —Survey and multilayer image inpainting
algorithms, IEEE Int. Conf. Information Technology and Application (2005), pp. 15–24.
110. E. Simoncelli and J. Portilla, Texture characterization via joint statistics of wavelet
coe±cient magnitudes, in Proc. 5th Int. Conf. Image Processing (1998), pp. 62–66.
111. J. Stark, M. Elad and D. Donoho, Image decomposition via the combination of sparse
representations and variational approach, IEEE Trans Image Process. 14(10) (2005)
1570–1582.
112. J. Sun, L. Yuan, J. Jia and H. Shum, Image completion with structure propagation, in
Proc. SIGGRAPH (2005), pp. 861–868.
113. D. Sun, L. Yuan, Y. Zhang and G. Pan, Structure-aware image completion with texture
propagation, IEEE 6th Int. Conf. Image and Graphics (ICIG) (2011), pp. 199–204.
114. X. Tai, S. Osher and R. Holm, Image inpainting using TV-Stokes equation, in Image
Processing Based on Partial Di®erential Equations (2007), pp. 3–22.
Image Completion
1554001-51
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

115. P. Tang, Application of non-parametric texture synthesis to image inpainting, Master
Thesis, University of New Mexico, Albuquerque (2004).
116. F. Tang, Y. Ying, J. Wang and Q. Peng, A novel texture synthesis based algorithm
for object removal in photographs, Advances in Computer Science - ASIAN 2004,
Higher-Level Decision Making, LNCS, Vol. 3321 (Springer Berlin Heidelberg, 2004),
pp. 248–258.
117. Z. Tauber, Z. Li and M. Drew, Review and preview: Disocclusion by inpainting for
image-based rendering, IEEE Trans. Syst., Man, Cybern. C, Appl. Rev. 37(4) (2007)
527–540.
118. A. Telea, An image inpainting technique based on the fast marching method, J. Graph.
Tools 9(1) (2004) 23–34.
119. K. Thangamani, T. Ishikawa, K. Makita and T. Kurata, Fast and texture-structure
preservable inpainting and its application for creating virtualized-reality indoor models,
Int. J. Comput. Sci. Inform. 1(3) (2012) 85–101.
120. L. Tonietto, M. Walter and C. Jung, Patch-based texture synthesis using
wavelets, SIBGRAPI, IEEE Symp. Computer Graphics and Image Processing (2005),
pp. 383–389.
121. A. Torralba, K. Murphy, W. Freeman and M. Rubin, Context-based vision system for
place and object recognition, ICCV (2003), pp. 273–280.
122. D. Tschumperle, Fast anisotropic smoothing of multi-valued images using curvature-
preserving PDE's, Int. J. Comput. Vis. 68(1) (2006) 65–82.
123. L. Vese and S. Osher, Modeling textures with total variation minimization and oscil-
lating patterns in image processing, J. Sci. Comput. 19(13) (2003) 553–572.
124. J. Wang, K. Lu, D. Pan, N. He and B. Bao, Robust object removal with an exemplar-
based image inpainting approach, Neurocomputing 123 (2014) 150–155.
125. Y. Wang, J. Yang, W. Yin and Y. Zhang, A new alternating minimization algorithm for
total variation image reconstruction, SIAM J. Imaging Sci. 1(3) (2008) 248–272.
126. Y. Wang and Y. Zhang, Total variation wavelet-based medical image denoising, Int.
J. Biomed. Imaging (2006) 1–6.
127. Z. Wang, F. Zhou and F. Qi, Inpainting thick image regions using isophote propagation,
IEEE Int. Conf. Image Processing (2006), pp. 689–692.
128. L. Wei, S. Lefebvre, V. Kwatra and G. Turk, State of the art in example-based texture
synthesis, in Eurographics'09, State of the Art Report, EG-STAR (2009), pp. 93–117.
129. Y. Wen, R. Chan and A. Yip, A primal–dual method for total-variation-based wavelet
domain inpainting, IEEE Trans. Image Process. 21(1) (2012) 106–114.
130. Y. Wexler, E. Shechtmam and M. Irani, Space time completion of video, IEEE Trans.
Pattern Anal. Mach. Intell. 29(2007) 463–476.
131. L. Wey and M. Levoy, Fast texture synthesis using tree-structured vector quantization,
in Proc. SIGGRAPH (2000), pp. 479–488.
132. J. Wu and Q. Ruan, Object removal by cross isophotes exemplar based image
inpainting, IEEE Int. Conf. Pattern Recognition (ICPR) (2006), pp. 810–813.
133. Z. Xu, X. Lian and L. Feng, Image inpainting algorithm based on partial di®erential
equation, in Proc. Int. Colloquium on Computing, Communication, Control, and
Management (2008), pp. 120–124.
134. Z. Xu and J. Sun, Image inpainting by patch propagation using patch sparsity, IEEE
Trans. Image Process. 19(5) (2010) 1153–1165.
135. H. Yamauchi, J. Haber and H. Seidel, Image restoration using multiresolution
texture synthesis and image inpainting, in Proc. Computer Graphics Int. (CGI) (2003),
pp. 120–125.
S. Zarif, I. Faye & D. Rohaya
1554001-52
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.

136. A. Yau, X. Tie and M. Ng, L0-Norm and total variation for wavelet inpainting, Scale
Space and Variational Methods in Computer Vision, LNCS, Vol. 5567 (Springer Berlin
Heidelberg, 2009), pp. 539–551.
137. S. Zarif, I. Faye and D. Rohaya, Fast and e±cient multichannel image completion
using local similarity, IEEE 21th Inter. Conf. Pattern Recognition (ICPR) (2012),
pp. 3116–3119.
138. S. Zarif, I. Faye and D. Rohaya, Fast and e±cient video completion using object prior
position, Lecture Note Computer Scaince (LNCS), Vol. 8237 (2013), pp. 241–252.
139. S. Zarif, I. Faye and D. Rohaya, Static object removal from video scene using local
similarity, IEEE 9th Int. Colloquium on Signal Processing and its Applications (2013),
pp. 54–57.
140. X. Zhang and T. Chan, Wavelet inpainting by nonlocal total variatio, Inverse Problems
Imaging 4(1) (2010) 1–20.
141. Y. Zhang, Y. Pu, J. Hu and J. Zhou, A class of fractional-order variational image
inpainting models, Appl. Math. Inf. Sci. 6(2) (2012) 299–306.
142. J. Zhou and A. Kelly, Image inpainting based on local optimisation, IEEE Int. Conf.
Pattern Recognition (2010), pp. 4440–4443.
143. H. Zhou and J. Zheng, Adaptive patch size determination for patch-based image com-
pletion, IEEE 17th Int. Conf. Image Processing (2010), pp. 421–424.
Sameh Zarif received his
BSc and MSc degrees in
Information Technology
from Meno¯a University,
Egypt, in 2005 and 2009,
respectively. He is cur-
rently pursuing his PhD
in Computer and Infor-
mation Sciences at the
Centre of Intelligent Sig-
nal & Imaging Research
(CISIR), Universiti Teknologi PETRONAS
(UTP), Malaysia. In addition to his current
research into image inpainting, texture synthe-
sis, and image completion, his interests lie in
image processing, computer vision, and com-
puter graphics.
Ibrahima Faye received
his BSc, MSc, and PhD in
Mathematics from the
University Paul Sabatier
of Toulouse, France and
specialized Master in En-
gineering of Medical and
Biotechnological Data,
from Ecole Centrale Paris.
He is currently an Associ-
ate Professor at the De-
partment of Fundamental and Applied Sciences
of Universiti Teknologi PETRONAS (UTP),
Malaysia. His research interests lie in engineering
mathematics, signal and image processing, mul-
tiresolution analysis, pattern recognition, and
CAD systems for breast and lung cancer detec-
tion. He is a member of IEEE EMBS and the
French Mathematical Society.
Dayang Rohaya re-
ceived her Bachelor of
Science degree in Com-
puter Science from Uni-
versity of Nebraska,
Omaha, USA in 1985 and
her Masters of Science
degree in Computer Sci-
ence from Western
Michigan University in
1987. She received her
Doctor of Philosophy from Loughborough Uni-
versity 2005. Her research specializations are
virtual reality in education and training, human
factors in VR and augmented reality in enter-
tainment and education. Currently she is an
Associate Professor in the Department of Com-
puter and Information Sciences at University
Technology Petronas, Malaysia.
Image Completion
1554001-53
Int. J. Patt. Recogn. Artif. Intell. Downloaded from www.worldscientific.com
by UNIVERSITI TEKNOLOGI PETRONAS (UTP) INFORMATION RESOURCE CENTRE on 03/04/15. For personal use only.