Content uploaded by Muhammad Adeel Azam
Author content
All content in this area was uploaded by Muhammad Adeel Azam on Apr 12, 2022
Content may be subject to copyright.
FOCUS
Smoke removal and image enhancement of laparoscopic images
by an artificial multi-exposure image fusion method
Muhammad Adeel Azam
1,2
•Khan Bahadar Khan
3
•Eid Rehman
4
•Sana Ullah Khan
5
Accepted: 2 March 2022
The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2022
Abstract
In laparoscopic surgery, image quality is often degraded by surgical smoke or by side effects of the illumination system,
such as reflections, specularities, and non-uniform illumination. The degraded images complicate the work of the surgeons
and may lead to errors in image-guided surgery. Existing enhancement algorithms mainly focus on enhancing global image
contrast, overlooking local contrast. Here, we propose a new Patch Adaptive Structure Decomposition utilizing the Multi-
Exposure Fusion technique to enhance the local contrast of laparoscopic images for better visualization. The set of under-
exposure level images is obtained from a single input blurred image by using gamma correction. Spatial linear saturation is
applied to enhance image contrast and to adjust the image saturation. The Multi-Exposure Fusion (MEF) is used on a series
of multi-exposure images to obtain a single clear and smoke-free fused image. MEF is applied by using adaptive structure
decomposition on all image patches. Image entropy based on the texture energy is used to calculate image energy strength.
The texture entropy energy determined the patch size that is useful in the decomposition of image structure. The proposed
method effectively eliminate smoke and enhance the degraded laparoscopic images. The qualitative results showed that the
visual quality of the resultant images is improved and smoke-free. Furthermore, the quantitative scores computed of the
metrics: FADE, Blur, JNBM, and Edge Intensity are significantly improved as compared to other existing methods.
Keywords Artificial multi-exposure fusion Smoke removal Laparoscopic Images Image fusion and enhancement
1 Introduction
Laparoscopic imaging modalities play a significant role in
navigation during operation and treatment planning. Med-
ical surgeons always focus on the quality of images that
determine the best medical decision for the operating
environment (Stoyanov 2012). In laparoscopic surgery, a
small size camera is injected into the human body through
a small incision. All the internal body structural and
functional information can be seen and monitored with the
help of an LCD screen placed in the operation room (Sdiri
et al. 2016). The CO
2
gas is inserted into the human
abdominal area to expand the internal space so that surgical
instruments can be easily operated on. The CO
2
gas and
dissection deformation of tissues produce smoke that cau-
ses the invisibility of organs (Kotwal 2016). The degra-
dation and artifacts in laparoscopic images produce also
due to many other factors such as dynamic homogenous
internal structure, blood flow, dynamic illumination factor,
optical instruments reflection, etc. (Hahn et al. 2017). The
smoke effect during laparoscopic can severely degrade the
Communicated by Jia-Bao Liu.
&Khan Bahadar Khan
kb.khattak@gmail.com
Muhammad Adeel Azam
adeel.azam@iit.it
Eid Rehman
eidrehmanktk@fui.edu.pk
Sana Ullah Khan
sana.ullah@kust.edu.pk
1
Department of Advanced Robotics, Istituto Italiano Di
Tecnologia, Genova, Italy
2
Department of Informatics, Bioengineering, Robotics, and
System Engineering, University of Genoa, Genoa, Italy
3
Department of Telecommunication Engineering, Faculty of
Engineering, The Islamia University of Bahawalpur,
Bahawalpur 63100, Pakistan
4
Department of Software Engineering, Foundation University,
Rawalpindi Campus, Islamabad, Pakistan
5
Institute Institute of Computing, Kohat University of Science
and Technology Kohat (KUST), KPK, Kohat, Pakistan
123
Soft Computing
https://doi.org/10.1007/s00500-022-06990-4(0123456789().,-volV)(0123456789().,-volV)
image quality and also its effects on radiance information
of image patches. The degraded and blurred images could
reduce the visibility of the surgeon for diagnosis and also
increase the probability of error during surgery. The smoke
removal could reduce not only the surgery time but also be
important for surgery planning and treatment. Therefore,
an accurate smoke removal algorithm is required for better
visualization of laparoscopic images (Sdiri et al. 2016;
Hahn et al. 2017; Baid et al. 2017). There are many clinical
applications of laparoscopy images, and it can help to
diagnose multiple diseases at a very early stage (Azam,
et al. 2021).
The smoke removal method is considered as image de-
hazing that existed in literature (Salazar-Colores et al.
2020; Tan 2008a). The image de-hazing algorithms are
classified into three groups (Bansal et al. 2017): image
restoration, image enhancement, and fusion-based methods
(He et al. 2011; Galdran 2018; Nair and Sankaran 2022). In
the image restoration category, the haze-free image is
obtained by using atmospheric degradation methods uti-
lizing prior knowledge of image depth information. The
prior information of hazy image derived first then by
applying physical degradation model to obtain haze-free
images. He et al. (He et al. 2011) proposed Dark Channel
Prior (DCP) technique that is based on the restoration
domain. In the image enhancement domain, there is no
need of using an atmospheric physical model and prior
estimation of depth information in images. In this method,
the correlation algorithms are mostly used to enhance the
local contrast of the images for better visualization (Li
et al. 2018a). In this category, some of the techniques are
the Retinex algorithm (Jobson 2004), histogram equaliza-
tion (Thomas et al. 2011; Yu and Bajaj 2004), and wavelet-
based algorithms (Rong and Jun 2014). In fusion-based
methods (Ancuti and Ancuti 2013), the resultant enhanced
image is obtained by fusing input blurred images (Azam
et al. 2021). However, the required detailed information at
a high level of accuracy in smoke-free images is still a
challenging task. Gamma correction is utilized to split
single input blurry and smoky images into different multi-
exposure images then the MEF technique is implemented
to fuse these multi-exposure images. The image contrast
and saturation are used as image fusion weights during the
fusion process (Ma et al. 2017). MEF techniques are used
for enhancing the visual quality of degraded images. The
advantages and drawbacks of these three domain encap-
sulated in Table 1.
In this article, we proposed a laparoscopic smoke
removal method that removes the smoke effect and also
enhanced the quality of the degraded images. The proposed
method is based on the PASD-MEF technique. The MEF
technique enhanced the local detail information of input
Table 1 The overview of various smoke removal techniques with their strength and limitation
Domain Techniques Advantages Disadvantages
Restoration
methods
Bayesian dehazing (Baid et al. 2017), Fattal et al.
(Fattal 2008), DCP (He et al. 2011), Tan et al.
(Tan 2008b), Tarel et al. (Tarel and Hautie
`re
2009), Deep learning (Fan et al. 2021)
Due to the use of a physical
model, the de-hazing power is
excellent
The image is apparent in thin or
homogenous smoke
Color restoration is excellent,
and the output image is nearly
identical to the original
There is limited work on dense smoke
images
A halo effect and color distortion occur
as the image is over-recovered
Dark colors are exaggerated when
they’re over-saturated
Enhancement
Method
Histogram equalization (Thomas et al. 2011;Yu
and Bajaj 2004),, Retinex (Nair and Sankaran
2022), Wavelet transform (Rong and Jun 2014),
Homographic filtering
Enhance the saturation and
global contrast of images
Compute time is also reduced
compared to other
methodologies in the same
area of study
Suitable for real-time
implementation
Usually neglects the local contrast
information of images
Image visual quality is affected due to
the missing of many local pixels
during the calculation of global
contrast
Fusion-based
methods
Multi-exposure fusion (Ma et al. 2017; Li et al.
2018b,2020) SR Fusion (Baid et al. 2017),
guided filtering fusion (He et al. 2010), Deep
learning fusion, Multi-scale decomposition (Qi
et al. 2020), patch structure decomposition (Li
et al. 2016)
The visual quality of degraded
images is enhanced
Superior performance in terms
of image quality due to
multiple image fusion
Enhanced the local detail of
image patches information
Due to the difficulty in acquiring
images, there are practical issues
Due to larger computation time, these
models cannot be implemented in
real-time
M. A. Azam et al.
123
laparoscopic images. A series of gamma corrections are
used to remove the blurry patches in the images and also
effectively increase the local contrast of the images.
Whereas, Spatial Linear Saturation (SLS) is used to
increase the color saturation of the laparoscopic images.
Then, a set of images with under-exposure levels are
formed. These under-exposure images now have high color
saturation and enhanced contrast but low exposure levels.
The proposed algorithm implemented a patch adaptive
structure (PAS) technique that works on MEF. The
advantage of using PAS and MEF is that they preserved the
structure of laparoscopic images. The significant contri-
bution of the proposed methodology is highlighted as
follows:
•Development of smoke removal self-fusion algorithm
on smoky and blurry input images in a spatial domain.
The smoke effect is removed with the help of contrast
and saturation correction. SLS is implemented to
increase the saturation contrast of images.
•PASD algorithm is proposed for the spatial domain,
MEF to enhance the visual quality of the degraded blur
laparoscopic images. The adaptive selection of different
patched size in images are obtained by using an
implementation of block size and texture energy.
Adaptive selection avoids the error of loss of informa-
tion in both local structure and texture detail informa-
tion of images during the smoke removal procedure.
•The proposed algorithm PASD-MEF is verified both in
a qualitative as well as quantitative manner. The article
demonstrated that the proposed algorithm not only
removes the smoke but also enhances the visual quality
of the laparoscopic image for better visualization and
diagnostic purposes.
•The proposed algorithm is compared with other state-
of-the-art smoke removal methods, and the proposed
method showed significantly improved performance in
terms of visual and statistical evaluation metrics.
The article arrangement is as follows: In Sect. 2, related
works associated with haze and de-smoke are presented
while Sect. 3describes the proposed methodology. In Sect.
4, the quantitative and qualitative results are encapsulated,
and the conclusion is drawn in Sect. 5.
2 Related works
There are many techniques presented in the literature for
de-smoke of laparoscopic images (Sdiri et al. 2016; Hahn
et al. 2017; Baid et al. 2017). A novel Bayesian inference
that consists of a probabilistic graphical technique is
applied on laparoscopic images (Baid et al. 2017). The
model includes a prior model and is implemented on
transmission map images. The transmission map is useful
for color attenuation that is caused by smoke. Then, this
work is extended in Salazar-Colores et al. (2020), to
achieve smoke-free, noiseless, and remove the specular
effect in images. Many other methods in the literature are
related to laparoscopic smoke removal. These techniques
use the atmospheric scattering model and work relatively
the same as the dehazing techniques in the literature. The
atmospheric model depends on the depth of images or the
transmission map (He et al. 2011; Tarel and Hautie
`re 2009;
Zhu et al. 2015). He at el. proposed a DCP technique that
relies on statistical observation and is implemented on
outdoor hazy images (He et al. 2011). In this method, it is
observed that most pixels have very low intensities values
in at minimum single-color channel. In the DCP method, a
prior estimation knowledge of image depth detail and
transmission map is implemented. The density of the
hazing scene acquired and high-quality non-hazy images
are formed. This algorithm not effectively works on out-
door images that have a very high white radiance effect.
However, some other methods do not require the estima-
tion of transmission maps or image depth information. Tan
et al. (2008b) directly enhance the local detail of images
without any use of a transmission map. In (Ancuti and
Ancuti 2013), a fusion-based method is proposed that relies
on white balance phenomena to enhance the input images.
A Laplacian pyramid representation technique is used for
fusion purposes, and this method works on per pixel. The
multi-scale fusion is implemented on hazy images and
derived from a single resultant image. Most of the image
smoke removal methods work as image restoration and
smoke removal. Koschmieder (He et al. 2011; Tarel and
Hautie
`re 2009) proposed an atmospheric scattering
scheme to solve the problem of degradation in images
caused by smoke. This model is described in Eq.1.
IyðÞ¼tyðÞ:JyðÞþA:ð1tðyÞÞ ð1Þ
where I(y) represent the degraded images while J(y) is the
haze-free image. The t(y) denotes the transmission medium
and represents the quantity of light that spreads toward the
target. In the above equation, the Adenotes global atmo-
spheric light. The product of t(y). J(y) represents the scene
radiance. The term A:1tyðÞðÞin Eq. 1denotes the air-
light. Air light produced by smoke dispersion increases the
intensity of the object, which is assumed to be the primary
cause of the color shift of the scene. This term for air light,
especially for thick smoke, would dominate the strength of
the scene. By rearranging the above equation, the haze-free
image J(y) will be achieved. The haze-free image only is
obtained when the value of Aand t(y) is already achieved
using apriori information and from the estimation solution.
Equation 2represents the rearranged form of Eq. (1). The
common limitation J(x) can also be limited by
Smoke removal and image enhancement of laparoscopic images by an artificial multi-exposure image…
123
implementing the maximum local contrast and saturation
or distributing the specific color pixels in RGB space.
JyðÞ¼
IyðÞA
tyðÞ þAð2Þ
The Multi-exposure fusion techniques are also used in
many image processing tasks where different sensors
sequence of images fused to obtain a resultant single image
(Ma et al. 2017). The gamma correction method is widely
used in literature for image enhancement (Li et al. 2016).
The existence of image fusion methods discussed in the
literature are based on sparse representation (Li et al.
2018b,2020), guided filtering techniques (He et al. 2010),
multi-scale decomposition fusion techniques (Qi et al.
2020), patch structure decomposition (Yin et al. 2019), and
multi-exposure image fusion. Galdran introduced multi-
exposure fusion based on Laplacian pyramid fusion (LPF)
for haze removal (Nan et al. 2016), then, in the space
domain, the haze removal is converted to increase image
contrast and saturation effect.
In this paper, we proposed a multi-exposure image
fusion method for smoke removal. The adjustment of
image saturation and contrast is done using gamma cor-
rection to split input images into multiple exposure images.
MEF methods are used for image smoke removal, and it
improved image enhancement. The fusion strategy helps to
manipulate image contrast and saturation that enhance the
visual quality of images. The gamma correction and image
enhancement in our research work in the spatial domain,
histogram equalization is added to gamma correction to
increase the image contrast. Whereas traditional image
enhancement methods are used for global contrast and
saturation transformation of images. In the proposed
methodology, the Adaptive Gamma Correction (AGC)
technique is used to increase the transmission map t(x) that
is used in Eq. (1) by the Koschmieder model. For further
improvement of AGC, we used Laplacian-based solutions.
Contrast adjustment solution integrated with AGC to
remove the blurred effect in images. The detailed
description of the proposed method is discussed in Sect. 3.
3 Proposed methodology
To avoid the estimation effect of atmospheric light and
transmittance described in Eq. (1), the contrast enhance-
ment and saturation adjustment technique in the spatial
domain is suggested to achieve smoke-free laparoscopic
images. According to Koschmieder model, the intensity
range of input blurred images I(y) lies between values 0 to
1. The following condition J(y)BI(y)Vyneeds to satisfy
to obtain a smoke-free image J(y). In this paper, we first
make a set of under-exposed images U={I
1
(y), I
2
(y),
I
3
(y)…...I
k
(y)} from the original smoke input image I(y).
The under-exposed images always reduce the intensity
variation in images. The under-exposure image I(y) inset of
multiple under-exposure images contains high contrast and
saturation but skip small detail structure information. These
under-exposure images now have low exposure levels. We
implemented a MEF technique to fuse all the under-ex-
posed sets of images U={I
1
(y), I
2
(y), I
3
(y)…...I
k
(y)} into a
single image to extract local detail information. The MEF
technique fused different regions of images with good
contrast and saturation level to obtain smoke-free single
image J(y). The flowchart of the proposed methodology is
shown in Fig.1. First, the set of multi-exposure images is
obtained with the help of gamma correction. The linear
adjustment associated with spatial saturation is also
implemented on the image to increase the visual quality.
Gamma correction is implemented for contrast level
adjustment of images. The increase of the contrast of
blurred areas in the images decreased the sharpness level of
that area. To overcome this problem, we utilized a MEF
technique that extracts those corresponding areas from
multiple images and fused them into a single image with
better contrast and saturation. For better fusion, it is
important to maintain texture and color detail as same as
the original image which is achieved by applying MEF
with adaptive structure decomposition (ASD) of the image
patch. In the proposed methodology, the texture informa-
tion components of the image are obtained by using car-
toon texture decomposition (Li et al. 2018c). The image
texture entropy is calculated from the gray difference
technique (Li et al. 2018c). The texture entropy value and
image block size are treated in an image decomposition
block. The overall image block is sub-divided into three
independent components. Each component is processed
individually to give the resultant fused smoke-free image.
The proposed methodology is explained in the following
sections.
3.1 Gamma Correction and Contrast Adjustment
The overall image intensity of degraded image I(y)is
adjusted by using gamma correction and modifying the
intensity of the image by a power function as shown in
Eq. (3).
IyðÞ!b:IðyÞlð3Þ
where the terms band lrepresent the positive constant.
The visual differences are more prominent in the dark areas
as compared to bright areas. The value of lhas chosen less
than one l\1 for compressed bright intensities while it
increases dark intensities in images for better visual detail.
With the value of l[1, more bright intensities are allotted
in a more extensive range after transformation, and dark
M. A. Azam et al.
123
intensities are compressed for that value range. The con-
trast of the image region can be expressed in Eq. (4).
CxðÞ¼Ix
max Ix
min ð4Þ
where Ix
max= max {I(y)|yex} and Ix
min= min {I(y)|yex}.
In Figs. 2e and 3e, the image shows overexposure, and
there is contrast detail information missing in both images.
After applying the l[1 operation, the contrast detail of
the image in Figs. 2g and 3g increases. In our proposed
algorithm, the adjustment of gamma correction is used to
modify the local contrast detail of input images. Gamma
correction also removes the blurred effect in images as
shown in Fig. 4h and 2h. In Figs. 2, 3, different exposure
levels of laparoscopic images are shown. The left side
images are over-exposure images while the move toward
the right side the exposure level of images decreases. The
resultant fused MEF images are shown on the rightmost
side of Figs. 4–2.
3.2 Artificial multi-exposure fusion
After the contrast enhancement, the Spatial Linear Satu-
ration (SLS) is implemented on multi-exposure laparo-
scopic images. The visual quality of images is improved by
using the adjustment of local contrast and brightness of the
images. The sequence of multi-exposure images
U={I
1
(y), I
2
(y), I
3
(y)…… I
k
(y)} from input image I(y)is
obtained with the help of gamma correction. For every
image UR
kyðÞ;UG
kyðÞ;UB
kyðÞ} in the set of multi-exposure,
the minimum and maximum components value of three-
channel R, G, and B can be manipulated by using Eqs. (5)
and (6). When D= (RGBmax -RGBmin)/255 [0, then
the saturation of every pixel can be manipulated by using
Eq. (7).
RGBmax ¼maxðmax R;GðÞ;BÞð5Þ
RGBmin ¼minðmin R;GðÞ;BÞð6Þ
S¼
D
value L\0:5
D
2value L0:5
8
>
<
>
:ð7Þ
The term value and Lcan be defined in Eq. (8). When
the saturation of every pixel value is computed then this
operation is applied on each channel of image RGB
described as in Eq. (9). We have taken the adjustment
range of saturation for an image as [0,100].
Fig. 1 Proposed methodology PASD-MEF framework
Smoke removal and image enhancement of laparoscopic images by an artificial multi-exposure image…
123
Value ¼RGBmax þRGBmin
255 ;where L¼value=2ð8Þ
U0
KyðÞ¼UkyðÞþ UkyðÞL255ðÞbð9Þ
b¼
1
ðS1Þpercent þS1
1
ðpercentÞelse
8
>
>
<
>
>
:
ð10Þ
The final image obtained after the saturation operation
applied on each channel of the image is described in
Eq. (11).
U0
KyðÞ¼ðUR0
kyðÞ;UG0
kyðÞ;UB0
kyðÞÞ ð11Þ
When the image saturation process is completed then
MEF is applied to obtain the local detail information of the
laparoscopic images. The proposed MEF scheme works on
adaptive decomposition based on patch structure. The
adaptive patch of an image determines using image texture
entropy and patch size. The resultant fuse image is
obtained by combining decompose patch images. The
image cartoon decomposition is used for the analysis of
structural information in an image (Li et al. 2018c) while
texture components of the image give detailed information
(Zhu et al. 2016). In the proposed work, the Vese Osher
(VO) model is implemented based on variational image
decomposition (Vese and Osher 2003) to the source ima-
ges. The cartoon-texture decomposition determines by
using Vese Osher (VO) model.
3.3 Adaptive Patch Structure and Image
Intensity
When the texture component is determined, the gray dif-
ference technique is implemented to compute the image
entropy value using texture features. Then adaptive path
size selection of the image is selected. If pixel point is
located at point (x,y) then a point p¼ðDx;DyÞfar away
from pixel point is represented as ðxþDx;yþDyÞ. The
grayscale based on different values can be calculated as in
Eq. (12).
mDx;yðÞ¼mx;yðÞmðxþDx;yþDyÞð12Þ
where m(x,y) denoted grayscale value, and mDx;yðÞrep-
resent the difference in grayscale value. The entropy value
of laparoscopic images can be determined by using
Eq. (13).
E¼X
n
i¼0
piðÞlog2½piðÞ ð13Þ
For complete image texture, the values of entropies can
be calculated in the form of set E={E1, E2, E3……., Ek,},
where E1, E2…... Ek is the entropy value of each image.
Then final entropy value can be calculated by using the
mean of all entropy values represented in Equation (14).
E¼1
KX
k
i¼0
Eið14Þ
The adaptive patch size scheme preserved more detailed
information during the fusion process. The optimal block
size of each image can be calculated by using Eq. (15).
Fig. 2 Multi-exposure laparoscopic images of video 10 with smoke
Level 4. aOver-exposed. bNormal exposed image, cunder-exposed
image. dResultant fused image obtained from images (a–c). eZoom-
in over-exposed image. fZoom-in of normally exposed image.
gZoom-in under-exposed. hZoom-in of the fused image
M. A. Azam et al.
123
Ws¼Ps0:1ðÞx
E
10
E
E
10
E
E
10
Eþ
E
10
E
!
þPsðeEx0:1ðÞð15Þ
And, Wsis image patch size. The optimal block size can
be achieved using the image entropy value. E in the above
equation represents the Entropy value of a given image,
these parameters are set for calculated image patch size.
When the optimal value of Wsachieved then a set of multi-
exposure images decompose into sub-image of Ws x Ws
size blocks. Structure decomposition algorithm (Ma et al.
2017) is implemented on each patch size of the image that
is further divided into the following components: I) Ck,
signal contrast strength II) signal structure strength Sk and
III) mean intensity Ik. These three parameters have pro-
ceeded further to achieve the desired fused image patches
b
X. To obtain an appropriate fused patch image, we need
three desired parameters that are d
Ck;c
Sk;b
I;these parameters
are explain below;
c
Ck= The desired contrast strength in the fused image
was obtained by merging the highest contrast of all source
sets of image patches with the same spatial position.
b
Sk= The desired signal structure fused block can be
calculated by assigning weighted average value to image
block contrast using input structure vector.
b
Ik= To obtain mean intensity components, the global
and local mean intensity of the current source image is used
as an input.
When d
Ck;Sk;b
Ikcomponents are calculated then fused
image patch b
Xobtained, and a new vector can be repre-
sented as shown in Eq. 16. The proposed MEF gives
smoke-free, well-exposed, and high contrast images by
artificially under-exposed/over-exposed images. The
smoke in the image represented in Eq. (1) always reduces
the intensity level of the images. The proposed algorithm
works only on under-exposed images. Furthermore, if the
exposure value increased then gamma correction can adjust
Fig. 3 Sample dataset videos frames (a1–a4) frames of video 1 where
a1 represent level1 smoke and smoke increase from left to right a4
represent dense smoke of level 4 (b1–b4) frames of video 5 (c1–c4)
frames of video 10 (d1–d4) frames extracted from video 15 while
(e1–e4) frames of video 20
Smoke removal and image enhancement of laparoscopic images by an artificial multi-exposure image…
123
Fig. 4 Multi-exposure laparoscopic images of video 1 with smoke
Level 3. aOver-exposed, bNormal exposed image, cunder-exposed
image, dResultant fused image obtained from images (a–c). eZoom-
in over-exposed image. fZoom-in of normally exposed image.
gZoom-in under-exposed. hZoom-in of the fused image
Fig. 5 Qualitative visual results of smoke level 3 laparoscopic images aInput smoke and blur laparoscopic images where b–e images are
resultant smoke-free and enhanced images. bDCP (Tan 2008a), cCAP (Azam et al. 2021), dMPM (Zhu et al. 2016), eProposed method
M. A. Azam et al.
123
the contrast of images and increase the visual quality of
blurred laparoscopic images.
b
X¼c
Ck:Skþb
Ikð16Þ
Multiple image patches of a fused image can be
obtained by sliding the window, the pixels in covering
patches are found the average valuse to give output. At that
point, the fused image is formed by using Eq.17.
JðxÞ¼X
n
i¼1b
xið17Þ
Gray difference technique implemented on gray-level
images to obtain grayscale differential output (Li et al.
2018c). This can be represented in Eq. 18.
Idelta x;yðÞ¼Ix;yðÞIðxþDelta xðÞ;yþDelta yðÞÞ
ð18Þ
Where Ishow image, xand yrepresent image points
location. The point pixel close to (x,y) point repre-
sented by (x?Delta (x), y?Delta (y)). Where Idelta
shows the gray image differential value in the image
I.
4 Experimental results
In this section, the dataset details and the proposed
methodology subjective/qualitative and objective/quanti-
tative results compared with other state-of-the-art tech-
niques such as Dark Channel Prior (DCP) (He et al. 2011),
Multilayer Perceptron Method (MPM) (Sebastia
´n Salazar-
Colores and Cruz-Aceves 2018), Color Attenuation Prior
(CAP) (Zhu et al. 2015) is presented. The proposed method
is implemented on MATLAB 2018a software where the
hardware specification is Intel
Core i3-4010U CPU of
clock speed 1.7GHz and RAM are 4GB.
4.1 Dataset
The dataset taken is a part of the ICIP LVQ Challenge
dataset. That is a collection of a total of 800 distorted
videos created using a set of 20 reference videos, each 10
seconds long (Khan, et al. 2020; Twinanda et al. 2017).
Obtain these videos from the Cholec80 dataset (http://
camma.u-strasbg.fr/datasets). The whole dataset consists of
ten category videos group such that smoke videos, blurry,
white Gaussian noise videos, etc. All videos with a 16:9
aspect ratio have a resolution of 512 by 288 and a 25 fps
frame rate. Screen blending video editing software was
Fig. 6 Qualitative visual results of smoke level 4 laparoscopic images aInput smoke and blur Laparoscopic images where b–eimages are
resultant smoke-free and enhanced images. DCP (Tan 2008a), cCAP (Azam et al. 2021), dMPM (Zhu et al. 2016), eProposed method
Smoke removal and image enhancement of laparoscopic images by an artificial multi-exposure image…
123
used to generate smoke videos. By using the technique, a
smoke video of a black background is mixed with the
reference video so that the original video’s black areas
remain untouched while the smoke region overlays. Four
various degrees of smoke intensity videos are created by
adjusting the strength of the smoke video. The smoke
group videos are a total of 80 in numbers. We collected 25
videos from the ICIP LVQ Challenge dataset among smoke
group videos for this experimentation purpose. Then
frames were extracted with a resolution of images 512 by
288 to test the proposed algorithm.
4.1.1 Qualitative visual results
The visual results of smoke images with level 3 smoke
distortion are shown in Fig. 5while the smoke images with
Table 2 Quantitative/objective
evaluation results of the smoke-
free images
Video ID Smoke frame Method FADE Blur JNBM Edge Intensity
1 Level-3 DCP 0.334 0.257 3.3802 69.124
CAP 0.443 0.261 3.3795 58.767
MPM 0.271 0.253 3.4095 78.458
Proposed 0.176 0.248 3.5073 79.536
Level-4 DCP 0.354 0.263 3.3161 66.767
CAP 0.457 0.265 3.3736 57.458
MPM 0.296 0.257 3.3960 75.598
Proposed 0.189 0.253 3.4551 77.325
5 Level-3 DCP 0.337 0.252 3.0253 68.498
CAP 0.468 0.255 3.1207 51.945
MPM 0.369 0.252 3.1151 66.230
Proposed 0.196 0.246 3.4417 62.743
Level-4 DCP 0.391 0.256 2.8429 65.644
CAP 0.556 0.261 3.1690 49.168
MPM 0.440 0.258 3.0726 62.196
Proposed 0.228 0.251 3.3052 59.926
10 Level-3 DCP 0.263 0.271 2.7363 86.330
CAP 0.385 0.278 2.7743 63.755
MPM 0.278 0.267 2.8444 83.162
Proposed 0.145 0.265 2.8172 85.386
Level-4 DCP 0.276 0.274 2.8540 84.565
CAP 0.402 0.281 2.8672 62.315
MPM 0.308 0.272 2.9426 79.911
Proposed 0.163 0.269 2.8681 81.597
15 Level-3 DCP 0.329 0.270 3.3597 55.406
CAP 0.508 0.278 3.1900 46.009
MPM 0.305 0.260 3.2100 66.943
Proposed 0.197 0.251 3.3964 58.358
Level-4 DCP 0.347 0.282 3.1051 55.445
CAP 0.558 0.291 2.9624 45.261
MPM 0.356 0.276 2.9541 62.988
Proposed 0.220 0.266 3.1330 57.523
20 Level-3 DCP 0.417 0.319 2.5731 38.031
CAP 0.561 0.317 2.5305 37.504
MPM 0.419 0.299 2.6118 47.585
Proposed 0.188 0.288 2.7140 55.808
Level-4 DCP 0.450 0.309 2.5195 37.749
CAP 0.624 0.304 2.4998 37.508
MPM 0.474 0.288 2.4910 46.795
Proposed 0.212 0.276 2.7138 55.012
M. A. Azam et al.
123
level 4 distortion are shown in Fig. 6. It is observed that the
DCP method can remove the smoke effect but the contrast
and saturation balance of images reduces. In the CAP
method, it is noticed that smoke is not well removed, and
an unbalance natural color of images is also seen. While
the MPM method, removes the smoke but local detail
information of laparoscopic images is not visible. The
proposed method not only removes the smoke from images
but also enhanced the local contrast information of the
images and the good saturation color are seen.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
Level-3 Level-4 Level-3 Level-4 Level-3 Level-4 Level-3 Level-4 Level-3 Level-4
1 5 10 15 20
FADE and Blur Metric
(lowest score represents better)
FADE Blur
Fig. 7 Graphical objective evaluation results of FADE and blur metric
0
10
20
30
40
50
60
70
80
90
100
0
0.5
1
1.5
2
2.5
3
3.5
4
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
DCP
CAP
MPM
Proposed
Level-
3
Level-4 Level-3 Level-4 Level-3 Level-4 Level-3 Level-4 Level-3 Level-4
1 5 10 15 20
JNBM and Edge Metrics
(Highest score reprsents better)
JNBM Edge Intensity
Fig. 8 Graphical objective evaluation result of JNBM and Edge intensity metric
Smoke removal and image enhancement of laparoscopic images by an artificial multi-exposure image…
123
4.1.2 Quantitative evaluation
In objective evaluation, we choose non-reference image
quality metrics because reference or any ground truth images
are absent. The evaluation of the proposed method is per-
formed by computing four metrics: FADE, JNBM, Blur, and
Edge intensity. Fog Aware Density Evaluator (FADE)
metric is used for analyzing smoke in the images (Choi et al.
2015). The perceptual fog density in the laparoscopic images
can be computed by computing the FADE metric. If the value
of FADE is lower, then it means that fog density is lower, for
better smoke removal its value should be lower. The JNBM
non-reference metric is based on sharpness and works best
for blurry images (Ferzli and Karam 2009,2006). This metric
evaluates the quantity level of visual sharpness in the images.
The higher value indicated that images are highly sharp and
best for perceptual view. Furthermore, an Edge intensity
metric is implemented, this metric gives information about
the edge intensities that are not visible in source images. The
higher value represented good edge intensity (Hautie
`re et al.
2008). The non-reference blur perceptual metric is used to
analyze blurriness in the image (Crete et al. 2007). Table 2
shows all the statistical results computed by these four non-
reference metrics. The proposed method shows a signifi-
cantly improved result as compared to other state of art
techniques. The bold values indicated better performance
results. The graphical objective evaluation results of smoke
level 3 and level 4 images are shown in Figs. 5,6. The bar-
plot result of FADE, JNBM, Blur, and Edge intensity metrics
is shown in Figs.7,8.
5 Conclusions
The proposed method of PASD-MEF is based on multi-
exposure image fusion. The MEF works on the adaptive
structure decomposition technique. A sequence of under-
exposed images is extracted from the input single smoke
and burry image. The Gamma correction is implemented to
achieve a set of under-exposed images while the SLA
scheme is applied for saturation adjustment. Adaptive
structure decomposition (ASD) is used during the MEF
procedure. The adaptive patch decomposition integrates all
common regions from a series of images that have better
contrast and saturation. Whereas MEF fused these sets of
images into a single de-smoke image. The qualitative, as
well as quantitative results, showed that the proposed
method significantly improves the visual quality of images
and also reduces the smoke from images. The main goal of
this paper is to remove smoke and enhance laparoscopic
images. The improved quality of images is useful in image-
guided surgery and also helpful for surgeons for better
visibility during surgery.
There are a few limitations, Fused image some-time
produces very high edges and due to high edges the global
brightness become a little dark as compared to the original,
This algorithm use PASD as the fusion optimization method.
A real-time implementation of this method is not possible.
The fusion algorithm’s efficiency will be improved in the
future by implementing an effective fusion optimization
algorithm. In the future, geometric data will be evaluated and
scrutinized in better detail to increase fusion performance.
Denoising and other image processing techniques will be
used in the present solution. In further work, we will attempt
to build fusion processes on a high-performance computing
infrastructure capable of handling massive datasets.
Author contributions Conceptualization: MAA, KBK; Investigation:
ER,SUK, Methodology: MAA, Supervision: KBK
Funding This research received no external funding.
Data availability Enquiries about data availability should be directed
to the authors.
Declarations
Conflict of interest The authors declare no conflict of interest.
References
Ancuti CO, Ancuti C (2013) Single image dehazing by multi-scale
fusion. IEEE Trans Image Process 22(8):3271–3282. https://doi.
org/10.1109/TIP.2013.2262284
Azam MA, Khan KB, Ahmad M, Mazzara M (2021) Multimodal
medical image registration and fusion for quality enhancement.
Comput Mater Contin 68(1):821–840. https://doi.org/10.32604/
cmc.2021.016131
Azam MA et al (2021) Deep learning applied to white light and
narrow band imaging videolaryngoscopy: toward real-time
laryngeal cancer detection. Laryngoscope. https://doi.org/10.
1002/lary.29960
Baid A, Kotwal A, Bhalodia R, Merchant SN, Awate SP (2017) Joint
desmoking, specularity removal, and denoising of laparoscopy
images via graphical models and Bayesian inference. Proc Int
Sympos Biomed Imaging. https://doi.org/10.1109/ISBI.2017.
7950623
Bansal B, Singh Sidhu J, Jyoti K (2017) A review of image restoration
based image Defogging algorithms. Int J Image Graph Signal
Process 9(11):62–74. https://doi.org/10.5815/ijigsp.2017.11.07
Choi LK, You J, Bovik AC (2015) Referenceless prediction of
perceptual fog density and perceptual image defogging. IEEE
Trans Image Process 24(11):3888–3901. https://doi.org/10.1109/
TIP.2015.2456502
Crete F, Dolmiere T, Ladret P, Nicolas M (2007) The blur effect:
perception and estimation with a new no-reference perceptual
blur metric. Hum vis Electron Imaging XII 6492:64920I. https://
doi.org/10.1117/12.702790
Fan Y, Chen R, Li Y, Zhang T (2021) Deep neural de-raining model
based on dynamic fusion of multiple vision tasks. Soft Comput
25(3):2221–2235. https://doi.org/10.1007/s00500-020-05291-y
M. A. Azam et al.
123
Fattal R (2008) Single image dehazing. ACM Trans Graph. https://
doi.org/10.1145/1360612.1360671
Ferzli R, Karam LJ (2006) A no-reference objective image sharpness
metric based on just-noticeable blur and probability summation.
Proc Int Conf Image Process ICIP 3:445–448. https://doi.org/10.
1109/ICIP.2007.4379342
Ferzli R, Karam LJ (2009) A no-reference objective image sharpness
metric based on the notion of Just Noticeable Blur (JNB). IEEE
Trans Image Process 18(4):717–728. https://doi.org/10.1109/
TIP.2008.2011760
Galdran A (2018) Image dehazing by artificial multiple-exposure
image fusion. Signal Process 149:135–147. https://doi.org/10.
1016/j.sigpro.2018.03.008
Hahn KY, Kang DW, Azman ZAM, Kim SY, Kim SH (2017)
Removal of hazardous surgical smoke using a built-in-filter
trocar: a study in laparoscopic rectal resection. Surg Laparosc
Endosc Percutaneous Tech 27(5):341–345. https://doi.org/10.
1097/SLE.0000000000000459
Hautie
`re N, Tarel JP, Aubert D, Dumont E
´(2008) Blind contrast
enhancement assessment by gradient ratioing at visible edges.
Image Anal Stereol 27(2):87–95. https://doi.org/10.5566/ias.v27.
p87-95
He K, Sun J, Tang X (2010) ECCV2010—guided image filtering.
Eccv 2010:1–14
He K, Sun J, Tang X (2011) Single image haze removal using dark
channel prior. IEEE Trans Pattern Anal Mach Intell
33(12):2341–2353. https://doi.org/10.1109/TPAMI.2010.168
Jobson DJ (2004) Retinex processing for automatic image enhance-
ment. J Electron Imaging 13(1):100. https://doi.org/10.1117/1.
1636183
Khan ZA et al (2020) Towards a video quality assessment based
framework for enhancement of laparoscopic videos. Electr Eng
Syst Sci. https://doi.org/10.1117/12.2549266
Kotwal A (2016) Joint desmoking and denoising of laparoscopy
images Department of Electrical Engineering Indian Institute of
Technology (IIT) Bombay Department of Computer Science and
Engineering Indian Institute of Technology (IIT) Bombay,
pp. 1050–1054
Li H, Qiu H, Yu Z, Zhang Y (2016) Infrared and visible image fusion
scheme based on NSCT and low-level visual features. Infrared
Phys Technol 76:174–184. https://doi.org/10.1016/j.infrared.
2016.02.005
Li Y, Miao Q, Liu R, Song J, Quan Y, Huang Y (2018a) A multi-scale
fusion scheme based on haze-relevant features for single image
dehazing. Neurocomputing 283:73–86. https://doi.org/10.1016/j.
neucom.2017.12.046
Li H, He X, Tao D, Tang Y, Wang R (2018b) Joint medical image
fusion, denoising and enhancement via discriminative low-rank
sparse dictionaries learning. Pattern Recognit 79:130–146.
https://doi.org/10.1016/j.patcog.2018.02.005
Li Y et al (2018c) A novel multi-exposure image fusion method based
on adaptive patch structure. Entropy 20(12):1–17. https://doi.
org/10.3390/e20120935
Li H, Wang Y, Yang Z, Wang R, Li X, Tao D (2020) Discriminative
dictionary learning-based multiple component decomposition for
detail-preserving noisy image fusion. IEEE Trans Instrum Meas
69(4):1082–1102. https://doi.org/10.1109/TIM.2019.2912239
Ma K, Li H, Yong H, Wang Z, Meng D, Zhang L (2017) Robust
multi-exposure image fusion: a structural patch decomposition
approach. IEEE Trans Image Process 26(5):2519–2532. https://
doi.org/10.1109/TIP.2017.2671921
Nair D, Sankaran P (2022) Benchmarking single image dehazing
methods. SN Comput Sci. https://doi.org/10.1007/s42979-021-
00925-w
Nan D, Bi DY, He LY, Ma SP, Fan ZL (2016) A variational
framework for single image dehazing based on restoration. KSII
Trans Internet Inf Syst 10(3):1182–1194. https://doi.org/10.
3837/tiis.2016.03.013
Qi G, Chang L, Luo Y, Chen Y, Zhu Z, Wang S (2020) A precise
multi-exposure image fusion method based on low-level fea-
tures. Sensors (switzerland) 20(6):1–18. https://doi.org/10.3390/
s20061597
Rong Z, Jun WL (2014) Improved wavelet transform algorithm for
single image dehazing. Optik (stuttg) 125(13):3064–3066.
https://doi.org/10.1016/j.ijleo.2013.12.077
Salazar-Colores S, Cruz-Aceves I (2018) Single image dehazing
using a multilayer perceptron. J Electron Imaging 27(4):043022
Salazar-Colores S, Alberto-Moreno H, Ortiz-Echeverri CJ, Flores G
(2020) Desmoking laparoscopy surgery images using an image-
to-image translation guided by an embedded dark channel.
pp. 1–9. http://arxiv.org/abs/2004.08947.
Sdiri B, Beghdadi A, Cheikh FA, Pedersen M, Elle OJ (2016) ‘‘An
adaptive contrast enhancement method for stereo endoscopic
images combining binocular just noticeable difference model
and depth information. IST Int Sympos Electron Imaging Sci
Technol. https://doi.org/10.2352/ISSN.2470-1173.2016.13.
IQSP-212
Stoyanov D (2012) Surgical vision. Ann Biomed Eng 40(2):332–345.
https://doi.org/10.1007/s10439-011-0441-z
Tan RT (2008a) Visibility in bad weather. Comput vis Pattern Recogn
CVPR 2008:1–8
Tan RT (2008b) Visibility in bad weather from a single image. 26th
IEEE Conf Comput vis Pattern Recognit CVPR. https://doi.org/
10.1109/CVPR.2008.4587643
Tarel JP, Hautie
`re N (2009) Fast visibility restoration from a single
color or gray level image. Proc IEEE Int Conf Comput vis
2009:2201–2208. https://doi.org/10.1109/ICCV.2009.5459251
Thomas G, Flores-Tapia D, Pistorius S (2011) Histogram specifica-
tion: a fast and flexible method to process digital images. IEEE
Trans Instrum Meas 60(5):1565–1578. https://doi.org/10.1109/
TIM.2010.2089110
Twinanda AP, Shehata S, Mutter D, Marescaux J, De Mathelin M,
Padoy N (2017) EndoNet: a deep architecture for recognition
tasks on laparoscopic videos. IEEE Trans Med Imaging
36(1):86–97. https://doi.org/10.1109/TMI.2016.2593957
Vese LA, Osher SJ (2003) Modeling textures with total variation
minimization and oscillating patterns in image processing. J Sci
Comput 19(1–3):553–572. https://doi.org/10.1023/A:
1025384832106
Yin L, Zheng M, Qi G, Zhu Z, Jin F, Sim J (2019) A novel image
fusion framework based on sparse representation and pulse
coupled neural network. IEEE Access 7:98290–98305. https://
doi.org/10.1109/ACCESS.2019.2929303
Yu Z, Bajaj C (2004) A fast and adaptive method for image contrast
enhancement. Proc Int Conf Image Process ICIP 5:1001–1004.
https://doi.org/10.1109/icip.2004.1419470
Zhu Q, Mai J, Shao L (2015) A fast single image haze removal
algorithm using color attenuation prior. IEEE Trans Image
Process 24(11):3522–3533. https://doi.org/10.1109/TIP.2015.
2446191
Zhu Z, Chai Y, Yin H, Li Y, Liu Z (2016) A novel dictionary learning
approach for multi-modality medical image fusion. Neurocom-
puting 214:471–482. https://doi.org/10.1016/j.neucom.2016.06.
036
Publisher’s Note Springer Nature remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.
Smoke removal and image enhancement of laparoscopic images by an artificial multi-exposure image…
123
