Content uploaded by Jacob Scharcanski
Author content
All content in this area was uploaded by Jacob Scharcanski on Nov 07, 2017
Content may be subject to copyright.
Macroscopic Pigmented Skin Lesion Pre-screening
Eliezer Bernarta, E. S. Floresa, Jacob Scharcanskia,∗
aInstituto de Inform´atica, Universidade Federal do Rio Grande do Sul.
Av. Bento Gon¸calves, 9500, Porto Alegre, RS - Brazil
Abstract
Melanoma is one of the deadliest types of pigmented skin lesions, and if identified
in the earlier stages can increase the patient survival rate. The use of digital
cameras as an alternative to other devices, such as dermatoscope, is gaining
space in skin lesion pre-screening with e-health systems used in the macroscopic
diagnose of pigmented skin lesion images. The traditional framework used to
classify macroscopic pigmented skin lesion images consists in a pre-processing
step to remove hair and shading effects, followed by the lesion area detection
and segmentation. Next, techniques are used to extract a set of features from
the obtained region, and these attributes make it possible to distinguish between
malignant and benign cases. Usually, the features extracted from data labeled
by a specialist, and are used to train a machine learning algorithm, which is
then used to suggest a diagnosis for an undiagnosed skin lesion image. In this
work, we present a review of some of the most recent advances in macroscopic
pigmented skin lesion segmentation and classification.
Keywords: Melanocytic lesion, Macroscopy, Segmentation, Classification
1. Introduction1
Driven by the expansion of the Internet of Things era and the advances2
obtained in technologies related to signal and image processing, e-health sys-3
∗Corresponding author
Email addresses: eliezer.bernart@inf.ufrgs.br (Eliezer Bernart),
esflores@inf.ufrgs.br (E. S. Flores), jacobs@inf.ufrgs.br (Jacob Scharcanski)
Preprint submitted to Elsevier April 24, 2017
Published in 'Encyclopedia of Biomedical Engineering', Elsevier
https://doi.org/10.1016/B978-0-12-801238-3.99956-2
tems are playing an important role by offering the population fast medical pre-4
assistance, contributing to the identification of several possible diseases even in5
their earlier stages.6
This is the case of pigmented skin lesions, a type of disease that begins in7
melanocytes, the cells responsible for producing the melanin pigment that colors8
the skin. In its different forms, pigmented skin lesions affect a large number of9
people around the world, especially in countries where people present light skin10
tones, and it can be caused by different reasons. Some of the main factors that11
contribute to the occurrence of such skin disease are the sun exposure and family12
history. Pigmented skin lesions can be grouped (roughly) in two main classes:13
malignant (melanomas) and benign (e.g., nevi or moles).14
The malignant form of a pigmented skin lesion is considered one of the15
deadliest types of cancer, killing an estimated number of 10,130 people in the16
United States annually based on data provided by the Skin Cancer Foundation.17
Therefore, the early identification of malignant cases significantly increase the18
survival chances of a patient.19
To better understand e-health systems in the context of pre-screening skin20
lesions, it is useful to overview how dermatologists diagnose skin lesions in the21
clinical practice. The most common approach consists of using a dermatoscope22
to obtain a digital image of the lesion area in the patient skin, after the removal23
of hair near the region and the application of gel. To use such device the24
specialist needs a special training, and for this reason, dermatoscopy is used25
mostly by trained dermatologists.26
In order to bring such evaluations to primary health care, where a dermatol-27
ogist or a trained physician is often not available to proceed in the traditional28
way, other alternatives of imaging devices can be considered. The most common29
choice is the use of digital cameras, which allows capturing an image of the skin30
lesion even with mobile devices.31
Images obtained with such devices are named macroscopic pigmented skin32
lesion (MPSL) images and are the main focus in this work. E-health systems33
capable of handling this category of imagery have to be able to treat problems,34
2
such as hair and illumination problems and offer a reliable remote diagnosis.35
The development of reliable techniques for skin lesion diagnosis still is an36
open problem in the medical imaging field, either using macroscopy or der-37
moscopy.38
The remainder of this work is organized as follows. Section 2 gives an39
overview of the typical framework used in the pre-screening of a MPSL image.40
Section 3 presents a series of methods used for pre-processing the MPSL input41
images in order to reduce some of the artifacts and other imaging problems.42
After that, in Section 4, a set of state-of-the-art methods for segmenting the43
MPSL image are presented and discussed. Additionally, Section 5 presents the44
current advances in feature extraction and approaches to discriminate between45
malignant and benignant cases.46
2. Overview of Skin Macroscopy47
Digital dermoscopy is a non-invasive technique designed to assist dermatol-48
ogists in diagnosing pigmented skin lesions. Dermoscopy images are acquired49
through a digital dermatoscope, which also acts as a filter and magnifier. Der-50
moscopy images typically present low levels of noise and consistent background51
illumination. However, the lack of training makes the use of a digital dermato-52
scope challenging even among dermatologists.53
Recently, macroscopy has been attracting the attention of researchers as54
an alternative non-invasive procedure for pre-screening pigmented skin lesions.55
Macroscopy images are commonly known as macroscopic images, and may be56
obtained using digital cameras. The characteristics of these digital cameras may57
vary from device to device, going from embedded digital cameras in smartphones58
to more sophisticated acquisition systems.59
The analysis of macroscopic images represents a very challenging task be-60
cause images are usually acquired from different distances and with a variety61
of illumination conditions. Furthermore, these images may have poor resolu-62
tion and the presence of artifacts such as hair, reflections, and shadows. On63
3
the other hand, macroscopy may allow the development of systems to classify64
skin lesions using simple photographs. Systems for establishing a priority or-65
der for specialized consultation in remote areas and smartphone applications66
to detect malignant cases of melanocytic skin lesions are examples of emerging67
applications that shall advance macroscopy.68
Macroscopy based systems can be divided into four steps: (a) pre-processing,69
where the input image is processed to facilitate the skin lesion segmentation;70
(b) segmentation, where the goal is to delimit the skin lesion region in the input71
image; (c) feature extraction, where the segmented skin lesion is represented by72
its features; and (d) classification, where the skin lesion is classified as benign or73
malignant based on the lesion features extracted. Figure 1 provides an overview74
of such systems.75
Segmentation
Segmentation
Pre-Processing
Pre-Processing Feature Extraction
Feature Extraction Classication
Classication
Benign or Malignant?
Figure 1: Typical framework for macroscopic pigmented skin lesion pre-screening using as
input an MSPL image.
3. Image Acquisition and Pre-Processing
76
The very first step for processing pigmented skin lesion images is the acqui-77
sition phase, where digital cameras are used to capture a digital representation78
of the area affected by the lesion in the patient skin.79
Images acquired by a dermatoscope allow the specialists to obtain an aug-80
mented view of the lesion area illuminated by a non-polarized light, improving81
the visualization of the affected region, capturing fine details of the skin lesion.82
This field is known as dermoscopy and several works in literature discuss how83
to handle this category of imagery, going from the pre-processing steps to the84
identification of malignant cases.85
4
On the other hand, the use and quality of images obtained by digital cameras86
have increased significantly in the recent years, making it possible to use them87
to support the dermatologist diagnosis and in patients follow-ups. Such images88
give a macro view of the lesion area, hence the name, macroscopy.89
Using macroscopy, different classes of problems may affect the visualization90
quality, making it more difficult to analyze and recognize specific patterns in91
pigmented skin lesion images. Problems like hair covering or shading over the92
skin region of interest can affect the segmentation step, confusing the algorithms93
that usually associate dark colors with the lesion area.94
To solve, or at least attenuate these issues in MPSL images, Section 3.195
presents approaches for shading attenuation, and Section 3.2 presents some of96
the existing pre-processing methods for hair removal.97
3.1. Shading Attenuation98
Differently from the images obtained with a dermatoscope, where digital99
cameras are used to capture skin lesions, very often macroscopic images can100
be affected by illumination problems. MPSL images can be captured without101
restricting the light sources, in most cases, under ambient light. The use of102
such uncontrolled illumination source combined with the skin reflectance tends103
to lead to a common problem in MPSL images, the shading effect.104
These shaded healthy skin regions can mislead algorithms that consider the105
darker regions in the MPSL image as being part of the lesion area during the106
segmentation step. Such segmentation errors can affect feature extraction, and107
hence, the final lesion classification result.108
In order to correct, or at least attenuate, the shadow regions affecting the109
segmentation quality, pre-processing algorithms that correct these illumination110
variations have been developed specifically for MPSL images, making it possible111
to create more robust approaches for melanoma diagnosis with MPSL images112
obtained under different lighting conditions.113
Some methods for shadding attenuation in human skin color images rely on114
the assumption that the human body can be represented by curved surfaces,115
5
what introduce smooth dark regions depending on the camera point of view.116
Once obtained, this illumination variation can be used to relight the original117
image.118
For instance, the channel V(value) from the HSV color space is selected119
because it contains more information about the shading in the image. Using120
the pixels in the Vchannel within four windows in the image corners to build121
a set S, the following quadratic function is adjusted:122
z(r, c) = P1r2+P2c2+P3rc +P4r+P5c+P6, (1)
where (r, c) is the pixel location and the quadratic parameters Piare minimizing
123
the error ǫ:124
ǫ=X
(r,c)∈S
[V(r, c)−z(r, c)]2. (2)
Dividing each pixel of the original Vchannel by the correspondent pixel of
125
the new estimated illumination channel z, a new Vvalue can be obtained. The126
old Vchannel is then replaced by the corrected V, and the image is converted127
back from the HSV to the RGB original color space. An important remark about128
such method is that the four corners must contain only normal skin pixels (i.e.,129
do not contain lesion pixels).130
The Multistage Illumination Modeling (MSIM) algorithm is another example131
of approach that can be used to correct the variation existent in MPSL images.132
The method starts with an approximation of the illumination map generated133
using a non-parametric modeling approach based on Monte Carlo sampling.134
After that, the final illumination map estimated is generated using parametric135
modeling, combined with the non-parametric map previously found considered136
as prior. At the end, the variation in illumination presented by the MPSL image137
is removed.138
Also, more complex illumination modeling and chromophore identification139
techniques to adjust the lighting variation in the MPSL images can be found, and140
6
such methods often rely on an adaptive bilateral decomposition and a weighted141
polynomial curve fitting.142
As one could observe, several advances were obtained to remove the effects143
of shading in MPSL imagery, however, building a pre-processing system capable144
of doing light correction that generalizes well in any environment is challenging.145
3.2. Hair Removal146
The existence of hair covering certain regions in MPSL images can affect the147
segmentation process and also the analysis of textural information.148
In some processes, such as in segmentation, the use of morphological oper-149
ators and the procedures for artifacts removal in the post-processing step can150
eliminate the hair affecting the lesion border. However, when it is necessary to151
extract information that goes beyond the lesion boundaries, hair removal as a152
pre-processing step becomes very important.153
Dullrazor R
is a pre-processing method proposed in the 90’s for removing154
hair in images of pigmented skin lesions. This technique follows a three-step155
methodology. First, hair is located using the closing morphological operator,156
and then hair pixels are replaced by neighboring pixels considered non-hair157
using interpolation, and smoothing the final resulting image. This technique is158
efficient for hair removal, however it can introduce a certain amount of noise in159
the output image.160
More recent approaches can be based on threshold decomposition. The161
algorithm converts the image to a luminance threshold-set representation, and162
after that, hair is located using a morphological gap detection algorithm. All163
hair candidates found are merged in a single mask and filtered to remove non-164
hair pixels, the remaining of the pixels (i.e., true hair) are removed using typical165
image inpainting algorithms.166
Even some of these approaches being mainly designed for dermoscopy, they167
still are effective when applied to MPSL images. Figure 2 presents a pre-168
processing example, where the Dullrazor R
method is employed to hair removal169
in a MPSL image shading attenuated. It is important to mention that a method17 0
7
that can remove very dense hair regions in MPSL images, and at the same time171
can preserve local textural information, still is challenging researchers.172
(a)
(b) (c)
Figure 2: Pre-processing example: (a) Original MPSL image; (b) Shading attenuated MPSL
image; (c) Shading attenuated MPSL image pre-processed by a hair removal technique.
4. Image Segmentation
173
After applying the pre-processing approaches mentioned in Section 3, the174
next step is detecting the lesion area in the image. The goal here is to use175
segmentation methods to automatically differentiate between healthy skin areas176
and regions affected by the lesion. This differentiation is usually represented by177
a binary mask, which should preserve the lesion rim characteristics and allow178
the extraction of meaningful features, further discussed in Section 5.179
In the remaining of this Section, some of the most recent representative180
approaches for MPSL image segmentation in the literature are outlined, and181
grouped in different categories: threshold and sparse representation based ap-182
proaches.183
8
4.1. Threshold Based Approaches184
In threshold based segmentation, each pixel intensity I(r, c) of a gray scale185
image is compared with a threshold ˆ
tand a binary mask M, which represents186
the segmentation result, is generated as follows:187
M(r, c) =
1, if I(r, c)<ˆ
t.
0, otherwise.
(3)
The goal of thresholding methods is to determine the threshold ˆ
tused in
188
Equation 3.189
The approaches for MPSL segmentation based in thresholding methods use190
as a constraint the fact that skin lesion regions are a type of skin depigmenta-191
tion, hence, the unhealthy pixels in the image present a darker skin color when192
compared to the healthy pixels. Different methods in the literature explore this193
constraint, some of them are discussed next.194
4.1.1. Otsu’s Thresholding195
Back in 1975, Nobuyuki Otsu proposed an adaptive thresholding method196
that has been one of the most widely used approaches for binary image segmen-197
tation. Over the years, Otsu’s Thresholding gained space in MPSL segmentation198
due to its compromise between low complexity cost and efficiency.199
Otsu’s algorithm main idea is that we can estimate an optimal thresholding200
value ˆ
tthat maximizes the separability between the two classes without using201
any prior, only looking at the image histogram. In the case of MPSL images,202
the classes are healthy skin (light color) and lesion (dark color). The Otsu’s203
threshold is the optimal threshold if the image has a bimodal distribution of its204
gray levels (i.e., if the image has a bimodal histogram).205
Let h(t) = nt/g be the histogram of an image Iwith ℓdifferent gray levels,206
where ntis the number of pixels with the gray level t, and npthe total number207
of pixels in the image. The Otsu’s threshold is the gray level ˆ
tthat maximizes208
9
the inter-class variance (or equivalently minimizes the intra-class variance):209
ˆ
t= arg max
1≤t<ℓ [µgω(t)−µ(t)]2
ω(t)[1 −ω(t)] , (4)
where µ(t) = Pt
i=1 ih(i) denotes the mean level of the first class, µg=Pℓ
i=1 ih(i)210
denotes the mean level of the whole image and ω(t) = Pt
i=1 h(i) denotes the211
cumulative distribution of the histogram up to the gray level t.212
There are many different ways to apply Otsu’s algorithm in MPSL seg-213
mentation, from choosing an appropriate color or representation channel to214
proposing modifications in the original algorithm to overcome problems such as215
over-segmentation and under-segmentation.216
In order to segment the affected regions, different frameworks use adaptive217
thresholding in gray scale images, while others explore the R channel from218
RGB color space, assuming that pigmented skin lesions can be better identified219
in this channel. Also, it has been proposed the use of Otsu’s thresholding in220
channels representing others aspects of the macroscopic images, such as textural221
variability, skin darkness information and color information attracted attention.222
4.1.2. Weighted Thresholding223
The Simple Weighted Otsu Thresholding (SWOT) technique is based on the224
Otsu’s thresholding method, and was designed specifically for MPSL segmenta-225
tion. The method starts by searching for the rectangular-shaped region R∗with226
its sides parallel to the outer borders of the image (e.g., the region delimited by227
black lines shown in Figure 3) that minimizes the total coefficient variation:228
T(R) = X
ch∈{L,a,b}
σch(R)
µch(R), (5)
10
Figure 3: Rectangular-shape R∗delimited by black lines, cross-diagonals delimited by blue
lines and the final segmentation obtained by the SWOT method in white.
where µch(R) and σch (R) denote, respectively, the mean and standard deviation
229
of the ch-th channel of the CIE L*a*b* color space in the region R.230
The search starts from the outer border and shrinks the search area up to a231
percentage (e.g., 25%) of the distance to the smaller side of the image, moving232
in non-overlapping steps of size gL, where gis a constant (e.g., 0.02) and Lis233
the size of the larger side of the image.234
The intensity image used for thresholding is then computed as follows:235
I(r, c) = p(L(r, c)−LR∗)2+ (a(r, c)−aR∗)2+ (b(r, c)−bR∗)2, (6)
where L(r, c), a(r, c) and b(r, c) denote the intensities of the Lab channels, and236
LR∗,aR∗and bR∗denote the medians of these channels in the region R∗.237
Finally, the threshold used for segment Iis obtained by:238
ˆ
t=αth+ (1 −α)ts, (7)
where this a threshold computed by modifying Otsu’s functional to be more
239
robust to imbalances and tsis a threshold estimated from R∗.240
11
The threshold this given by:241
th= arg max
1≤t<ℓ σ2
1(t)
φ1
+σ2
2(t)
φ2, (8)
where σ2
1(t) denotes the inter-class variance of all intensity pixels, and σ2
2(t) de-
242
notes the inter-class variance of a subset of pixels located in the cross-diagonals243
of the image (e.g., the region delimited by blue lines shown in Figure 3), ℓ244
denotes the number of intensities and the normalizing constants φkare given245
by:246
φk=q(σ2
k(1))2+ (σ2
k(2))2+. . . + (σ2
k(ℓ))2. (9)
The other threshold, ts, is given by:
247
ts= Γ0.05 +β·(Γ0.50 −Γ0.05), (10)
where Γ0.05 and Γ0.50 are the 5th and 50th percentiles of the intensity image in
248
the region R∗.249
The parameters αand βadd flexibility to the thresholding process and can250
be obtained using a separate training set.251
4.1.3. Thresholding Ensemble252
Another approach for obtaining a segmentation mask of the skin lesion re-253
gion consists in ensembling different segmentation masks obtained by adaptive254
thresholding methods in an unsupervised way to identify and refine “healthy255
skin”, “skin lesion” and “suspicious regions” areas.256
We can use as an example a method that has as input a MPSL image Iof di-257
mensions nr×nc, represented in the RGB color space, such that Ich(r, c)∈[0,1]258
is the pixel intensity value at coordinate (r, c) for the channel ch ∈ {R, G, B}.259
After a pre-processing step using a standard Gaussian filter with σ(e.g.,260
σ= 0.5) and µ= 0 computed in w×wwindows (e.g., w= 5) to reduce the261
presence of noise, for each one of the ch color channels is obtained a thresholding262
value tch using the Otsu’s thresholding algorithm. Assuming that in RGB color263
12
space the lesion area is darker than the healthy pixels, the binary masks Mch
264
can be computed as follows:265
Mch(r, c) =
1,if Ich(r, c)< tch,
0,otherwise.
(11)
If the values at coordinates (r, c) in all the thresholded channels Mch indi-
266
cate unanimous agreement, i.e., are dark or light with respect to tch, they are267
respectively labeled as “skin lesion” or “healthy skin” regions. In regions where268
there was no total agreement among the initial segmentation masks, the pixels269
are assigned to a region denominated as “suspicious region”.270
This initial process generates an uncertainty map ψ(r, c), depicted in Fig-271
ure 4, composed by three disjoint regions: “skin lesion”, containing the group272
of pixels labeled with high confidence to represent lesion areas; “healthy skin”,273
containing the pixels labeled with high confidence to represent healthy skin ar-274
eas; and “suspicious regions”, that could contain lesion regions, where lesion275
retracted beneath the skin or even present darker skin tones. In the next steps,276
sets of these pixels identified as “suspicious region” will be further refined.277
Figure 4: The first line shows shading attenuated MPSL images. The second line shows the
uncertainty maps ψobtained by the thresholding ensemble approach. In black, are the regions
identified as “skin lesion”; in gray, the regions identified as “healthy skin”; and in white, the
“suspicious regions”.
The desired output for this image segmentation approach is a binary mask.278
In order to achieve that, it is necessary to re-classify the suspicious pixels as one279
13
of the other two classes. The intuition used is that different skin regions can be280
modeled by different stochastic textures, assuming that asymmetric distribu-281
tions, such as Gamma and Rayleigh, represent the gradient magnitude in skin282
areas (i.e., the gradient magnitude distributions are less random in skin lesions283
than in healthy skin).284
We can compute the magnitude of the gradient for each channel ch using the285
Sobel operator Gch =|∇Ich |, and a randomness index in 7 ×7 sliding windows286
is estimated by:287
Kch(r, c) = σGch (r, c)
µGch (r, c)−1
,(12)
where µGch (r, c) and σGch(r, c) denote, respectively, the mean and standard devi-288
ation of Gch in the sliding window centered at (r, c). Larger values for Kch(r, c)289
indicate less randomness at this location. The index that describes the stochas-290
tic property of the texture is associated with the RGB color space information291
to create a feature vector for each one of the image pixels (r, c):292
X(r, c) = {IR(r, c), IG(r, c), IB(r, c), KR(r, c), KG(r, c), KB(r, c)}.(13)
Next, the same number of pixels associated with “skin lesion” and “healthy
293
skin” regions are randomly selected from Xto train a k-Nearest Neighbors294
(k-NN) classifier. The way these samples were initially labeled leads to an295
unsupervised classification approach, even making use of a supervised classifier.296
As final step, the trained classifier is used to refine the pixels that are part of297
the “suspicious regions”, generating a final binary mask containing only two298
disjoint regions: “skin lesion” and “healthy skin”.299
As a post-processing step, the method also selects only the largest lesion300
area in the image and labels all the pixels within this region as “skin lesion”,301
assuming that there is only one lesion per macroscopic image. To conclude, a302
Gaussian filter is applied to smooth the border regions and reduce the noise303
introduced by the refinement in a pixel level, resulting in the final segmentation304
mask.305
14
The reader may note in this ensemble approach that not only methods based306
on thresholding could be used as initialization, but any binary mask previously307
obtained, leading to a robust framework for combining automatically generated308
segmentation.309
4.2. Sparse Representation Based Approaches310
Several approaches for MPSL segmentation aim to obtain a sparse image311
representation, where the “skin lesion” regions and the “healthy skin” regions312
can be better differentiated. In these approaches, the input image is first repre-313
sented by a matrix Y∈Rm×n, where each column yi∈Rmrepresents a small314
image region (e.g., a patch).315
Sparse representations (i.e., sparse texture models) learn a small number of316
texture models (e.g., texture patches) from Yto characterize the entire image.317
There are different ways to learn the sparse model, including clustering or by318
an optimization problem formulation.319
A new trend to segment MPSL is to learn a sparse texture model by employ-320
ing a dictionary-learning algorithm, where a dictionary D∈Rk×mis learned321
from Yand the projections xi∈Rkof the image regions yiin Dare divided322
(i.e., segmented) into two disjoint groups: “skin lesion” regions and “healthy323
skin” regions.324
The main differences between sparse representation based approaches for325
MPSL segmentation are: 1) the number of descriptors per image region mand326
the number of image regions n; 2) the sparse representation model; and 3)327
the segmentation method. Next, we describe some recent sparse representation328
approaches for MPSL segmentation.329
4.2.1. Texture Distinctiveness Lesion Segmentation330
The Texture Distinctiveness Lesion Segmentation (TDLS) is a metric that331
uses the concepts of sparse representation to segment MPSL. In this method,332
the input RGB image corrected by a shadding attenuation algorithm is first333
converted to the XYZ color space. Then, for each pixel location (r, c), a vector334
15
yi(column of Y) is used to represent the w×w×3 patch centered at (r, c).335
Thus, m= 3w2and n=nr ·nc, with nr and nc denoting, respectively, the336
numbers of rows and columns in the input image.337
Next, a Gaussian Mixture Model (GMM), with parameters {µj}k
j=1,{Σj}k
j=1
338
and {αj}k
j=1, is obtained by the Expectation Maximization (EM) algorithm,
339
which approximate the maximum of the log-likelihood function defined as:340
f1=
n
X
i=1
k
X
j=1
log(αjP(yi|µj,Σj)). (14)
The initial parameters of the GMM are provided by the k-means algorithm
341
(e.g., k= 10) and random initialization.342
Let uand vbe the indexes that represent two Gaussians distributions in the343
GMM, the similarity between these two distributions is computed as follows:344
S(u, v) = 1
2(N(u, v) + N(v, u)), (15)
where N(a, b) is the probability that the mean of one distribution indexed by a
345
is a realization of the mean of the other distribution indexed by b:346
N(a, b) = 1
p(2π)m|Σa|exp −1
2(µa−µb)TΣ−1
a(µa−µb). (16)
Next, the TDLS metric D(u) is computed to measure the dissimilarity of the
347
distribution indexed by ufrom all the other distributions:348
D(u) =
10
X
v=1
(1 − S(u, v ))P(v|I), (17)
where P(v|I) is the probability of pixel association with the distribution indexed
349
by vin the image I. Each pixel is associated with the distribution in which it350
presents the higher probability. Figure 5 gives an illustrative example of the351
TDLS metric.352
To segment the MPSL image, the SRM (Statistical Region Merging) algo-353
rithm is first applied to divide the image into regions. Then, for each region R,354
16
the region TDLS metric DR(R) is evaluated as follows:355
DR(R) =
10
X
u=1
D(u)P(u|R), (18)
where P(u|R) denotes the probability of pixel association with the distribution
356
indexed by uin the region R. Finally, the Otsu’s threshold method is applied357
to generate the binary mask corresponding to the MPSL image segmentation.358
(a) (b)
Figure 5: (a) Shading attenuated MPSL image; (b) Five distributions have been obtained and
the pixel intensities depend on the TDLS metric Dof the distribution associated with each
pixel.
4.2.2. Non-negative Matrix Factorization359
It is also possible to use Non-negative Matrix Factorization (NMF) to seg-360
ment MPSL images. In this approach, the input RGB image is represented by361
its non-overlapping square windows (i.e., patches). Thus, each column yiof Y362
represents a different patch with size w×w×3. Consequently, m= 3w2and363
n=⌊nr
w⌋⌊nc
w⌋
364
The Alternating Least Squares (ALS) algorithm is used to obtain two non-365
negative matrices Dand Xto minimize the following objective function:366
f2(D, X ) = 1
2kY−DXk2
F, (19)
where kAkFdenotes the Frobenius norm of A. Each column djof Dcan be
367
referred as an atom and the number of atoms na in the dictionary Dcan be368
seen as an input parameter of the ALS algorithm.369
17
Figure 6 illustrates some atoms obtained by the technique exploring the370
NMF approach.371
(a) (b)
Figure 6: (a) Shading attenuated MPSL image; (b) Nine dictionary atoms (in a square form)
obtained to represent the image in (a).
The k-means clustering method (with k= 2) is employed to segment the
372
patches yivia its projections xi(i.e., columns of X), the k-means++ algorithm373
is used to select the initial seeds. The patches in the larger cluster are labeled as374
“healthy skin”. Finally, each image pixel is labeled as “skin lesion” or “healthy375
skin” according to its patch location.376
4.2.3. Unsupervised Information-Theoretic Dictionary Learning377
The Unsupervised Information-Theoretic Dictionary Learning (UITDL) method378
is used to select a subset of atoms from a NMF learned dictionary in order to379
maximize a mutual information based measure of dictionary compactness and380
reconstruction.381
The UITDL method is an unsupervised version of the Information-Theoretic382
Dictionary Learning (ITDL) method, which allows to obtain more robust MPSL383
segmentation if compared with other dictionary learning based approaches.384
In the UITDL approach, the input RGB image is first converted to HSV.385
Then, the V channel is retained and next filtered, which emphasizes the local386
textural variation. Thus, each column yiof Yrepresents a different w×w387
non-overlapping patch in the filtered V channel. Consequently, m=w2and388
n=⌊nr
w⌋⌊nc
w⌋.
389
18
The ALS algorithm for NMF is used to learn a dictionary D0with na atoms.390
Then, the UITDL method initializes D=∅and for l= 1, . . . , na puts the atom391
d∗
l∈D0−Dthat provides the largest increase in the following function as the
392
the l-th column of D:393
f3(dl) = λ1[MI(D∪dl, D0−(D∪dl)) −MI(D, D0−D)]
+λ2[MI(Y, D ∪dl)−MI(Y, D )], (20)
where, MI(A, B) denotes the mutual information between Aand B.
394
Let na′the number of atoms that provides the f3(d∗
l) peak in Equation 20,
395
the UITDL method discard the atoms of the columns na′+ 1, . . . , na of D.396
Then, the patches in Yare projected in Das follows:397
X= (DTD)−1DTY. (21)
Finally, the Normalized Graph Cuts (NGC) method is employed to segment
398
the projections xi, and hence the patches yi, that are used to generate the binary399
image corresponding to the final skin lesion segmentation.400
5. Discrimination Between Malignant and Benign Pigmented Skin401
Lesions402
In order to obtain a final classification result for a given MPSL image (i.e.,403
malignant or benign), using the previously obtained segmentation mask, the404
pre-screening system must extract a set of features from the lesion area that405
will be used to train a machine learning algorithm responsible for inferring a406
skin lesion diagnosis.407
These features can be obtained by analyzing information such as Assimetry,408
Border irregularity, Color variation, and Differential Structures, which are part409
of a well know rule used by the dermatologists, and often referred to as the410
ABCD rule of dermatology.411
19
Next, we discuss a series of methods used to extract these features (see412
Section 5.1), and how they can be used to train algorithms to identify cases413
where MPSL images are indicating malign or benign cases of pigmented skin414
lesions (see Section 5.2), helping the patient to obtain the proper treatment.415
5.1. Lesion Feature Extraction416
Having as input the lesion area segmentation, MPSL image classification has417
as a previous step the extraction of different lesion representations, also called418
features. Features are an important part in these e-health systems, because they419
are the information used to train the algorithm and infer if a certain lesion is420
malignant or benign.421
The number of features can vary in the existing approaches available in the422
literature, as well as in the way they are selected. The most used representations423
try to reproduce the specialist behavior when checking a patient. The ideal424
feature set is the one that maximizes the class separability. In the following425
Sections we are going to present some of the most common features used in the426
classification of MPSL images.427
5.1.1. Region Based Features428
To describe the skin lesion area automatically detected during the segmen-429
tation step, a different number of features can be extracted according to the430
ABCD rule. Table 1 shows a set of features commonly used in the literature to431
represent asymmetric and border aspects in pigmented skin lesion images, we432
are going to define this set of features as Eregion ={f1, ..., f17 }. These features433
can be named “region based” since they present only information related to the434
whole lesion area.435
The first part of the region based feature set f1−f11 assume that a skin436
lesion may have the shape of an ellipsoid, hence, it is possible to estimate ratios437
with respect to the major principal axis L1and the minor principal axis L2. It438
also includes other aspects related to area, diameter and circularity.439
20
Table 1: Set of common region based features.
f1Ratio between the lesion area Aand convex hull area.
f2Ratio between Aand bounding box area.
f3Equivalent diameter, 4A/(L1π), where L1is the major principal
axis.
f4Circularity, 4πA/(L1p), where pis the lesion perimeter.
f5Ratio between principal axes (L2/L1), where L2is the minor prin-
cipal axis.
f6Ratio between sides of lesion bounding box.
f7Ratio between the lesion perimeter pand A.
f8(B1−B2)/A, where B1and B2are the areas in each side of axis L1.
f9(B1−B2)/A, where B1and B2are the areas in each side of axis L2.
f10 B1/B2, where B1and B2are the areas in each side of axis L1.
f11 B1/B2, where B1and B2are the areas in each side of axis L2.
f12−14 Average gradient magnitude of pixels in the lesion extended rim, in
each one of {ˆ
I1,ˆ
I2,ˆ
I3}channels.
f15−17 Variance of the gradient magnitude of the pixels in the lesion ex-
tended rim, in each one of {ˆ
I1,ˆ
I2,ˆ
I3}channels.
The remaining of the features f12 −f17 are related with the skin lesion rim440
and how color and texture change when transitioning from lesion area to healthy441
skin regions. Such information can be obtained by observing the average and442
variance in the gradient magnitude values within the lesion extended rim of the443
three specialized channels {ˆ
I1,ˆ
I2,ˆ
I3}. These channels represent respectively,444
the textural variability, the variation in the skin tone and the image color infor-445
mation obtained through principal component analysis (PCA).446
Note that it is important to obtain an accurate segmentation to extract a447
set of features summarizing all lesion characteristics, otherwise, features can be448
corrupted by noise and affect the final classification result.449
21
5.1.2. Locality Based Features450
Analyzing the information such as asymmetry and border irregularity, the451
first two points of the ABCD rule of dermatology, can be achieved by looking452
to the entire lesion area. However, to have a greater perspective in the other453
two points, color and differential structures, an alternative way to represent the454
skin lesion giving more emphasis to local textures and color consistence may be455
preferable.456
Other approaches over segmented a MPSL image in smaller regions, resulting457
in a likelihood of malignancy or benignity for the whole detected skin lesion.458
One way of generating such regions while preserving the local information is by459
over segmenting the MPSL image with a superpixel algorithm, resulting in color460
and spatially coherent smaller lesion subregions.461
We can define the whole set of subregions ˆsiof a skin lesion as ˆ
S. All the462
pixels inside ˆsiare represented by a concatenated feature vector denoted as:463
φ(ˆsi) = {L(ˆsi), a(ˆsi), b(ˆsi),ˆ
I1(ˆsi),ˆ
I2(ˆsi),ˆ
I3(ˆsi)}, (22)
where {L, a, b}are the pixel values in the CIE L*a*b* color space and {ˆ
I1,ˆ
I2,ˆ
I3}
464
are the specialized channels representing respectively, the textural variability,465
the variation in the skin tone and the image color information obtained through466
principal component analysis. Figure 7 shows a visual representation of the467
local feature vector as lesion subregions for a given MPSL image.468
22
Figure 7: Representation of the φfeature vector for every ˆsisuperpixel. In the first line,
representations of the CIE L*a*b* color space, and in the second line, the textural variation
channel ˆ
I1(ˆsi), the skin tone channel ˆ
I2(ˆsi) and the color representation using PCA ˆ
I3(ˆsi).
However, for each one of these subsets the number of pixels inside ˆsimay
469
differ, so, in order to train and classify each one of them independently it is470
necessary to find a representation able to compress the information as a set of471
features, defined here as Elocal, to be used in the lesion classification phase.472
There are mainly two ways of approaching this problem. The first one is473
to represent each one of the subsets as a vector storing local descriptors of the474
data distribution, such as mean, median, minimum, and maximum values:475
Elocal(φ, ˆsi) = {mean(φ(ˆsi)), median(φ(ˆsi)), min(φ(ˆsi)), max(φ(ˆsi))}. (23)
Another possible solution is to represent ˆsilocal statistical information as a
476
histogram with a given βnumber of bins, where a larger value for βlead to a477
high dimensional representation of Elocal:478
Elocal(φ, ˆsi, β ) = {histogram(φ(ˆsi), β)}. (24)
In this way, each subregion ˆsiis described by a feature vector Elocal that
479
can be used in the classification phase. It is also possible to combine such local480
23
representation with region based features E(ˆsi) = Elocal(ˆsi), Eregion(ˆ
S) forming481
a robust representation.482
Once all the subregions of a MPSL were classified as malignant or benign,483
it is possible to estimate an index or likelihood to the image be assigned to one484
of the classes. For instance, the malignancy index is given by the total number485
pixels classified as malignant divided by the total number of pixels inside the486
whole lesion area. If this index is higher than a threshold value the whole lesion487
is identified as malignant.488
The use of superpixels allow methods to preserve and give importance to489
local information, analyzing textural and color information at a very low-level,490
exploring smaller texture distributions that sometimes are ignored in methods491
for macroscopic pigmented skin lesion images.492
5.1.3. High-Level Intuitive Features493
The feature extraction problem can be seen from a different perspective,494
using low-level features such as mean, variance, maximum and other simple495
distribution metrics by themselves are not able to individually model all the496
skin lesion aspects. More than that, the use of such low-level features could497
result in an unnecessarily high-dimensional feature space.498
To approach this issue, High-Level Intuitive Features (HLIF) describing499
human-observable characteristics in macroscopic pigmented skin lesions can be500
used, following the ABCD rule of dermatology previously mentioned. The main501
idea is that using a smaller set of features, can increase the classification accu-502
racy and at same time avoid dimensionality problems.503
There are HLIFs related to lesion assimetry, such as color assimetry and504
structural assimetry. The color assimetry in a skin lesion can be estimated505
by computing the Earth Mover’s Distance (EMD) between the clustered CIE506
L*a*b* color space distributions in either sides of the main axis of separation.507
To estimate the structural assimetry we assume that the lesion shape is less508
likely to be symmetric as it deviates from the ideal elliptical structure, in order509
to quantify the structural complexity, Fourier descriptors can be used, followed510
24
by a reconstruction step.511
In a similar process, lesion shapes deviating from the elliptical geometrical512
form can represent coarse border irregularities in a MPSL. It is possible to513
represent the lesion borders by comparing the low-frequency reconstruction (e.g.,514
using a Fourier descriptor) and the original border lesion. Also, the fine border515
irregularities can be described by the application of morphological operators,516
allowing to manipulate the skin lesion shapes on a local scale.517
The pigmented skin lesion color is another lesion aspect that can be consid-518
ered a HLIF, and a 4-step framework can be used to capture the complexity of519
the color distribution: 1) transform the original image to a perceptually uni-520
form color space; 2) construct a color-spatial representation to model the color521
information for a local patch; 3) cluster the patch representations into kcolor522
clusters; and 4) quantify the variance found using the original lesion and the k523
representative colors. Using this framework, information like the reconstruction524
error, the color complexity evolution, and the mean color differences can be525
obtained from the MPSL.526
Using such high-level intuitive features combined with the state-of-the-art527
low-level features tends to increase the classification accuracy, since the set of528
HLIFs tend to present statistical significance and help discriminate between529
malignant and benign macroscopic pigmented skin lesions images.530
5.1.4. Patient Related Features531
Another way to improve the final pre-screening accuracy is to use external532
features associated with the macroscopic pigmented skin lesion information.533
One of the most common approaches assumes that people presenting certain534
physical characteristics are more likely to develop some form of skin cancer.535
For instance, a Bayesian classifier can combine the results obtained in the536
image classification step (using region based features), with the general epidemi-537
ological risk of cancer, given patient specific information, such as skin color, age,538
gender and the body region affected by the pigmented skin lesion.539
The major drawback of using these features is the lack of training data,540
25
considering that most of the available data sets do not provide such patient541
related information.542
5.2. Lesion Classification543
Considering the different methods used to extract features based in different544
aspects of a skin lesion, the last step in the pre-screening framework for MPSL545
images is the classification step.546
Our focus here is in supervised algorithms, which are methods that require547
a training phase that uses a set of images manually evaluated by one or more548
specialists. After the training step, the algorithm is used to assign a class to549
the non-evaluated images, indicating if the macroscopic pigmented skin lesion550
belongs to one of the two possible classes, benign or malignant.551
This process will be more detailed in the next Sections, starting in Sec-552
tion 5.2.1 with a review of the most used data sets in literature and the existent553
techniques to validate classification results. Next, Section 5.2.2 discusses the554
importance of feature scaling. To conclude, in Section 5.2.3, we present some555
of the machine learning algorithms used to classify macroscopic pigmented skin556
lesions.557
5.2.1. Evaluation and Validation558
To compare different techniques and evaluate how the approaches for MPSL559
diagnosis will behave in real world conditions, it is necessary to evaluate the560
classification and segmentation results in macroscopic pigmented skin lesions561
data sets. These data sets contain a different number of MPSL images with the562
respective dermatologist lesion area delimitation and diagnosis.563
Below are listed some popular data sets used for evaluating segmentation564
and classification methods:565
•DermNet 1: this data set presents 152 MPSL selected images, where 45566
were identified as benign and 107 as malignant. Each 24 bits RGB color567
1Dermnet Skin Disease Image Atlas, http://www.dermnet.com.
26
image has a size of 720 pixels on the larger side, and dimensions between568
439 and 706 pixels on the smaller side. DermNet is used to evaluate many569
segmentation and classification approaches.570
•DermIS 2: contains a total of 69 images, where 43 are cases of melanomas571
and 26 of non-melanomas. Images were obtained from the Dermatology572
Information System database.573
•DermQuest 3: this data set contains 137 MPSL images selected from574
the DermQuest database, from this total, 76 images are melanomas and575
61 non-melanomas.576
•Dermofit 4: compared with other data sets, this data set is large, contain-577
ing 1300 MPSL images with 10 different lesion categories: Actinic Ker-578
atosis, Basal Cell Carcinoma, Melanocytic Nevus/Mole, Squamous Cell579
Carcinoma, Seborrhoeic Keratosis, Intraepithelial Carcinoma, Pyogenic580
Granuloma, Haemangioma, Dermatofibroma, and Malignant Melanoma.581
Supervised classification techniques require at least two different sets of im-582
ages, one for the training phase and another for the test phase. One possible583
approach is to train the algorithm in one data set, and test in another, but584
sometimes there is a small number of publicly available sources of images and585
even a small number of images, the alternative is to use as much data as possible586
through cross-validation (CV).587
There are different ways of cross-validate a classification algorithm, and one588
of the simplest is the holdout approach. When using holdout cross-validation,589
the data set is randomly divided in two parts, the first is used in training and590
the second in validation phase. After that, accuracy results are computed, and591
the roles are inverted. The second set is now used for training and the first for592
2Dermatology Information System, http://www.dermis.net.
3DermQuest, http://www.dermquest.com.
4Dermofit Image Library, http://www.licensing.eri.ed.ac.uk/i/software/dermofit-image-
library.html.
27
validation. When all data is classified, the validation accuracy can be averaged593
giving an idea of how the method will perform on new images and with different594
training samples. The problem with holdout CV is the high variability of the595
final results, given a small number of trials.596
To reduce the variability in the trials, K-fold CV is a more flexible approach,597
where the full data set of images is divided into Kpartitions, where K−1 sets598
are used to train the algorithm and one is used for validation. After all folds are599
classified (as in holdout CV), the classification rates can be averaged considering600
all the Ktrials. The holdout CV is a special case of K-fold CV, where K= 2.601
Leave-one-out CV is another special case of K-fold CV, where the total602
number of folds is equal to the number of samples Nin the data set. In practice,603
the N−1 samples are used to train the algorithm, and the image left out is604
used to test. This is an exhaustive test, since every image will be individually605
evaluated at a cost of a larger number of trials. However, the variability obtained606
often is smaller, considering that it is possible to observe much better how the607
algorithm will perform for new images.608
5.2.2. Feature Scaling609
When features present different ranges for their values, some of the image610
aspects may incorrectly assume more importance than others, when in fact all611
image aspects should have similar weights.612
To avoid or minimize this scaling problem, different methods for feature613
scaling can be used to represent all the feature values in the same range or in614
the same scale, which tends to improve the performance of classifiers.615
One of the most common ways to normalize feature data is using a linear616
scaling function to scale values in the interval between 0 and 1:617
ˆ
X=X−min(X)
max(X)−min(X), (25)
where Xis the feature data and the ˆ
Xis the normalized feature vector.
618
Another popular way to address this issue is using feature standardization619
28
to transform the feature vector Xto have zero mean and unit variance:620
ˆ
X=X−µ
σ, (26)
where µand σare the sample mean and sample standard deviation of a feature.
621
In MPSL image classification, the z-score transformation has been used.622
Assuming that extracted features are normally distributed, it guarantees that623
99% of ˆ
Xwill be between 0 and 1 by operating a shift and re-scaling operation:624
ˆ
X=
X−µ
3σ+ 1
2, (27)
the values outside this range are saturated to 0 or 1.
625
Now that the macroscopic skin lesion features were correctly normalized,626
we can move to the classification step, where they are used to train a machine627
learning algorithm responsible to learn the representation of each one of the628
classes based in the training set, and finally, inferring the diagnosis for a given629
MPSL image.630
5.2.3. Machine Learning Algorithms631
Macroscopic pigmented skin lesion classification can be challenging and a632
careful evaluation of methods is required in order to produce trustworthy re-633
sults. Machine learning algorithms chosen to classify the set of MPSL features634
extracted often need to deal with unbalanced classes, small training sets, and635
with class overlapping in the feature space.636
Another issue that needs to be addressed is the feature selection problem,637
where algorithms are used to identify the most representative set of lesion638
attributes to obtain acceptable classification accuracies. This is the case of639
Correlation-based Feature Selection used to improve the results obtained with an640
ensemble of weak classifiers, and the Classification and Regression Tree (CART)641
method.642
Other frameworks use a SVM (Support Vector Machine) classifier. The643
SVM classifiers are very flexible, exploring the concept of kernel methods (e.g.644
29
polynomial and radial) to construct solutions which are non-linear in feature645
space and search for discriminant functions that maximize the margin between646
the classes, increasing the algorithm generalization ability.647
Also, methods using non-parametric algorithm can be found in the literature,648
such as k-NN (k-Nearest Neighbors classifier). This class of machine learning649
algorithms has only a small number of parameters to be determined from the650
data set, and in the case of k-NN, no particular data distribution is assumed.651
The only parameters to be defined are the distance metric, usually Euclidean,652
and the number of neighbors kto be evaluated, where kis an odd value to avoid653
ties.654
More recent approaches use a linear classifier trained with features extracted655
from a Convolutional Neural Network (CNN) pre-trained in a data set with nat-656
ural images. This classifier showed potential to diagnose macroscopic pigmented657
skin lesions images presenting different types of skin cancer. The method does658
not require any segmentation mask. However, the learned features do not rep-659
resent any specific characteristic of the skin lesion itself, what makes it difficult660
to do a more in depth evaluation of possible misclassification problems.661
We expect that methods that do not require a segmentation phase and design662
of features probably will receive more attention in the future. On the other663
hand, there is interest in controlling what features are used in dermatologist-664
like systems.665
Further Reading666
[1] C. M. Bishop, Pattern Recognition and Machine Learning (Information Sci-667
ence and Statistics), Springer-Verlag New York, Inc., Secaucus, NJ, USA,668
2006.669
[2] E. Bernart, J. Scharcanski, S. Bampi, Segmentation and classification of670
melanocytic skin lesions using local and contextual features, in: Image671
Processing (ICIP), 2016 IEEE International Conference on, IEEE, 2016,672
pp. 2633–2637.673
30
[3] E. Flores, J. Scharcanski, Segmentation of melanocytic skin lesions using674
feature learning and dictionaries, Expert Systems with Applications 56675
(2016) 300–309.676
[4] J. F. Alc´on, C. Ciuhu, W. Ten Kate, A. Heinrich, N. Uzunbajakava,677
G. Krekels, D. Siem, G. De Haan, Automatic imaging system with decision678
support for inspection of pigmented skin lesions and melanoma diagnosis,679
IEEE Journal of Selected Topics in Signal Processing 3 (1) (2009) 14–25.680
[5] J. Glaister, A. Wong, D. A. Clausi, Segmentation of skin lesions from digital681
images using joint statistical texture distinctiveness, IEEE Transactions on682
Biomedical Engineering 61 (4) (2014) 1220–1230.683
[6] J. Glaister, R. Amelard, A. Wong, D. A. Clausi, MSIM: Multistage illumi-684
nation modeling of dermatological photographs for illumination-corrected685
skin lesion analysis, IEEE Transactions on Biomedical Engineering 60 (7)686
(2013) 1873–1883.687
[7] J. Kawahara, A. BenTaieb, G. Hamarneh, Deep features to classify skin le-688
sions, in: 2016 IEEE 13th International Symposium on Biomedical Imaging689
(ISBI), 2016, pp. 1397–1400.690
[8] J. Koehoorn, A. C. Sobiecki, D. Boda, A. Diaconeasa, S. Doshi, S. Paisey,691
A. Jalba, A. Telea, Automated digital hair removal by threshold decompo-692
sition and morphological analysis, in: International Symposium on Mathe-693
matical Morphology and Its Applications to Signal and Image Processing,694
Springer, 2015, pp. 15–26.695
[9] J. Scharcanski, M. E. Celebi, Computer vision techniques for the diagnosis696
of skin cancer, Springer, 2014.697
[10] M. E. Celebi, G. Schaefer, Color medical image analysis, Vol. 6, Springer698
Science & Business Media, 2012.699
31
[11] M. Zortea, E. Flores, J. Scharcanski, A simple weighted thresholding700
method for the segmentation of pigmented skin lesions in macroscopic im-701
ages, Pattern Recognition 64 (2017) 92–104.702
[12] N. Otsu, A threshold selection method from gray-level histograms, Auto-703
matica 11 (285-296) (1975) 23–27.704
[13] P. G. Cavalcanti, J. Scharcanski, C. B. Lopes, Shading attenuation in hu-705
man skin color images, in: International Symposium on Visual Computing,706
Springer, 2010, pp. 190–198.707
[14] P. G. Cavalcanti, J. Scharcanski, C. E. Mart´ınez, L. E. Di Persia, Segmen-708
tation of pigmented skin lesions using non-negative matrix factorization,709
in: 2014 IEEE International Instrumentation and Measurement Technol-710
ogy Conference (I2MTC) Proceedings, IEEE, 2014, pp. 72–75.711
[15] R. Amelard, J. Glaister, A. Wong, D. A. Clausi, High-level intuitive features712
(hlifs) for intuitive skin lesion description, IEEE Transactions on Biomed-713
ical Engineering 62 (3) (2015) 820–831.714
32
