Content uploaded by Bing Wang
Author content
All content in this area was uploaded by Bing Wang on Apr 23, 2022
Content may be subject to copyright.
IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 71, 2022 5005610
A Real-Time Steel Surface Defect Detection
Approach With High Accuracy
Wenyan Wang , Chunfeng Mi ,ZihengWu ,KunLu , Hongming Long ,BaigenPan ,
Dan Li ,JunZhang ,PengChen , and Bing Wang ,Senior Member, IEEE
Abstract— Surface defect inspection is a key step to ensure the
quality of the hot rolled steel surface. However, current advanced
detection (DET) methods have high precision but low detection
speed, which hinders the application of the detector in actual
production. In this work, a real-time detection network (RDN)
focusing on both speed and accuracy is proposed to solve the
problem of steel surface defect detection. RDN takes ResNet-
dcn, a modular encoding, and decoding network with light
weights, as the basic convolutional architecture whose backbone
is pretrained on ImageNet. To improve the detection accuracy,
a skip layer connection module (SCM) and a pyramid feature
fusion module (PFM) are involved into RDN. On the standard
dataset NEU-DET, the proposed method can achieve the state-of-
the-art recognition speed of 64 frames per second (FPS) and the
mean average precision of 80.0% on a single GPU, which fully
meets the requirements of the detection accuracy and speed in
the actual production line.
Index Terms—Detection (DET) speed and accuracy, hot rolled
steel surface, pyramid feature fusion module (PFM), skip layer
connection module (SCM), surface defect detection.
I. INTRODUCTION
AS THE basic raw material of iron and steel industry,
the surface quality of hot rolled strips seriously affects
the physical and chemical properties of downstream and
final products [1]–[3]. However, traditional detection (DET)
methods based on artificial observation are time-consuming
Manuscript received July 28, 2021; revised October 9, 2021; accepted Octo-
ber 19, 2021. Date of publication November 15, 2021; date of current version
March 3, 2022. This work was supported by the National Natural Science
Foundation of China under Grant 62172004, Grant 61672035, and Grant
61872004; and in part by the Educational Commission of Anhui Province
under Grant KJ2019ZD05. The Associate Editor coordinating the review
process for this article was Dr. Jing Lei. (Corresponding authors: Bing Wang;
Pen g Ch e n. )
Wenyan Wang and Hongming Long are with the School of Metallurgical
Engineering, Anhui University of Technology, Ma’anshan, Anhui 243002,
China, and also with the Key Laboratory of Metallurgical Emission Reduction
and Resources Recycling, Ministry of Education, Anhui University of Tech-
nology, Ma’anshan 243002, China (e-mail: wenyanwang9203@gmail.com;
13956233905@126.com).
Chunfeng Mi, Ziheng Wu, Kun Lu, Dan Li, and Bing Wang are
with the School of Electrical and Information Engineering, Anhui
University of Technology, Ma’anshan, Anhui 243032, China (e-mail:
michunfeng64@gmail.com; wziheng@ahut.edu.cn; kunlu0819@gmail.com;
lanldok@163.com; wangbing@ustc.edu).
Baigen Pan is with the Anhui Provincial Quality Supervision and Inspection
Center for Motor Products and Parts, Xuancheng, Anhui 242500, China
(e-mail: pbg770302@aliyun.com).
Jun Zhang and Peng Chen are with the Co-Innovation Center for Information
Supply and Assurance Technology, Anhui University, Hefei, Anhui 230032,
China (e-mail: junzhang@ahu.edu.cn; pchen@ahu.edu.cn).
Digital Object Identifier 10.1109/TIM.2021.3127648
and unreliable. Therefore, it is desirable to allow a machine
to automatically detect surface quality.
Automatic defect detection technology based on machine
vision has good performance in object classification tasks
[4]–[9]. However, the defect features it uses require manual
operation, and the changeable steel surface defects with low-
dimensional artificial features generally have poor general-
ization performance, while the extraction process of a large
number of features will slow down the detection process,
which hinders the application of machine vision-based defect
detection methods in the actual industrial production [9].
In recent years, the deep-learning-based methods have made
many breakthroughs in the classification of surface defects
of hot rolled steel strip [10]–[12]. Konovalenko et al. [11]
proposed a number of convolutional neural networks (CNNs)
based on residual network ResNet-50, in which the class
weight, minority class over-sampling, and focal loss strategies
are adopted to solve the data imbalance of training samples.
The experimental results show that this method can effectively
detect defects with high accuracy of 96.91% [11]. Wang et
al. [13] developed an improved CNN model with reduced
training parameters and improved the accuracy and speed
of defect classification to 99.63% and 333 frames per sec-
ond (FPS), respectively. Azimi et al. [14] first segment the
microstructure of low carbon steel by full convolution neural
network, and then identify the category of components through
these structures. This kind of research provides a robust and
objective method for steel quality appreciation [14].
However, in the classification task, the model is only
required to give each defect image a defect category. When
an image contains two or more types of defects, as shown in
Fig. 1, the classification model will not classify these defects in
detail. In contrast, object detection technology can enumerate
all types of defects and accurately indicate the location of these
defects. Wang et al. [15] proposed an improved ResNet50 and
enhanced faster RCNN for steel surface defect detection, and
achieved the classification accuracy and speed of the model
of 98.2% and 63 ms, which significantly improves the perfor-
mance of the model compared with other methods. Aiming
at the surface defect detection of hot-rolled strip and on the
NEU-DET dataset with the characteristics of large intraclass
difference and high interclass similarity, He et al. [16] pro-
posed an end-to-end defect detection network that integrating
multilevel features into one feature map, and achieved the
mean average precision (mAP) of 82.3%. However, when they
1557-9662 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Anhui University of Technology. Downloaded on April 09,2022 at 09:04:44 UTC from IEEE Xplore. Restrictions apply.
5005610 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 71, 2022
Fig. 1. Surface defect image of hot rolled strip containing two kinds of
defects.
increased the detection speed to 20 FPS, the mAP of the
detection model drops to 70% [16]. Also on the NEU-DET
dataset, Hatab et al. [17] used the YOLO network for defect
detection, and achieved the mAP of 70.66% and the detection
speed of 85 ms per image. Kou et al. [18] developed an end-to-
end defect detection model based on YOLO-V3, and yielded
72.2% mAP on the NEU-DET dataset. Aiming at the problem
that deep learning method is difficult to recognize small and
complex targets, Zhao et al. [19] proposed an improved Faster-
RCNN method, and achieved the mAP of 75%. To describe
the defect information in more detail, some segmenta-
tion models are also proposed in recent years [20], [21].
However, it is impractical to segment defects of a steel plate
in industrial production, and the accuracy and speed of the
detection methods are out of balance.
To achieve faster detection speed and improve the general-
ization performance of the model, a real-time defect detection
network (RDN) is proposed in this work, which has three
innovations in the structure. First, compared with the existing
classic object detection network, such as Faster RCNN, which
usually uses a regional proposal network (RPN) to generate
bounding boxes, RDN is an end-to-end detection network.
Furthermore, different from one-stage detection method that
lists as many proposals as possible, such as YOLO, SSD, and
M2Det, RDN uses the center point of the target to generate
a bounding box, which is an anchor free method with faster
detection speed.
Second, the highest-level feature map containing rich
semantic information is uaually used to detect defects in deep-
learning-based detection networks [22], [23]. However, in the
NEU-DET dataset, there are many small defects that need to
be detected, and the characteristics of large similarity between
defect categories and small differences within classes make
the model difficult to effectively detect defect categories and
location information for a single feature map. It has been
proven that different levels of features in CNN represent
different information about an object [24], i.e., low-level
features contain precise location information, while high-level
features have rich contextual semantics. Therefore, in this
work, the features extracted from the last convolution layer
of each stage of the backbone are integrated to make full use
of the shallow layer information, which is called the skip-
layer connection module (SCM). In addition, to improve the
detection performance of defect images with large intraclass
differences and high interclass similarity, a pyramid feature
fusion module (PFM) that combines multiple high-feature
layers is also proposed.
Thirdly, inspection speed is an important problem for the
application of the model into actual production. In order to
accelerate the detection speed, a lightweight and modular
ResNet-18 network is used as the feature extraction backbone
in this work. In particular, ResNet-18 is more suitable for
small sample datasets because it has few parameters to be
trained. The experimental results show that the lightweight
backbone and feature fusion strategy can significantly improve
the accuracy and speed of the detection model.
In summary, the main contributions of this work are as
follows.
1) For steel defect classification and localization, an end-to-
end accuracy-speed balanced defect detection network is
introduced.
2) A SCM between shallow layers is proposed, which can
provide more accurate position information for low-
contrast images.
3) The PFM is proposed to improve the network’s ability to
detect defects with large intraclass differences and high
interclass similarity.
4) Using a simple backbone ResNet-18, our network runs
at 64 FPS with mAP of 80.0%, which fully meets the
speed requirements in actual production.
II. REAL-TIME DEFECT DETECTION NETWORK
The overall architecture of the balanced defect detection
network proposed in this work is shown in Fig. 2. RDN
uses baseline convolution architecture and multilevel feature
fusion to extract features from the input image, and then uses
the keypoint evaluation and regression method to obtain the
center point and size attributes of the object. Considering
the characteristics of the NEU-DET dataset mentioned above,
this work constructed two multilevel feature fusion modules
with few parameters. One is the SCM which combines the
features with the same resolution, the other is the PFM. More
details of the backbone selection and feature fusion module
are described below.
A. Baseline Convolution Architecture
Generally, a network backbone with a small number of
parameters and a pretrained model on ImageNet can avoid
over-fitting and accelerate the speed of model detection, espe-
cially for small datasets [26]. Compared with some backbones
widely used in object classification and detection, such as
VGG, DLA, and ResNeXt-101, ResNet with 18 convolution
layers have fewer model parameters and modular convolution
Authorized licensed use limited to: Anhui University of Technology. Downloaded on April 09,2022 at 09:04:44 UTC from IEEE Xplore. Restrictions apply.
WANG et al.: REAL-TIME STEEL SURFACE DEFECT DETECTION APPROACH WITH HIGH ACCURACY 5005610
TAB L E I
DETAILED STRUCTURES OF RESNET18-DCN
structure, which make it easier to integrate and deploy on the
industrial devices [22], [23], [25], [27]. In addition, its superior
performance in the task of surface defect classification of hot
rolled strip has been proven in this work. Therefore, ResNet-18
is selected as the backbone of the industrial defects detection
model.
Different from the previous object detection network, where
the highest-level feature map with the smallest size of back-
bone is adopted to detect and classify the attributes of targets,
an encoding and decoding network ResNet18-dcn with large
output resolution is proposed in this work as the baseline
convolution architecture since the highest-level feature map
has a large receptive field, which is not conducive to the
detection of small targets [24], [28]. In the decoding network
of ResNet18-dcn, the channel number and resolution of the
feature map are adjusted by the deformation convolution and
deconvolution. The detailed structures of ResNet18-dcn are
given in Table I.
B. Skip-Layer Connection Module
In deep learning methods, the category and location infor-
mation of defects are mainly obtained by multiple convolution
of the input image. However, the receptive field of a deep
feature map will expand with the increase of the number
of layers, which weakens the perception of local location
information, and therefore affects the accurate positioning.
Particularly, the loss of location information aggravates the
difficulty of small object detection. The size of the receptive
field is defined as follows:
lk=lk−1+fk−1∗
k−1
i=1
si(1)
where lk−1is the receptive field size of the (k−1)th layer,
fkis the size of the filter or pooled kernel, and siis the stride
of layer i.
The schematic of receptive filed is shown in Fig. 3. It can
be found that the shallow layer has a small receptive field,
which is beneficial to detect small-scale defects. Therefore,
this work involves the shallow features into the deep feature
maps to make full use of the location information of defects.
In addition, a decoding network is added to alleviate the
location loss caused by very small feature map.
It has been proven that skip layer connection (SLC) can
simplify the network learning process, enhance gradient propa-
gation and make the feature expression of input variables more
comprehensive [23], [27], [29], [30]. Additionally, relevant
research also shows that long SLC with multiple SLCs can per-
form in-depth supervision of the entire network structure [31].
To improve the detection performance of the model for small
objects, an SCM that contains multiple long SLCs is designed
into the network without increasing the calculation of the
model. In RDN, the SCM is introduced between two feature
maps of the same size, as shown by the blue connecting line
in Fig. 2.
C. Pyramid Feature Fusion Module
In detection networks, the feature pyramid is an important
component for detecting objects at different scales [24], [28].
However, it has a slow detection speed and a large amount
of parameters. To accurately distinguish the defect types with
a large intraclass difference and interclass similarity, a simple
PFM which integrates multiple high-level features into one
feature map is designed to promote the highest-level feature
map have sufficient sample feature information.
To fuse features with different sizes in the inverted pyramid,
as shown in Fig. 2, PFM is attached to the decoding structure
of the network, which is connected by yellow lines. In PFM,
four feature maps at different levels in the decoding network
are used as input. The output is the sum of the corresponding
elements of these feature maps after size transformation.
To keep the output resolution of the network, four input
branches in PFM are expanded to the same size by deconvo-
lution operations with different convolution kernel sizes and
strides.
III. TRAINING AND INFERENCE
A. Training
The defect detector based on keypoints model an object as
a single point (the center point of its bounding box), and then
uses this keypoint to get the target category, size (width and
height), and center point offset. In the RDN structure shown
in Fig. 2, the cls layer is used to output the class probability
of the target and a keypoint heatmap. Let I∈RW×H×3be
an input image of width Wand height H,thecls layer will
produce a keypoint heatmap ˆ
Yxyc =[0,1](W/R)×(H/R)×Cwhere
Ris the output stride of the network and Cis the number
of defect categories. In this work, Ris 4, and the defect
type is 6. A prediction ˆ
Yxyc =1 corresponds to a detected
object, while ˆ
Yxyc =0 is background. The training objective
is a penalty-reduced pixel-wise logistic regression with focal
loss
L=−1
N
xyc 1−ˆ
Yxycαlogˆ
Yxyc,if Yxyc =1
1−Yxycβˆ
Yxyc log1−ˆ
Yxyc,otherwise
(2)
Authorized licensed use limited to: Anhui University of Technology. Downloaded on April 09,2022 at 09:04:44 UTC from IEEE Xplore. Restrictions apply.
5005610 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 71, 2022
Fig. 2. Overall architecture of RDN.
where ˆ
Yxyc represents the probability of category cat the
position (x,y),Yxyc is the value corresponding to the ground-
truth center point, which can be obtained by Gaussian kernel
exp (−((x−˜p2
x)+(y−˜p2
y)/2σ2
p)),where ˜p=(p/R)is
the center point position of ground truth object kon the low-
resolution output feature map, pis ground-truth center point
location, and σpis the standard deviation of object adaptation.
αand βare used to increase the loss of samples that are
difficult to classify and reduce the penalty of locations around
the ground truth, respectively, [32], [33]. Nis the number
of objects in an image. In focal loss, the previous work has
shown that, more weight becomes concentrated on the hard
negative examples substantially as αincreases from 0 and 2,
substantially more weight becomes concentrated on the hard
negative examples. When α=2, the model achieves the best
detection performance [32]. On this basis, Law and Deng [33]
proposed a variant of focal loss and set βto 4, which allows
the network to mainly optimize the hard samples. In this work,
the same target detection algorithm is studied. Therefore, the
values of αand βarealsosetto2and4.
Suppose the truth coordinates of target kis (x1,y1,x2,y2),
then its size sk, i,e., width and height, is (x2−x1,y2−y1).
The loc_wh layer is used to predict the size of k, and it can
be trained with a smoothL1loss function defined as
Lsize =1
N
N
k=1
smoothL1ˆ
Sk−sk(3)
where
smoothL1=0.5x2,if |x|<1
|x|−0.5,otherwise.
ˆ
Skis the predicted size.
To compensate for the input image pixel position error
caused by down pooling operation in the network, the offset
size of center-point position is predicted in this work, and the
loss function is defined as
Loff =1
N
xyc
smoothL1ˆ
O−p
R−˜p (4)
where ˆ
Odenotes the predicted center offset of target, pis
ground-true center point location, and ˜pis the position of p
on the low-resolution output feature map.
According to the above definitions, the overall training
objective of this work is to minimize the multitask loss
function, which is defined as
Lxyc =L+λsizeLsize +Loff (5)
where λsize =0.1, and its value is determined by the experi-
mental results in Section IV-D of this work.
B. Inference
In the inference phase, to suppress the overlap box, the
3×3 max-pooling operation is first used on the heatmap of
each category, and then the top 50 peaks of the remaining
response values are kept. Let ˆ
Pcbe the set of ndetected center
points ˆp=(ˆxi,ˆyi)n
i=1of class c, the detection coordinate of
the target is
ˆxi+δˆxi−ˆwi
2,ˆyi+δˆyi−
ˆ
hi
2,ˆxi+δˆxi+ˆwi
2,ˆyi+δˆyi−
ˆ
hi
2
(6)
where (δ ˆxi,δˆyi)is the center point offset and (ˆwi,ˆ
hi)is the
predicted size. For a specific test image, the detection result
is that the rectangular boxes produced by multiple scales and
filtered after the nonmaximum suppression method [34].
IV. EXPERIMENTS
A. Implementation Details
For all experiments based on RDN, the input image size
is 384 ×384, the initial learning rate is 1.25 ×10−4,and
Authorized licensed use limited to: Anhui University of Technology. Downloaded on April 09,2022 at 09:04:44 UTC from IEEE Xplore. Restrictions apply.
WANG et al.: REAL-TIME STEEL SURFACE DEFECT DETECTION APPROACH WITH HIGH ACCURACY 5005610
TAB L E I I
DETECTION RESULTS OF DIFFERENT BASELINE NETWORKS
Fig. 3. Schematic of receptive field size.
then decrease it to 1.25 ×10−5and 1.25 ×10−6at 60 and
120 epochs, and stop at 160 epochs. RDN is developed with
PyTorch v1.3.1. We conduct experiments on the NVIDIA Titan
X GPU hardware platform. The batch size is set to 24, and
Adam is used to optimizing the overall objective. To avoid
over fitting of the model, several data augmentation methods,
such as shift, rotation, reflection, etc., are performed in training
phase. It is worth noting that the values of these hyperparame-
ters are obtained by referring to some related object detection
documents and verified and/or adjusted through a large number
of experiments [11], [24], [38]–[44].
The data used in this work comes from a surface defect
dataset named NEU-DET constructed by the Northeast Uni-
versity, China [14]. The database collects gray images of six
typical surface defects of the hot-rolled strip, including rolled-
in scale (Rs), scratches (Sc), pitted surface (Ps), inclusion (In),
patches (Pa), and crazing (Cr), and provides annotation files
for these defects. The dataset contains 300 images for each
type and therefore a total of 1800 for six types of surface
defects of the hot-rolled strip. Some of them are shown in
Fig. 4. For the defect detection task in this work, 70% of the
images (210 images for each of them) are randomly selected
to train the model, and the remains are used for model testing.
B. Defect Classification on NEU
Defect classification is one of the tasks of defect detec-
tion, and therefore a good defect classification performance
should be positively correlated with a stronger defect detection
performance. Therefore, we report the performance of the
feature extraction method used in our work, such as VGG16,
Fig. 4. Six typical defect sample images in NEU-DET database. (a) PS.
(b) In. (c) RS. (d) Cr. (e) Sc. (f) Pa.
ResNet18, ResNet34, ResNet50, DLA34, ResNeXt50, and
ResNeXt101 [22], [27], [35]. As a result, each network can
classify all kinds of defects correctly, and the accuracy is
100%, except VGG16 which achieves the classification accu-
racy of 99.63%. Moreover, the number of training parameters
are 250M, 11M, 21M, 23M, 15M, 42M, and 22M. Therefore,
ResNet18, which has the best classification performance and
the least parameters, is the best basic network backbone.
C. Detection Performance of RDN
1) Detection Performance on NEU-DET: To achieve better
detection speed in detecting specific steel surface defects,
an end-to-end detection network is proposed in this work. The
detection results of defect samples using different improved
network structures are shown in Table II. It can be seen
that three baseline networks, i.e., ResNet18-dcn, ResNet18-ds,
and ResNet18-dsf, can detect the defects of the steel surface.
The mAP of these networks is 71.5%, 77.6%, 80.0%, and
the speed is 70, 70, and 64 FPS, respectively. Under the
baseline ResNet18-dsf, RDN achieves the best performance
with the mAP of 80.0% and the speed of 64 FPS. The average
precision (AP) is 87.0% for Ps, 84.9% for In, 64.4% for Rs,
53.7% for Cr, 95.9% for Sc, and 92.4% for Pa.
Compared with ResNet18-dcn, it can be found that
ResNet18-ds with SLC improves the mAP by 6.1%. For each
defect, the AP increased by 2.5%, 5.5%, 2.9%, 21.5%, 4.7%,
and 0.4%, respectively, and the detection speed remained
at 70 FPS, which shows that SCM integrated with shallow
Authorized licensed use limited to: Anhui University of Technology. Downloaded on April 09,2022 at 09:04:44 UTC from IEEE Xplore. Restrictions apply.
5005610 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 71, 2022
TABLE III
ABLATION STUDY OF RDN
Fig. 5. Influence of different improved modules on test results. (a) Influence
of SCM. (b) Influence of PFM.
features can provide more accurate object information without
adding a lot of calculations. In more detail, as shown in
Fig. 5(a), it can be seen that the SCM is effective for small tar-
get detection. When the PFM is introduced into ResNet18-ds,
the AP of each defect is improved to some extent, and the
overall mAP is increased to 80.0%. This demonstrates that the
fusion of the pyramid feature is beneficial to the identification
and location of steel surface defects with large intraclass
differences and high interclass similarity. Specifically, a large
number of defect images with the high similarity between
target and background, such as crazing and pitted surfaces,
are successfully located, as shown in Fig. 5(b). Without bells
and whistles, RDN using the ResNet18-dsf as the baseline
architecture achieves the best tradeoff between speed and
precision.
2) Real-Time Analysis: In RDN, only 8.71 s were spent
when 540 images were detected, which means the average time
to detect an image is 0.016 s and the detection speed is 64 FPS.
In the actual production line, the shooting field of a single
camera is 50–100 cm and the maximum production speed is
typically 30 m/s, which requires the detection equipment to
have at least 30–60 FPS detection speed [36]. Therefore, the
64 FPS detection speed of the proposed RDN in this work can
meet the real-time requirements in actual production.
D. Additional Experiments
Since RDN is composed of multiple subcomponents, it is
necessary to measure their effectiveness to the final perfor-
mance. In addition, to obtain the optimal detection perfor-
mance on the same detection architecture, some experiments
are carried out to select appropriate hyperparameters, such as
regression loss function and loss weight.
1) Ablation Study: In the defect detection algorithm, the
backbone not only affects the model performance, but also
relates to the inference speed. In this work, to obtain the
best detection speed, three experiments with ResNet18-dcn,
ResNet34-dcn, and DLA34 as backbone were performed.
It can be seen from the second, third, and fourth columns of
Table III that as the complexity of the backbone decreases, the
model performance gradually deteriorates, while the detection
speed shows a positive change trend. In particular, when
ResNet18-dcn is used as the backbone, the model detection
speed can reach 70 FPS. To improve the model performance,
SCM is introduced, and the model detection accuracy is
improved from 71.5% to 77.6% as illustrated in column 5.
Furthermore, when the PFM is involved in our network, the
final detection accuracy is improved to 80% mAP, as shown
in the sixth column.
2) Regression Loss: In object detection methods, the loca-
tion and size information related to the bounding box of the
object is usually obtained by regression. L1, L2, balanced L1,
and smooth L1 are main regression loss functions [38]. In this
work, we compare the detection performance of these four
regression losses. As a result, their corresponding mAP are
78.7%, 78.1%, 70.8%, and 80.0%, respectively. It can be found
that the smoothL1is superior to other loss functions in mAP.
3) Bounding Box Size Weight: To analyze the sensitivity
of the bounding box size weight λsize,wehavecompared
the performance of mAP with the increments of λsize with
0.05 and 0.1 when the values of λsize are in the range of (0,0.5]
and (0.5,1], respectively. The performance between (0,0.5]is
78.6%, 80.0%, 78.9%, 78.7%, 78.6%, 78.9%, 77.4%, 77.5%,
77.7%, and 76.7%, respectively. It can be seen that the
mAP performance is better when λsize is set to 0.1. In the
range of (0.5,1], the detection performance of the model is
76.8%, 76%, 74.2%, 73.8%, and 74.1%, respectively, which
shows a downward trend with the increase of λsize. Therefore,
loc_wh =0.1 is the best bounding box size weight.
Authorized licensed use limited to: Anhui University of Technology. Downloaded on April 09,2022 at 09:04:44 UTC from IEEE Xplore. Restrictions apply.
WANG et al.: REAL-TIME STEEL SURFACE DEFECT DETECTION APPROACH WITH HIGH ACCURACY 5005610
TAB L E I V
COMPARISON WITH STATE-OF -THE-ART METHODS
Fig. 6. Three kinds of SCM styles. (a) Same level composition, named SCM_S. (b) Higher-level composition, SCM_H. (c) Lower level composition, SCM_L.
dec and pool refer to deconvolution and max pooling.
E. Comparison With State-of-the-Art
To evaluate the effectiveness of the proposed RDN, several
state-of-the-art one-stage and two-stage detection methods are
compared in the predictive mAP and the detection time per
image when the same NEU-DET dataset is used. Two-stage
approaches, i.e., Faster RCNN [24] and Cascade RCNN [38],
which use the region proposal network to generate candi-
date bounding box, are considered for comparison. Another
two-stage steel surface defect detection structure named DDN
presented by Song and Yan [12] has also been compared.
For one-stage methods, seven anchor-based and anchor-free
detectors, i.e., M2Det [39], SSD [40], FCOS [41], ATSS [42],
YOLOv3 [43], improved YOLOv3 [17], and CenterNet [44],
are implemented. Among them, multiple backbones are per-
formed to get more competitive detection precision or speed,
such as faster RCNN, DDN, M2Det and CenterNet, and
so on. The detection results of different models are shown
in Table IV.
It can be seen from Table IV that two-stage detectors can
achieve better detection accuracy, and the mAP of Faster
RCNN, Cascade RCNN, and DDN is 77.9%, 73.3%, and
82.3%, respectively. However, the inference process for each
image of these models is very slow. The detection speed
of the DDN method is 11 FPS when ResNet-50 is selected
as the backbone. In contrast, one-stage methods can get
better detection speed, but the detection precision is unstable
relatively. Therefore, in this work, a portable detector based on
the pretrained model is constructed. Compared with the DDN
which has the highest detection mAP of 82.3% and speed
of 11 FPS, this work introduces the SLC and PFM to improve
the detection precision and uses ResNet with 18 convolution
layers as the backbone to reduce the number of model para-
meters. As a result, our proposed method achieves the mAP of
80.0%, and the speed of 64 FPS, which demonstrates that the
RDN method proposed in this work achieves the best detection
speed in detecting steel surface defects.
V. D ISCUSSION
A. Comparisons of Different SCM Styles
In Section IV-C, it is briefly demonstrated that SCM
[as shown in Fig. 6(a)] can improve detection performance by
fusing features with the same resolution. However, whether
other combination manners of these layers may result in
better performance? To this end, this work designs two other
composite styles to integrate feature maps of different sizes,
as shown in Fig. 6(b) and (c).
Authorized licensed use limited to: Anhui University of Technology. Downloaded on April 09,2022 at 09:04:44 UTC from IEEE Xplore. Restrictions apply.
5005610 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 71, 2022
TAB L E V
DETECTION PERFORMANCE OF DIFFERENT COMPOSITION STYLES
TAB L E V I
DETECTION PERFORMANCE OF COMBING DIFFERENT PFM LAYE R S
It can be seen from Table V that the lower-level composition
style that combines the large feature maps in the encoding
network and the small feature map in the decoding network
obtains the mAP of 77.2% and the speed of 61 FPS. The
higher-level composition style that combines the small feature
maps in the encoding network and the large feature map in
the decoding network obtains the optimal mAP of 80.2%.
However, the deconvolution operation added in this style
reduces the detection speed of the detector to 59 FPS, which is
slower than the same level composition style. As a comparison,
SCM with the same level combination, named SCM_S, has the
mAP of 80.0% and the speed of 64 FPS, which can meet the
minimum speed requirements of different industrial production
scenarios.
B. Comparison of Different Combination PFM Layers
To explore which level of feature integration can achieve the
best precision improvement without reducing the resolution
of the output feature map, this work compares the detection
performance of fusing features of different levels, such as
conv5 module in encoding network, up-conv1, up-conv2 and
up-conv3 module in decoding network. As in Section V-A,
we also take ResNet18-dcn as the baseline. The detection
results of different integration manners are shown in Table VI.
It can be found that after introducing SCM, the mAP of
the model is 77.6%. With the increase of the introduced
feature map, the detection accuracy is improved in different
degrees, while the detection speed is not significantly reduced.
The mAP of merging two- or three-layer features is 77.8%,
78.6%, and 78.5%, respectively, and their speed bias is only
1 FPS. These show that the feature at different levels can
provide unique defect information and fusion more of them
is an effective way to improve the detection accuracy of steel
surface defects.
VI. CONCLUSION
In view of the high accuracy and low-speed problems of
current steel surface defect detectors, this work proposes a
real-time steel surface defect detection method, called RDN.
This is an end-to-end detection network that can provide defect
categories and precise locations. RDN uses a portable and
modular encoding and decoding network as the basic network
architecture to improve the detection speed. At the same
time, the SCM and PFM are designed to improve detection
accuracy. As a result, the detection method proposed in this
work achieves a real-time detection speed of 64 FPS and the
detection mAP of 80.0%.
Compared with the most state-of-the-art algorithms, our
model not only has faster detection speed but also has more
competitive detection accuracy and it can well identify defects
with a similar background. In addition, our model can detect
more defect morphology than the segmentation algorithm.
However, it cannot provide information on defect localization
in the pixel level as the segmentation algorithms can do. The
performance of our model is developed and obtained on the
NVIDIA Titan X GPU hardware platform. However, its detec-
tion speed may be affected by the computing power. Therefore,
a middle-end hardware device is a necessary prerequisite for
the application of our model.
In the future, our work can be improved in the following
aspects: 1) trying to use segmentation algorithm to get pixel-
level information of defects; 2) designing a detector to detect
the low-quality defect images independently to further improve
the accuracy of the model; and 3) try to embed our proposed
algorithm into mobile devices in the real production-line.
REFERENCES
[1] H. Di, X. Ke, Z. Peng, and Z. Dongdong, “Surface defect classification
of steels with a new semi-supervised learning method,” Opt. Lasers Eng.,
vol. 117, pp. 40–48, Jun. 2019.
[2] L. Xu, G. Tian, L. Zhang, and X. Zheng, “Research of surface defect
detection method of hot rolled strip steel based on generative adversarial
network,” in Proc. Chin. Autom. Congr. (CAC), Nov. 2019, pp. 401–404.
[3] J. Gao, W. Yu, and C. He, “The research on defect recognition method
for rail magnetic flux leakage detecting,” in Proc. Int. Conf. Meas., Inf.
Control, vol. 2, May 2012, pp. 745–750.
[4] Q. Luo, Y. Sun, P. Li, O. Simpson, L. Tian, and H. Yigang, “Generalized
completed local binary patterns for time-efficient steel surface defect
classification,” IEEE Trans. Instrum. Meas., vol. 68, no. 3, pp. 667–679,
Mar. 2019.
[5] K. Xu, S. Liu, and Y. Ai, “Application of Shearlet transform to
classification of surface defects for metals,” Image Vis. Comput., vol. 35,
pp. 23–30, Mar. 2015.
[6] K. Liu, H. Wang, H. Chen, E. Qu, Y. Tian, and H. Sun, “Steel
surface defect detection using a new Haar–Weibull-variance model in
unsupervised manner,” IEEE Trans. Instrum. Meas., vol. 66, no. 10,
pp. 2585–2596, Oct. 2017.
[7] Z. Xue-Wu, D. Yan-Qiong, L. Yan-Yun, S. Ai-Ye, and L. Rui-Yu,
“A vision inspection system for the surface defects of strongly reflected
metal based on multi-class SVM,” Expert Syst. Appl., vol. 38, no. 5,
pp. 5930–5939, May 2011.
[8] S. Mei, H. Yang, and Z. Yin, “An unsupervised-learning-based approach
for automated defect inspection on textured surfaces,” IEEE Trans.
Instrum. Meas., vol. 67, no. 6, pp. 1266–1277, Jun. 2018.
[9] Q. Luo et al., “Surface defect classification for hot-rolled steel strips
by selectively dominant local binary patterns,” IEEE Access,vol.7,
pp. 23488–23499, 2019.
[10] L. Yi, G. Li, and M. Jiang, “An end-to-end steel strip surface defects
recognition system based on convolutional neural networks,” Steel Res.
Int., vol. 88, no. 2, Feb. 2017, Art. no. 1600068.
[11] I. Konovalenko, P. Maruschak, J. Brezinová, J. Viˇnáš, and J. Brezina,
“Steel surface defect classification using deep residual neural network,”
Metals, vol. 10, no. 6, p. 846, Jun. 2020.
[12] K. Song and Y. Yan, “A noise robust method based on completed local
binary patterns for hot-rolled steel strip surface defects,” Appl. Surf. Sci.,
vol. 285, no. 21, pp. 858–864, Nov. 2013.
Authorized licensed use limited to: Anhui University of Technology. Downloaded on April 09,2022 at 09:04:44 UTC from IEEE Xplore. Restrictions apply.
WANG et al.: REAL-TIME STEEL SURFACE DEFECT DETECTION APPROACH WITH HIGH ACCURACY 5005610
[13] W. Wang et al., “Surface defects classification of hot rolled strip based
on improved convolutional neural network,” ISIJ Int., vol. 61, no. 5,
pp. 1579–1583, 2021.
[14] S. M. Azimi, D. Britz, M. Engstler, M. Fritz, and F. Mücklich,
“Advanced steel microstructural classification by deep learning meth-
ods,” Sci. Rep., vol. 8, no. 1, Dec. 2018, Art. no. 2128.
[15] S. Wang, X. Xia, L. Ye, and B. Yang, “Automatic detection and
classification of steel surface defect using deep convolutional neural
networks,” Metals, vol. 11, no. 3, p. 388, Feb. 2021.
[16] Y. He, K. Song, Q. Meng, and Y. Yan, “An end-to-end steel surface
defect detection approach via fusing multiple hierarchical features,”
IEEE Trans. Instrum. Meas., vol. 69, no. 4, pp. 1493–1504, Apr. 2020.
[17] M. Hatab, H. Malekmohamadi, and A. Amira, “Surface defect detection
using YOLO network,” in Intelligent Systems and Applications,K.Arai,
S. Kapoor, and R. Bhatia, Eds. Cham, Switzerland: Springer, 2021,
pp. 505–515.
[18] X. Kou, S. Liu, K. Cheng, and Y. Qian, “Development of a YOLO-V3-
based model for detecting defects on steel strip surface,” Measurement,
vol. 182, Sep. 2021, Art. no. 109454.
[19] W. Zhao, F. Chen, H. Huang, D. Li, and W. Cheng, “A new steel defect
detection algorithm based on deep learning,” Comput. Intell. Neurosci.,
vol. 2021, pp. 1–13, Mar. 2021.
[20] H. Dong, K. Song, Y. He, J. Xu, Y. Yan, and Q. Meng, “PGA-Net: Pyra-
mid feature fusion and global context attention network for automated
surface defect detection,” IEEE Trans. Ind. Informat., vol. 16, no. 12,
pp. 7448–7458, Dec. 2020.
[21] H. Wang, J. Zhang, Y. Tian, H. Chen, H. Sun, and K. Liu, “A sim-
ple guidance template-based defect detection method for strip steel
surfaces,” IEEE Trans. Ind. Informat., vol. 15, no. 5, pp. 2798–2809,
May 2019.
[22] K. Simonyan and A. Zisserman, “Very deep convolutional networks
for large-scale image recognition,” CoRR, vol. abs/1409.1556, pp. 1–14,
Apr. 2015.
[23] S. Xie, R. Girshick, P. Dollar, Z. Tu, and K. He, “Aggregated residual
transformations for deep neural networks,” in Proc. IEEE Conf. Comput.
Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 5987–5995.
[24] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards
real-time object detection with region proposal networks,” IEEE Trans.
Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, Jun. 2017.
[25] A. G. Howard et al., “MobileNets: Efficient convolutional neural net-
works for mobile vision applications,” 2017, arXiv:1704.04861.
[26] Y. He, K. Song, H. Dong, and Y. Yan, “Semi-supervised defect classifi-
cation of steel surface based on multi-training and generative adversarial
network,” Opt. Lasers Eng., vol. 122, pp. 294–302, Nov. 2019.
[27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for
image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jun. 2016, pp. 770–778.
[28] J. Dai, Y. Li, K. He, and J. Sun, “R-FCN: Object detection via
region-based fully convolutional networks,” in Proc. Adv. Neural Inf.
Process. Syst. (NIPS). Red Hook, NY, USA: Curran Associates, 2016,
pp. 379–387.
[29] R. K. Srivastava, K. Greff, and J. Schmidhuber, “Training very deep
networks,” in Proc. Adv. Neural Inf. Process. Syst. (NIPS),vol.2.
Cambridge, MA, USA: MIT Press, Dec. 2015, pp. 2377–2385.
[30] A. Veit, M. Wilber, and S. Belongie, “Residual networks are exponential
ensembles of relatively shallow networks,” in Proc. Adv. Neural Inf.
Process. Syst., May 2016, pp. 1–9.
[31] O. Ronneberger, “Invited talk: U-Net convolutional networks for
biomedical image segmentation,” in Bildverarbeitung für die Medi-
zin, K. H. Maier-Hein, K. Fritzsche, T. M. Deserno, T. Lehmann,
H. Handels, and T. Tolxdorff, Eds. Berlin, Germany: Springer, 2017,
p. 3.
[32] T. Lin, P. Goyal, R. B. Girshick, K. He, and P. Dollár, “Focal loss for
dense object detection,” CoRR, vol. abs/1708.02002, pp. 1–9, Aug. 2017.
[33] H. Law and J. Deng, “CornerNet: Detecting objects as paired keypoints,”
Int. J. Comput. Vis., vol. 128, pp. 642–656, Dec. 2020.
[34] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in
deep convolutional networks for visual recognition,” IEEE Trans. Pattern
Anal. Mach. Intell., vol. 37, no. 9, pp. 1904–1916, Sep. 2015.
[35] F. Yu, D. Wang, E. Shelhamer, and T. Darrell, “Deep layer aggregation,”
2017, arXiv:1707.06484.
[36] J. Li, Z. Su, J. Geng, and Y. Yin, “Real-time detection of steel strip
surface defects based on improved YOLO detection network,” IFAC-
PapersOnLine, vol. 51, no. 21, pp. 76–81, 2018.
[37] J. Pang, K. Chen, J. Shi, and H. Feng, “Libra R-CNN: Towards balanced
learning for object detection,” in Proc. IEEE/CVF Conf. Comput. Vis.
Pattern Recognit. (CVPR), Jun. 2019, pp. 821–830.
[38] Z. Cai and N. Vasconcelos, “Cascade R-CNN: High quality object
detection and instance segmentation,” IEEE Trans. Pattern Anal. Mach.
Intell., vol. 43, no. 5, pp. 1483–1498, May 2021.
[39] Q. Zhao, T. Sheng, Y. Wang, Z. Tang, Y. Chen, L. Cai, and H. Ling,
“M2Det: A single-shot object detector based on multi-level feature
pyramid network,” in Proc. AAAI Conf. Artif. Intell., vol. 33, 2019,
pp. 9259–9266.
[40] W. Liu et al., “SSD: Single shot multibox detector,” in Computer
Vision—ECCV 2016, B. Leibe, J. Matas, N. Sebe, and M. Welling, Eds.
Cham, Switzerland: Springer, 2016, pp. 21–37.
[41] Z. Tian, C. Shen, H. Chen, and T. He, “FCOS: Fully convolutional
one-stage object detection,” in Proc. IEEE/CVF Int. Conf. Comput. Vis.
(ICCV), Oct. 2019, pp. 9626–9635.
[42] S. Zhang, C. Chi, Y. Yao, Z. Lei, and S. Z. Li, “Bridging the gap between
anchor-based and anchor-free detection via adaptive training sample
selection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jun. 2020, pp. 9756–9765.
[43] J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,”
2018, arXiv:1804.02767.
[44] X. Zhou, D. Wang, and P. Krähenbühl, “Objects as points,” 2019,
arXiv:1904.07850.
Wen ya n Wa ng received the B.S. degree from the
School of Electrical Engineering, Industrial and
Commercial College, Anhui University of Technol-
ogy, Ma’anshan, China, in 2015, and the M.S. degree
from the School of Electrical Information and Engi-
neering, Anhui University of Technology, in 2018,
where she is currently pursuing the Ph.D. degree
with the School of Metallurgical Engineering.
Her research interests include deep learning, intel-
ligent inspection, data mining, and bioinformatics.
Chunfeng Mi received the B.S. degree from the
School of Electrical Engineering, Industrial and
Commercial College, Anhui University of Tech-
nology, Ma’anshan, China, in 2019, where he is
currently pursuing the M.S. degree with the School
of Electrical Information and Engineering.
His research interests include intelligent inspec-
tion, deep learning, and bioinformatics.
Ziheng Wu received the B.S. degree from
Tongling University, Tongling, China, in 2009, the
M.S. degree from Zhejiang Sci-Tech University,
Hangzhou, China, in 2012, and the Ph.D. degree
from the University of Science and Technology of
China, Hefei, China, in 2018.
He is currently a Lecturer with the Anhui Univer-
sity of Technology, Ma’anshan, China. His research
interests include machine learning, artificial intel-
ligence, gray systems theory, medical informatics,
science engineering, and intelligent control.
Kun Lu received the B.S. degree from the School of
Electrical Information and Engineering, Hohai Uni-
versity Wentian College, Ma’anshan, China, in 2016,
and the M.S. degree from the School of Electrical
Information and Engineering, Anhui University of
Technology, Ma’anshan, in 2019, where he is cur-
rently pursuing the Ph.D. degree with the School of
Management Science and Engineering.
He research interests include machine learning,
data mining, and bioinformatics.
Authorized licensed use limited to: Anhui University of Technology. Downloaded on April 09,2022 at 09:04:44 UTC from IEEE Xplore. Restrictions apply.
5005610 IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, VOL. 71, 2022
Hongming Long received the Ph.D. degree from
the Department of Iron Metallurgy, Central South
University, Changsha, China, in 2007.
He is the Director of the Key Laboratory of Met-
allurgical Emission Reduction and Resources Recy-
cling, Ministry of Education, Ma’anshan, China.
From 2007 to present, he works with the Anhui
University of Technology, Ma’anshan. His research
interest includes emission reduction of flue gas
pollutants (NOx, dioxin and PM2.5) in metallur-
gical industry, and comprehensive utilization of
metallurgical solid waste resources.
Mr. Long is a Member of Expert Committee of China Metal Society and a
Secretary-General of the Branch of Metallurgical Solid Waste Resource.
Baigen Pan received the B.S. degree from the
Department of Physics, School of Physics, Anqing
Normal University, Anqing, China, in 2003, and the
M.S. degree from the School of Instrument Science
and Opto-Electronics Engineering, Hefei University
of Technology, Hefei, China, in 2010.
His research interests include deep learning, data
mining, and bioinformatics.
Dan Li was born in 1976. She received the
bachelor’s and master’s degrees from the Anhui
University of Technology, Ma’anshan, China, in
1997 and 2004, respectively, and the Ph.D. degree
from the Nanjing University of Aeronautics and
Astronautics, Nanjing, China, in 2008.
She is mainly engaged in image processing,
machine vision, and autonomous navigation.
Jun Zhang received the bachelor’s degree from
the Hefei University of Technology, Hefei, China,
in 1995, the master’s degree from the Institute of
Intelligent Machine, Chinese Academy of Sciences,
Hefei, in 2004, and the Ph.D. degree from the
University of Science and Technology of China,
Hefei, in 2007.
He served with the University of Louisville,
Louisville, KY, USA, from 2009 to 2011, as a Post-
Doctoral Fellow. He is currently an Associate Pro-
fessor with the School of Electrical Engineering and
Automation, Anhui University, Hefei. He has published more than 40 articles
in international conferences and journals. He focuses on deep learning with
application to bioinformatics, cheminformatics, computer vision, and so on.
Peng Chen received the bachelor’s degree from the
Electronic Engineering Institute, PLA, Hefei, China,
in 1997, the master’s degree from the Kunming
University of Science and Technology, Kunming,
China, in 2003, and the Ph.D. degree from the
University of Science and Technology of China,
Hefei, in 2007.
He served with the City University of Hong Kong,
Hong Kong, in 2006, as a Senior Research Asso-
ciate, Howard University, Washington, DC, USA,
from 2008 to 2009, as a Post-Doctoral Fellow,
Nanyang Technological University, Singapore, from 2009 to 2010, as a
Research Fellow, and the King Abdullah University of Science and Tech-
nology (KAUST), Thuwal, Saudi Arabia, from 2012 to 2014, as a Post-
Doctoral Fellow. He is a Professor with the School of Computer Science and
Technology, Institute of Physical Science and Information Technology, Anhui
University, Hefei. He specializes in machine learning and data mining with
applications to bioinformatics, drug discovery, computer vision, and so on.
Bing Wang (Senior Member, IEEE) received the
B.S. and M.S. degrees from the Hefei University of
Technology, Hefei, China, in 1998 and 2004, respec-
tively, and the Ph.D. degree from the University of
Science and Technology of China, Hefei, in 2006.
He worked as a Senior Research Associate with
the City University of Hong Kong, Hong Kong,
from 2006 to 2007, and a Post-Doctoral Fellow with
the University of Louisville, Louisville, KY, USA,
and Vanderbilt University, Nashville, TN, USA,
from 2008 to 2012. He is currently serving as a Full
Professor with the School of Electrical and Information Engineering, Anhui
University of Technology, Ma’anshan. He has more than 150 publications.
His research interests mainly focus on machine learning, image processing,
computational biology, and chemoinformatics.
Authorized licensed use limited to: Anhui University of Technology. Downloaded on April 09,2022 at 09:04:44 UTC from IEEE Xplore. Restrictions apply.
