Multimodal Biometric Recognition System Using Face and Finger Vein Biometric Traits with Feature and Decision Level Fusion Techniques

Abstract

A lot of research is going on biometrics security systems due to the high increase in spoofing attacks. To provide enhanced security using biometric applications, researchers showed more interest in multimodal biometrics. Using Multimodal biometrics applications, the complex model structure can be designed which provides a low risk of a spoofing attack. This paper discussed a hybrid model designed using the multilevel fusion of multimodal biometrics. This model considered two biometrics modalities face and finger vein, and also two levels of fusion feature level and decision level. In this work five classifiers Ensemble discriminant, K-Nearest Neighbor, Linear Discriminant, Ensemble subspace K-Nearest Neighbor (ESKNN), and SVM for majority voting are used. In this work rich information image is created by up sampling the image using bilinear interpolation techniques. The proposed model advances the recognition rate over unimodal biometric systems. Index Terms-Multimodal, biometrics, feature level fusion, decision level fusion.

Full text

Abstract—A lot of research is going on biometrics security
systems due to the high increase in spoofing attacks. To provide
enhanced security using biometric applications, researchers
showed more interest in multimodal biometrics. Using
Multimodal biometrics applications, the complex model
structure can be designed which provides a low risk of a
spoofing attack. This paper discussed a hybrid model designed
using the multilevel fusion of multimodal biometrics. This
model considered two biometrics modalities face and finger vein,
and also two levels of fusion feature level and decision level. In
this work five classifiers Ensemble discriminant, K-Nearest
Neighbor, Linear Discriminant, Ensemble subspace K-Nearest
Neighbor (ESKNN), and SVM for majority voting are used. In
this work rich information image is created by up sampling the
image using bilinear interpolation techniques. The proposed
model advances the recognition rate over unimodal biometric
systems.
Index Terms—Multimodal, biometrics, feature level fusion,
decision level fusion.
I. INTRODUCTION
A lot of terrorist activities are going on around the world.
To provide a high-security system, there are a lot of
limitations like noisy data, the differences in intra-class and
inter-class, etc. are faced in the unimodal biometric systems.
To overcome the problems faced in the unimodal biometric
systems, multimodal biometric recognition is introduced.
More than one biometric trait is used in this model.
Multimodal provides very rich information compared to
unimodal. The rich information provided by multimodal is
very much useful to overcome the drawbacks of unimodal
systems. The information received from multimodal
biometrics can be combined at various levels of fusion. For
person identification or recognition system, various types of
biometrics traits can be used. All biometrics traits are
grouped into physiological, behavioral, and soft biometrics [1,
2]. Under these three categories, various biometric traits are
classified. Fig. 1 shows three groups of biometric traits. From
these categories of biometrics, more than one biometric trait
is used in the Multimodal Biometric recognition systems.
This work shows the improvement of the recognition
performance of a biometric system using more than one
biometric trait. This work also concentrated on the use of
Multi-levels of fusion of different biometric traits.
Manuscript received March 1, 2021; revised June 1, 2021.
Arjun B. C. and H. N. Prakash are with Rajeev Institute of Technology,
Hassan, Affiliated to Visvesvaraya Technological University, Karnataka,
India (e-mail: bc.arjun@gmail.com, prakash_hn@yahoo.com).
Fig. 1. Physiological, behavioral and soft biometric traits (sources: Google
Image [ImaG]).
Arjun B. C. and H. N. Prakash
Multimodal Biometric Recognition System Using Face
and Finger Vein Biometric Traits with Feature and
Decision Level Fusion Techniques
International Journal of Computer Theory and Engineering, Vol. 13, No. 4, November 2021
123
DOI: 10.7763/IJCTE.2021.V13.1300

II. RELATED WORK
Mohammad et al. [3] discussed feature level fusion using
Discriminant Correlation Analysis for multimodal biometrics
and obtained good results. Madasu et al. [4] used three
different biometric traits and applied score level fusion and
obtained FAR of 0.01%. Chaudhary et al. [5] showed using
multilevel fusion recognition rate can be increased.
Sangeetha et al. [6] and Anil jain et al. [7] created their own
data set and explored different fusion levels. Arjun et al. used
bilinear enhanced data samples for feature level fusion and
obtained good results [9]. Arjun et al. discussed SNBI
samples 1:4 up-sampled data concatenation which gave a
high recognition rate at feature level fusion [10]. Table I
shows the survey of algorithms and fusion methods used at
various levels on different biometric traits.
From the above literature survey, it has been observed that
using different levels of fusion biometric recognition rates
can be increased, and also not much work is done in
multilevel fusion with the combination of face and finger
vein for standard data sets.
TABLE I: SURVEY ON BIOMETRIC FUSION LEVEL
References
Biometric Traits used
Used Algorithms
Dataset
Methods of Fusion
[3]
Face and Ear
DCA
WVU
Feature level
[4]
Hand geometry, Palm-print ,
Hand vein
Frank t-norm
IITD PolyU XM2VTS
Score level
[5]
Palmprint , dorsal hand veins
Sum rule, product rule, hamacher
t-norm, frank t-norm
I.I.T. Delhi, Bosphorus
Feature level & score
level
[6]
Iris, Fingerprint
Gabor wavelets, Chain Code based
feature extractor with contour
following to detect minutiae
Own created
Score Level
[7]
Fingerprint, Face, Speech
Minutia, Eigen face, HMM and LPC
Own dataset created
Decision Level
[8]
Finger knuckle, finger vein
FFF Optimization, Repeated line
tracking, k-SVM
I.I.T. Delhi,
SDUMAL-HMT
Feature level & score
level
[9]
Face and Signature
Bilinear enhanced data
Concatenation
SDUMLA-HMT
Feature Level
[10]
Finger Vein and Iris
SNBI samples 1:4 up sampled data
concatenation
SDUMLA-HMT
Feature Level
III. DATA BASES
Data sets of a face used are AT& T Cambridge standard
databases. Faces of six different poses of each individual of
forty persons are considered. A total of 240 samples are used.
Fig. 2. AT& T Cambridge standard face database samples.
Fig. 3. Shandong University (SDUMLA) finger vein standard database
samples.
Data sets images are in .pgm format shown in Fig. 2. Three
samples are used for training and 3 samples for testing each
individual.
Data sets of finger vein used are Machine Learning and
Data Mining Lab, Shandong University (SDUMLA) standard
databases. Finger vein of six left indexed finger with different
position of each individual of forty persons is considered. A
total of 240 samples is used. Data sets images are in .bmp
format shown in Fig. 3. Three samples are used for training
and 3 samples for testing each individual.
IV. METHODOLOGY
In this work face and finger vein, Biometric data sets are
used. Initially, uniform local binary pattern features are
extracted from each data set of face and finger vein. For
Unimodal Biometric system after extracting Local Binary
Pattern of both face and finger vein data sets are trained and
tested separately using ensemble subspace discriminant and
K-nearest neighbor classifier using decision level fusion
AND and OR operations as shown in Fig. 4. The same
procedure is applied for bilinear interpolated up-sampled data
sets and results are compared.
In the proposed model both standard data sets and 1:2 ratio
up-sampled datasets are trained separately and tested in
unimodal. And for Multimodal biometrics standard data sets
and 1:2 ratio up-sampled are fused at feature level using
concatenation technique.
International Journal of Computer Theory and Engineering, Vol. 13, No. 4, November 2021
124

(a)
(b)
(c)
Fig. 4. Proposed research work modal for unimodal biometrics using Decision level fusion (a) General model (b) face biometric model (c) finger vein model.
(a)
(b)
Fig. 5. Proposed research work modal for multilevel fusion of Feature and Decision level fusion (a) General model (b) face and finger vein multi model.
Above experimentation falls under three major steps
features extraction, feature level fusion, and decision level
fusion.
Fig. 5. shows the proposed research model for a multilevel
fusion of Feature and Decision level fusion where decision
rule is AND and OR is used. And further research work is
extended to Decision level fusion where majority voting
decision rule is used with different machine learning
algorithms.
A novel framework modal is designed of face and finger
vein data sets with Uniform Local Binary pattern features of
Bilinear interpolated data sets are fused in feature level and
decision level using five classifiers for majority voting.
Fig. 4 shows the proposed research modal for with only
decision level fusion. The data base samples are enhanced
using bilinear interpolation technique and from enhanced
International Journal of Computer Theory and Engineering, Vol. 13, No. 4, November 2021
125

sample local binary patterns are extracted. These features of
face and finger vein are trained and tested separately as
shown in Fig. 4. In decision level AND and OR techniques
are used.
Fig. 5 shows the proposed research modal for a multilevel
fusion of Feature and Decision level fusion. In this proposed
model both feature level and decision level techniques are
applied. After extracting features of face and finger vein,
using feature level fusion technique face and finger vein
features are combined to obtain new feature vector. This new
feature vector is used in decision level fusion as shown in Fig.
5.
The proposed research model for multilevel fusion is
further extended to Decision level fusion where majority
voting decision rule is used with different machine learning
algorithms.
Above experimentation falls under three major steps
features extraction, feature level fusion, and decision level
fusion.
A. Feature Extraction
Local binary pattern features are extracted from individual
data samples of face and finger vein separately. The Same
method is applied for up-sampled bilinear interpolated data
samples. Up-sampled data means an increased resolution by
two times of the actual image [9, 10].
B. Feature Level Fusion
Using the concatenation feature level method all the
features are joined together which results in a single vector of
each sample of both face and finger vein [9, 10].
C. Decision Level Fusion
Fused data is set as an input to five classifiers Ensemble
subspace discriminant (ESD), K-Nearest Neighbor (KNN),
Linear Discriminant (LD), Ensemble subspace K-Nearest
Neighbor (ESKNN) and Support Vector Machine (SVM),
After obtaining the results of classifier AND and OR decision
level method [11] and Majority voting decision level method
[11] is applied and results are plotted.
V. EXPERIMENTAL RESULTS
The unimodal experiments results are shown in Table II
where experiments are conducted by applying decision level
fusion technique. The four models were designed and
explored, in 1st model and 2nd model only face biometric is
used, classification accuracy showed better results for
enhanced images. From the model 3 and 4 it is observed that
original data base image showed better results for finger vein
samples.
From the Table II, it has been observed that Up-sampled
data sets using bilinear interpolation technique data samples
got good results compared to standard data sets for Face
biometrics. Classification accuracy of ensemble subspace
discriminant classifier showed more accuracy for up-sampled
Finger vein data sets compared to standard data sets.
Overall for unimodal biometric proposed modal
multi-level fusion at feature level concatenation technique
and decision level OR technique showed better classification
accuracy, results are shown in Table II and Fig. 6.
TABLE II: UNIMODAL BIOMETRIC SYSTEM COMPARISON OF
CLASSIFICATION ACCURACY
Fig. 6. Classification accuracy of unimodal biometric system.
From the Fig. 6 we can observe that finger vein samples
shows better performance when compared with face samples,
when unimodal experiments are conducted. And also in Fig.
6. Finger vein unimodal experiment with OR technique
showed more classification accuracy.
The Table III shows the experiments results of multimodal
biometrics with multilevel fusion. Two models were
developed, model 1 for standard database and model 2 for
enhanced data base. The enhanced samples after interpolation
technique performed well over standard database. From the
Table III we can observe that multimodal fusion of face and
finger vein for enhanced samples showed better results. That
is model 2 performed well over model 1 listed in Table III.
TABLE III: MULTIMODAL MULTI LEVEL FUSION BIOMETRIC SYSTEM
COMPARISON OF CLASSIFICATION ACCURACY
From the Fig. 7 we can observe that multimodal biometric
face and finger vein enhanced samples shows better
performance when compared with standard samples, when
multimodal experiments are conducted. And also in Fig. 7,
Face and Finger vein multimodal experiment with OR
technique showed more classification accuracy.
International Journal of Computer Theory and Engineering, Vol. 13, No. 4, November 2021
126

Fig. 7. Classification accuracy of Multimodal biometric system.
Experiments conducted on the multimodal biometric
system for ensemble and KNN classifier in this case both
feature level and decision level technique is used. AND and
OR decision level technique is applied. OR operation showed
good results shown in Table III and Fig. 7.
The proposed Multimodal Multi-level fusion Biometric
system using feature level and Majority Voting decision level
fusion showed good results after experimentation at various
levels. Experiments are setup for choosing variation in
classifiers at decision level. Initially three classifiers are used
for decision making, in the next stage 4 classifiers are used
and in the last stage 5 classifiers are used. Majority voting of
2 when 4 classifiers are considered accuracy rate is increased,
for the same model up-sampled data sets are used which
shows improvement. All the experiment variations and
results are shown the Table IV and Fig. 8. In all the cases
ensemble subspace discriminant classifier showed better
result.
TABLE IV: MULTIMODAL MULTI LEVEL FUSION BIOMETRIC SYSTEM USING FEATURE LEVEL AND MAJORITY VOTING
DECISION LEVEL FUSION COMPARISON OF CLASSIFICATION ACCURACY
Fig. 8. Classification accuracy of Multi-level fusion biometric system.
Three models were designed as shown in Table III, for all
the models two experiments were conducted both for actual
images and enhanced images of face and finger vein. In first
model four machine learning classifier is used and 2 out of 4
decision is considered. And in second model 2 out of 3
decisions and for last model 3 out of 5 classifier decision are
considered.
From the experiments listed in Table III, we can observe
that if 50% or less than 50% weightage is considered then
classification accuracy will increase. But if greater than 50%
weightage is given then classification accuracy will reduce.
Experiments were conducted on two levels of fusion,
feature level, and decision level. Initially, the feature level
concatenation method is applied and in the next stage,
decision level fusion majority voting is done. Variation in the
International Journal of Computer Theory and Engineering, Vol. 13, No. 4, November 2021
127

number of classifiers is considered for majority voting and
results are plotted. From the experimentation, it has been
observed that 50% voting out of four classifiers for
up-sampled data with both feature level and decision level
fusion got good results as shown in Table IV and Fig. 8.
VI. CONCLUSION
In this work, two biometric traits face and finger veins are
considered. Both biometrics traits are considered from the
standard databases, face data sets from AT&T, and finger
vein data sets from SDUMLA-HMT. In this work, multilevel
fusion of two biometrics with both standard data sets and
up-sampled data sets are used. Bi linear method is used to
up-sample the data sets. Experiments showed good results for
multimodal biometrics when multilevel fusion is applied.
When OR and AND method is applied at decision level for
up-sampled feature level fusion of face and finger vein got
high classification accuracy rate. The Majority voting at the
decision level for up-sampled feature level fused image of a
face and finger vein also showed a low classification error
rate. The above experiment shows that the multilevel fusion
of multimodal biometrics for up-sampled data sets of the face
and finger vein method will give high classification accuracy.
Compared to previous experimentation in this work bigger
data sets are used. Further using different biometric traits and
their up-sampled images in multilevel fusion techniques
classification rate can be increased.
CONFLICT OF INTEREST
The authors declare no conflict of interest.
AUTHOR CONTRIBUTIONS
Author Arjun B. C. performed the experimentation of the
work by identifying the new method for data fusion. Author
Prakash H N provided guidelines for using techniques and
evaluated their performance. Both have approved the final
version.
REFERENCES
[1] A. K. Jain, P. Flynn, and A. A. Ross, “Handbook of biometrics,”
Springer Science and Business Media, 2008.
[2] A. K. Jain, A. A. Ross, and K. Nandakumar, “Introduction to
biometrics,” Springer Science, 2011.
[3] Mohammad et al., “Discriminant correlation analysis: Real-time
feature level fusion for multimodal biometric recognition,” IEEE
Transactions on Information Forensics and Security, vol. 11, issue 9,
September 2016.
[4] M. Hanmandlu, J. Grover, A. Gureja, and H. M. Gupta, “Score level
fusion of multimodal biometrics using triangular norms,” Pattern
Recognition Letters, vol. 32, issue 14, pp. 1843-1850, 2011.
[5] Chaudhary et al., “Multi-level fusion of palmprint and dorsal hand
vein,” Information Systems Design and Intelligent Applications, vol. 3,
pp. 321-330, 2016.
[6] S. Sangeetha and N. Radha, “A new framework for IRIS and
fingerprint recognition using SVM classification and extreme learning
machine based on score level fusion,” in Proc. 7th International
Conference on Intelligent Systems and Control, 2013.
[7] A. Jain, L. Hong, and Y. Kulkarni, “A multimodal biometric system
using fingerprint, face and speech,” in Proc. International Conference
on Audio and Video Based Biometric Person Authentication, pp.
182-187, 1999.
[8] S. Veluchamy and L. R. Karlmarx, “System for multimodal biometric
recognition based on finger knuckle and finger vein using feature-level
fusion and k-support vector machine classifier,” IET Biometrics, vol. 6,
no. 3, pp. 232-242, 2017.
[9] B. C. Arjun and H. N. Prakash, “Multimodal biometric recognition:
fusion of modified adaptive bilinear interpolation data samples of face
and signature using local binary pattern features,” International
Journal of Engineering and Advanced Technology, vol. 9, issue 3,
February 2020.
[10] B. C. Arjun and H. N. Prakash, “Feature level fusion of seven neighbor
bilinear interpolation data sets of finger vein and iris for multimodal
biometric recognition,” International Journal of Advanced Trends in
Computer Science and Engineering, vol. 9, no. 2, March-April 2020.
[11] A. Ross, K. Nandakumar, and A. K. Jain, Handbook of Multibiometrics,
Berlin, Germany: Springer, 2006.
Copyright © 2021 by the authors. This is an open access article distributed
under the Creative Commons Attribution License which permits unrestricted
use, distribution, and reproduction in any medium, provided the original
work is properly cited (CC BY 4.0).
Arjun B. C. received degree in bachelor of computer
science and engineering from Visvesvaraya
Technological University, Belgaum, Karnataka, India
and M.Tech in industrial automation and robotics
from Visvesvaraya Technological University,
Belgaum, Karnataka, India. Currently he is pursuing
Ph.D. in computer science and engineering at
Visvesvaraya Technological University, Belgaum,
Karnataka, India. He is currently working as an
assistant professor in Department of Information Science & Engineering at
Rajeev Institute of Technology, Hassan Karnataka, and India. He has 2 year
of industry experience and 9 years of teaching experience. His research
interest includes image processing, pattern recognition, artificial
intelligence, artificial neural network, deep learning and machine learning.
he has published papers in national conferences and journal.
H. N. Prakash received degree in bachelor of
electronics and communications and engineering
from Mysore University, Karnataka, India, M.Tech in
electronics instrumentation from National Institute of
Technology Warangal, Telangana, India and Ph.D in
computer science from Mysore University,
Karnataka, India. He is currently working as a
professor and the head in Department of Computer
Science & Engineering at Rajeev Institute of Technology, Hassan,
Karnataka, India. He has 29 years of teaching experience. His research
interest includes image processing, pattern recognition, artificial
intelligence, artificial neural network and machine learning. he has published
many papers in national and international conferences and journals including
IEEE Transaction in Pattern Recognition and Machine Intelligence.
International Journal of Computer Theory and Engineering, Vol. 13, No. 4, November 2021
128