Content uploaded by Nur Intan Raihana Ruhaiyem
Author content
All content in this area was uploaded by Nur Intan Raihana Ruhaiyem on Nov 17, 2022
Content may be subject to copyright.
Image Modeling Through Augmented Reality
for Skin Allergies Recognition
Nur Intan Raihana Ruhaiyem(B)and Nur Amalina Mazlan
School of Computer Sciences, Universiti Sains Malaysia, USM, 11800 Gelugor,
Penang, Malaysia
intanraihana@usm.my
Abstract. Skin rashes and allergies are common on human body. To date, we
could find many skin care products sold not only in pharmacy but also from
individual business. However, not all products suitable for all skin types. As a
normal human, we sometimes not know the type of rashes or allergies that we
faced. Meeting dermatologist would not be the first choice for many patients –
given that the fees are expensive especially. Skin rashes can occur to anybody
and an early recognition could avoid the rash become worse. Seeking information
online would be the first choice, however patients still in high possibilities in
mistakenly buy skin care products. Therefore, the development of the augmented
reality application for skin rashes and allergies detection is expected can solve the
problem. With the help of dermatologist and healthcare people, the information
in this application is established and trustable. Among the advantages of this
application are the ability in detecting of different types of skin rashes, displaying
informative details on the detected skin rashes to reduce wrong judgement on the
allergies the patient faced, and reasonable processing speed on mobile screen.
Keywords: Augmented reality ·Skin rashes ·Image processing ·3D modeling ·
Mobile application
1 Introduction
Augmented reality is using technology to integrate the digital information from the
user’s environment in real time. By using the augmented reality, the application will
allow to overlay new information on top of the existing environment. 3-Dimensional
(3D) modeling or 3D program is the main feature in augmented reality application
as it will allow the developer to store the 3D animation or digital information in the
computer program to an augmented reality marker in the real world. When the device
of the augmented reality application receives digital information from a known marker,
the application will execute the marker’s code and layer the correct 3D modeling or
animation. In this research project, augmented reality will be used to solve the problem
of skin rashes. This research work needs to get the data for each type of skin rashes
and/or allergies, so that the application can differentiate the types of skin rashes, create
3D modeling for skin rashes and allergies and display the information on the mobile
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2021
F. Saeed et al. (Eds.): IRICT 2020, LNDECT 72, pp. 72–79, 2021.
https://doi.org/10.1007/978-3-030-70713-2_8
Image Modeling Through Augmented Reality 73
screen. To solve this problem, the augmented reality technology is used to process the
patient’s skin problem image through scanning. At the system interface the target object
will pop out on the focus area screen. The user needs to locate the camera at the targeted
skin rashes within the target object focus area screen. After a complete scanning, the
screen will display the expected 3D modeling of the skin rashes type to the user together
with the detailed information. Skin rashes or allergies can make someone feel itchy and
not comfortable. From baby to elderly are exposed to the skin rashes. There are about
50 types of rashes and in certain cases it looks exactly like the other type of rashes or
allergy. Many of which would look like to each other. Although users prefer to seek
information such as from Google image to search about the type of the skin rashes, but
mistakes would happen perhaps because of user’s perception.
Helping users to overcome the skin rashes problem in daily life has been the moti-
vation for this research work as well as drive to the development of this application.
This application is believed to help users who are very shy to ask about their skin prob-
lems, especially on their private part. The main objective of this system is to develop a
complete augmented reality application for skin allergies that can be used by users or
patients. The application system also equipped with special feature where the user is
able to get to see the expected 3D modeling of the skin rashes and allergies from the
scanning, which help the user to understand more what happen to their skin and the type
of the skin rashes. Moreover, this application is a new development which is expected
to be used by many types of users.
Augmented reality (AR) is a technology that produces an information including
processing 3D modeling from the user’s real time environment through Unity; a software
which utilized for a good visualization and interaction of mobile AR (Kim et al. 2014),
with large programming toolsets, (Eriksen et al. 2020) and able to improve the user
interface (Kim et al. 2014; Nuryono and Iswanto 2020). Image processing, 3D model
and database are the features in AR application. There are four special features of this
application, firstly, the rotation speed of the 3D modeling – this will ease user to explore
underneath the skin rashes (in video format). Secondly, information text display, for
example the information such as symptoms to help users differentiate the type of skin
rashes. Thirdly, the Vuforia target manager where the Vuforia SDK can detect and track
from the image targets which represent images. Last one is the 3D modeling which will
model 3D of that skin rashes to increase users’ understanding of the expected outcome
from the image processing (i.e. early diagnosis). Four modules offered in the system
development, tabulated in Table 1.
The application would give benefits and impact to end users such as it helps the
patient to get early diagnosis at home if the patient is too shy to go out, helps parents
identify baby’s skin rashes, and it is believed could save more time for multiple users,
such as the patients, parents as well as pharmacist. The uniqueness of the application
is there is no skin rashes application that use augmented reality where this allows 3D
model appear to give the probability or good percentage on early diagnosis using image
processing compared to website which generally provide only photos.
74 N. I. R. Ruhaiyem and N. A. Mazlan
Table 1. Description of system module.
Module Description
Image database The image will be store in database of the Vuforia Target
Manager. In Unity, the image will be the AR marker which
will trigger the 3D model to be display
3D model of type skin rashes 3D model of skin rashes is created and build by Blender as
the platform. The complete 3D model is imported into the
Unity. The 3D model will be setting with selected image
from database according the types of the skin rashes
Real-time skin allergy tracking Skin allergy is detected in real-time. The natural features
found in the image itself is detected using SDK by
comparing these natural features against a known target
resource database
Real-time skin allergy result The result of the skin allergy is in 3D model of the type skin
rashes or allergy. The expected 3D model is added (layer) to
the skin allergy of the user in real world through the AR
application. The information in forms of text also displayed
1.1 Related Work
As mentioned earlier, there is no similar mobile application for skin rashes detection
through augmented reality technology. There is one mobile application which is very
close to this project; called Doctor Mole Skin Cancer app, which is using the standard
Asymmetry, Border, Color, Diameter and Risk (ABCDE) approach in order to determine
and give instant risk feedback (Doctor Mole 2015). This app focusing on skin cancer
and detecting the malignant lesions. Thus, the detection and recognition techniques are
different. Doctor Mole is a medical app which used to detect skin cancer by using AR and
camera to scan and analyze the suspicious mole in real time. The captured photo is saved
and can be used again to see the evolution changes from time to time. Another similar
approach for AR technology is tracking with fine object segmentation (TFOS) which
originally proposed in year 1989, where it introduced the basic properties of three new
variational problems which are suggested by applications to computer vision (Mamford
and Shah 1989). In 2013, taking advantage of TFOS, a novel method for on-line, joint
object tracking and segmentation was introduced (Konstantinos and Antonis 2013).
Image Modeling Through Augmented Reality 75
1.2 Proposed System
Beside diagnosing the skin rashes or allergies in real time, the app also can be used to
educate patients and users about skin rashes or allergies through 3D modeling. The main
features in this app are; real time detection or scanning that allows the application being
used directly to the human body and display 3D model as the outcome or the result from
the diagnosis, the offline capabilities, and the application priority where the speed to
display the result to user is one of the application priority and the application will not
make the user to wait while the application loading.
2 System Analysis, Design, and Implementation
This application system can be a service or a product to the client or user. The main
features in this application; firstly, the real time detection – where the augmented reality
allows the application being used on human body or skin directly and display 3D model
as the outcome or result from the detection, and secondly the offline capability – where
the apps surely can be used without internet connection. The system capabilities are
including the detection speed and the 3D object modeling, where both will take advantage
of the smartphone camera (at least 5 MP and above for better detection and results). Like
any other application, this app also has its own limitation such as the application can’t
be used in the dark or insufficient light place, there is also no social media sharing
information and no sound integrated with this system. These limitations are something
can be focused on future. The architecture diagram depicted in Fig. 1shows the overview
of the app on how it works from the detection phase until the production of results. For
the image database –images of skin rashes are taken from the trusted website and from
collection of private photos. The images are stored in database of the Vuforia Target
Manager, where the features are tabulated in Table 2.
Table 2. Features of the Vuforia target manager.
Feature Description
Rating This rating is displayed in the Target Manager and the range rating from
0 to 5 for any given image. The higher the rating of an image target, the
stronger the tracking and detection ability it contains. Zero rating
indicates the image target is not tracked at all by the AR system. Rating
at five indicates that the image target is easily tracked by the AR system
Add Target Add the more the image target by uploading the image using the Add
Target button to the Target Manager database
Download Database The downloaded database in form of unity package allow all the image
target been import into the Unity
76 N. I. R. Ruhaiyem and N. A. Mazlan
Fig. 1. System architecture of the AR application on skin rashes and allergies.
Image Modeling Through Augmented Reality 77
3 System Testing and Evaluation
In Unity system, there is a play button to render the scene. Once it is tuned on, it will show
the result whether the app is working well or not. The system is considered successfully
working when the 3D model appears after the camera target the image, 3D model can be
rotated, and the text is displayed. Other scenario can occur such as with different target
image and different setting of lighting such as different values of hue/saturation used.
Generally, the test results are good as the speed of tracking and detection is fast.
4 System Interface Design
Unity is the main software used for development of the app where the images were
created to serve as AR marker (Fig. 2). Here, all settings including camera and image
position will be fine-tuned. For 3D models of the skin rashes which used for providing
extra information to users, Blender is applied. This software has the ability not only
in creating a static 3D image, but also capable to generate motion 3D graphic (Fig. 3).
As one of the objectives of this system is to produce an AR application with learning
tool on skin rashes, 3D model is produced for learning purpose once the rashes detected
and recognized (Fig. 4). Some information including the skin rashes, tips on how to
recognize them and tips to heal them will pop out on the screen as well. The evidence
that the application meets the requirement and work is in the application, the 3D model
is working, the database can be stored and uploaded in a package into Unity, the SDK
can detect and track the image and the 3D model rotation is working give the opportunity
to user to explore 360° under the skin.
Fig. 2. Unity interface showing the settings of AR camera (left side) and image position (right
side) and other settings important for AR marker.
78 N. I. R. Ruhaiyem and N. A. Mazlan
Fig. 3. Blender interface showing all available tools for 3D model development (left side) and
render settings (right side).
What are the symptoms
of Mosquito bite?
1. Puff y bump on the
skin immediately
after the bite
2. Reddish brown,
itchy bumps
3. Dark spots or
bruises caused by
itching
4. Mild fever and
body ache
Fig. 4. Interface of working AR app, 3D model together with the information will be pop up after
the skin rashes successfully detected (which prove that AR marker is working well).
5 Conclusion
Using AR as one of a new technology approach for medical field which easily can
be used by a lot of people with interaction in real time is something interesting to be
explored. Educating or displaying information through 3D model also something should
be widely used as it can give a real scenario in real life and easy to understand it (Loke
and Ruhaiyem 2020; Teh et al. 2020). Furthermore, this application also provides extra
benefit in educating users to know more about skin problems. The important findings
found is to know that Unity 3D can be used to create android application in cooperation
with AR technology. To create the 3D model, many 3D modeler software (e.g. Maya,
Blender, and 3D studio max) has been tested before Blender is chosen. In future however,
there are rooms for other technologies could be explored for better AR findings as
problem solver application.
Image Modeling Through Augmented Reality 79
References
Doctor Mole app 2015 Homepage. https://apkpure.com/doctor-mole-skin-cancer-app/com.rev
soft.doctormole. Accessed 02 Sep 2020
Eriksen, K., Nielsen, B.E., Pittelkow, M.: Visualizing 3D molecular structures using an augmented
reality app. J. Chem. Educ. 97(5), 1487–1490 (2020)
Kim, S.L., Suk, H.J., Kang, J.H., Jung„ J.M., Laine, T., Westlin, J.: Using Unity 3D to facilitate
mobile augmented reality game development: In IEEE World Forum on Internet of Things
(WF-IoT) (2014).
Konstantinos, E.P., Antonis, A.A.: Integrating tracking with fine object segmentation. Image Vis.
Comput. 31(10), 771–785 (2013)
Loke, H.K., Ruhaiyem, N.I.R.: A conceptual perspective in mathematics through augmented real-
ity and 3D image modeling application. In: 8th International Conference on Multidisciplinary
Research on European Proceedings of Social and Behavioral Sciences, pp. 613–620. European
Publisher (2020)
Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associated
variational problems. Commun. Pure Appl. Math. 42(5), 577–685 (1989)
Nuryono, A.A., Iswanto, A.M.: Comparative analysis of path-finding algorithm unrestricted virtual
object movable for augmented reality. Int. J. Sci. Technol. Res. 1(1), (2020)
Teh, Y.X., Ruhaiyem, N.I.R., Syed-Mohamad, S.M.: MYTOXAPP: A mobile system toxicology
emergencies through image processing. In: 8th International Conference on Multidisciplinary
Research on European Proceedings of Social and Behavioral Sciences, pp. 694–700. European
Publisher (2020)
