Content uploaded by Ahmed Mohammed Alghamdi
Author content
All content in this area was uploaded by Ahmed Mohammed Alghamdi on Mar 06, 2024
Content may be subject to copyright.
Journal of Autonomous Intelligence (2024) Volume 7 Issue 4
doi: 10.32629/jai.v7i4.1502
1
Original Research Article
Using augmented reality and deep learning to enhance tourist
experiences at landmarks in Makkah
Adel Bahaddad1, Khalid Almarhabi2, Ahmed Alghamdi3,*
1 Department of Information System, Faculty of Computing and Information Technology, King Abdulaziz University,
Jeddah 21589, Saudi Arabia
2 Department of Computer Science, College of Engineering and Computing in Al-Qunfudah, Umm Al-Qura University,
Makkah 24381, Saudi Arabia
3 Department of Software Engineering, College of Computer Science and Engineering, University of Jeddah, Jeddah
21493, Saudi Arabia
* Corresponding author: Ahmed Alghamdi, amalghamdi@uj.edu.sa
ABSTRACT
Communicating with visitors in tourist areas is one of the best means of conveying tourist information to them
and introducing and presenting these areas to end users. Therefore, the use and activation of a new technical and
digital service will help to deliver appropriate and reliable information to end users even if they speak different
languages. With the current rapid pace of the industrial revolution, there is an increasing need to create a space to
deal consistently with tourism in general. Therefore, innovation is gaining importance when it comes to the creation
and utilisation of emerging technologies to promote tourism goals. Augmented reality (AR) has revitalised many
areas by delivering immersive experiences in the digital world and bringing them to life in the real world. This
proposed study sought to enrich the experience of users by displaying various tourist spots in the Makkah region to
them with the relevant multimedia information to enable them to build a better connection with the archaeological
areas and sites in the city of Makkah, which is the religious capital of the Kingdom of Saudi Arabia (KSA) and is
considered as the cradle of Islam. This was where the Islamic civilisation was launched and the call of the prophet,
peace and blessings be upon him, began, and there are many areas that are rich in ancient history, where diverse
situations and information can be presented in a beautiful and attractive way. This study proposed the use of
electronic glasses linked to a smart device application based on the use of AR to review archaeological areas using
deep learning (DL) and multimedia information that support visitors through a database that was previously fed by
databases dedicated to this matter, as well as by using some websites and online videos for the same purpose. A
convolutional neural network (CNN) was used by sensors attached to the glasses to correctly identify artifacts and
thus, display information associated with the sites in question. To increase the level of accuracy, feedback was
obtained through a questionnaire that carefully evaluated the presented information using relevant evaluation models
through a place experience scale (PES) as well as the experience of using the triple interaction of the AR. The results
of the study were discussed and evaluated comprehensively for its future development using statistical methods. The
results of the study will serve to enhance competitiveness by showing the archaeological monuments in the Makkah
region and providing visitors with reliable information about them through multiple media that will automatically
identify what is presented to them according to the different languages of the visitors.
Keywords: deep learning; tourism; digital transformation; augmented reality; Saudi Arabia
1. Introduction
1.1. Tourist experiences at landmarks in Makkah
2
Makkah al-Mukarramah is the holiest city for Muslims around
the world. The Holy Mosque and the Kaaba, which is the direction
(qiblah) that Muslims face when they pray five times a day, are found
here. It is located in the western area of the Kingdom of Saudi Arabia
(KSA), 72 km from Jeddah city and the Red Sea coast. The nearest
seaport to Makkah is the Jeddah Islamic Port and the nearest
international airport is the King Abdulaziz International Airport in
Jeddah, too. The size of the central area surrounding the Holy Mosque
is about 6 km2[1].
In the beginning, it was just a small village located in a dry valley,
surrounded by mountains on every side. Then, people began to come
and settle there during the era of the Prophet Abraham, after he left
his wife, Hajar, and his son, Ismail, in that place[2]. Hajar and her son
complied with the command of Allah and stayed in the valley, and by
a divine miracle, the Zamzam well emerged in that place. During that
period, the bases of the Kaaba were raised by the hands of the Prophet
Abraham and his son Ismail, peace be upon them[3].
According to statistics, the population of Makkah in 2015 was
about 1,578,722 people[2]. They were distributed among the old and
new districts of Makkah. Makkah includes many sacred Islamic
monuments, such as the Holy Mosque, the Kaaba, Mount Safa, the
Black Stone, Mount Thor, Hira Cave, and many other landmarks. It
represents the holiest places on earth for Muslims, due to the direction
of the Kaaba faced by Muslims in their prayers. Also, Makkah is the
destination of Muslims during the two main seasons of the Umrah
throughout the year, as well as the Hajj, which represents the fifth
pillar of Islam[1]. Therefore, Muslims from all over the world visit it
annually, making it the largest annual event for Muslims in the KSA.
Currently, around 1.8 to 2.5 million people come to perform the
Hajj every year. The number of pilgrims decreased significantly due
to the coronavirus pandemic between 2020 to 2021 as countries
around the world resorted to taking precautionary and preventive
measures that included suspending international travel and restricting
movement locally, especially with the rapid spread of the virus and
the inability to provide treatment for it, which made it very difficult
to control as shown in Figure 1[4]. Therefore, the KSA took early
precautionary steps to preserve human safety, and this included
temporarily suspending the Umrah and pilgrimages in general, and
temporarily closing the two Holy Mosques in Makkah and Madinah
for periodic and intensive sterilisation purposes[5].
The number of tourist attractions in Makkah exceeds 100,
according to the statistics of the Saudi Ministry of Tourism[1]. A
number of books that focus on the various religious monuments in
Makkah show a much larger number than that, but some of these
monuments have disappeared over time, and some of them are more
ARTICLE INFO
Received: 21 December 2023
Accepted: 12 January 2024
Available online: 4 March 2024
COPYRIGHT
Copyright © 2024 by author(s).
Journal of Autonomous Intelligence is
published by Frontier Scientific Publishing.
This work is licensed under the Creative
Commons Attribution-NonCommercial 4.0
International License (CC BY-NC 4.0).
https://creativecommons.org/licenses/by-
nc/4.0/
3
than 3000 years old[5,6].
Tourist attractions are of great value to the KSA’s Vision 2030. One of major ways by which knowledge
is shared and disseminated to visitors to the KSA is through its rich culture and religion. Also, the KSA has
the most important cities of a religious nature in the world, namely Mecca and Medina. These two cities have
made the KSA one of the main destinations for tourists, especially Muslims, from all over the world. Therefore,
one of the transformation programmes in the vision of the KSA is the shift towards tourism and heritage[7–9].
Additionally, the KSA represents the first destination for religious tourism in the world, and it has great
geographical and historical potential, and enormous natural resources that make it an important tourist
destination.
Figure 1. The number of pilgrims during the last ten years[4].
The Ministry of Tourism has made great efforts to facilitate tourism in the KSA. They were issuing more
than 450,000 visas per year before the coronavirus pandemic, and they are now facilitating the electronic
processing of tourist visas. The Saudi Ministry of Tourism has also announced several tourist attraction
initiatives. They have promotions to important tourist sites and destinations around the KSA, including
Makkah, which display the efforts of the KSA in tourism[10].
Tourism is the top pillar in the KSA’s Vision 2030 as tourism contributes to the diversification of the
national economy, and is one of the most important alternatives to a post-oil economy. It provides many job
opportunities, and is an important way to attract investments to the KSA. The aim of the KSA is to increase
the number of annual visits to the KSA to about one hundred million visitors by 2030, of which pilgrims
represent no less than 50%, in addition to reaching a target of 30 million pilgrims for the performance of the
Hajj by 2030. This vision will create more than one million job opportunities, and it is expected to achieve an
estimated increase of 10% annually in the domestic product of the KSA[11].
1.2. The correlation between technology and tourist attractions
Countries around the world are interested in preserving the legacies of ancient civilisations and tourist
attractions that have a historical dimension, where they are located within their geographical borders. They
give the countries an important archaeological value that helps to increase their economic value[7]. They are
also important cultural symbols that are taken care of by both international agencies and organisations, which
will help to increase the power of the economy and tourism in the future[4].
The latest technology has been adopted with tourist attractions in public places or museums in order to
make dealing with civilisations more useful and usable. Therefore, smartphone technologies play a vital role
in this fourth industrial revolution as they have the ability to reach the general public and use artificial
intelligence (AI) to enhance the user experience with touristic monuments[8]. Promising new technologies help
1980249
2085238
1952817
1862909
2352122
2368873
2489406
10000 58745
926062
1845045
0
500000
1000000
1500000
2000000
2500000
3000000
2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023
4
to achieve and enhance the experiences of visitors and fill the gap between visitors and landmarks using various
technologies such as AR, virtual reality (VR), and deep learning (DL) by providing a better level of interaction
in a three-dimensional (3D) space that can be used in an experimental industrial environment[9]. Additionally,
great importance is given to the customer experience with regard to the removal of boundaries between tourists
and monuments to provide a rich interactive experience using AR. Can existing AR frameworks provide an
improved experience for visitors at open-air attractions? Also, tools that depend on AR to increase the level of
satisfaction in the customer experience help to develop AR that is compatible with smartphones, and therefore,
the big questions that arise here are: What are the ways and means that can be employed to gain benefits from
the current online content used for landmarks? How can the scientific interest and historical background of it
be maximised as required?
1.3. The correlation between technology and tourist attractions
Augmented reality (AR) is a technology that is based on the use of a number of virtual objects and
information in a real environment to provide additional information or direction, in contrast to VR, which is
based on the use of real objects in a virtual environment[3]. The user can manipulate the information and virtual
objects in AR through several devices, whether they are mobile devices such as a smartphones or wearable
devices such as glasses, contact lenses, and tracking system devices that provide accurate projections and
tracking such as global positioning (GPS), and as inputs that are interacted with through applications[9].
Furthermore, AR researchers seek to display virtual objects in an integrated manner with the user’s real
environment in order to improve the user’s vision as required[8]. The development of devices, whether portable
or wearable, plays a major role in shaping the future of AR and improving user interaction with its applications.
Smartphone devices are designed to contain the infrastructure that is needed by AR applications for
sensors as well as technologies such as GPS, compasses, vibrating motors, and cameras. Smartphones have a
prominent role to play in the future development of AR as it has been discovered that most people download
applications on their smartphones, regardless of whether they have a high level of cognitive ability in the field
of AR or not. There has been a huge development in the number of existing applications that are downloaded
in view of the list of software downloads in smart devices within the scope of this research.
In this study, AR techniques were utilised to deal with various applications in smart devices that interact
with archaeological sites and the real presence on the ground. Also, advantage was taken of mobile cameras
or smart glasses to help to distinguish between sites and to provide the required information within a business
model in the extensive review presented in the following sections.
1.4. The importance of deep learning (DL)
Artificial intelligence (AI) experts believe that computers work routinely to model, simulate, design, and
improve computational and educational processes by means of algorithms that are performed using various
software tools[12], whereby diverse solutions with a high level of intelligence are found and collected to increase
the practical experience of these algorithms. Thus, these solutions are transferred to computers, which then
solve problems with a high level of professionalism, complexity, and sufficient ability to research and verify
the presented solutions[9].
Therefore, AI systems will use deep search methods to determine the best path to convert a set of
information into interrelated information to answer target questions. Thus, DL can be defined as a part of
machine learning (ML) that gives computers the ability to learn without programming so the machine can
imitate intelligent human behaviour without being explicitly programmed. Additionally, DL varies in its uses
in a number of different fields, such as in cars, space, manufacturing, electronics, medical research, and other
fields. Also, DL is used in self-driving cars to automatically detect road signs and pedestrians[9].
5
Deep learning (DL) requires a strong interdependence to be able to simulate the way the mind works with
less human intervention, as DL alone ensures the process of extracting characteristics or features. Therefore, a
high level of computer performance is required with DL to process a very huge amount of data. Clearly, the
large amount of data will reduce human intervention in the process of clarifying the characteristics or features
of the machine. Thus, the learning through DL may turn into a disadvantage because the logic of the machine
may be very different from human logic. So, the most prominent applications of DL are:
• Deception detection techniques: These technologies have become necessary in the era of social media platforms as DL
algorithms help detect misleading and false news, and then, sort them in a smarter way to prevent confusing public opinion
and relevant issues that have occurred in public life.
• Natural language processing (NLP): DL engineers work to make the machine understand the language spoken by humans
through understanding the grammatical rules, expressions, and meanings related to words, as well as by acquiring the ability
to issue the appropriate and required responses as a person does. It is possible to benefit from learning verbal commands and
dialogues with the psychological state and nature of the question.
• Visual recognition: It works to recognise faces. This is a very complex field that involves enabling the machine to see and
deal with an image in a more intelligent and professional manner. With visual recognition, the machine is capable of
distinguishing and identifying human facial images as well as handwriting, where handwriting recognition is a sub-field of
visual recognition.
• Entertainment services through social media platforms or various service companies such as Google, Facebook, Netflix,
Amazon, and YouTube provide a more personalised service to their customers.
This study focused on the use of techniques and algorithms to display information related to historical
facts and monuments in general. It can help to increase the level of knowledge related to entertainment services,
as well as provide benefits from visual recognition techniques to increase the level of knowledge outcomes in
DL to achieve satisfactory services for Hajj and Umrah destinations in Makkah and the surrounding area.
1.5. Purpose and structure of the study
A comprehensive review was carried out of the current structures and frameworks of AR-based
archaeological sites that are facing some restrictions and the factors that need be improved through AR-
mobility[9]. As shown in Figure 2, head-mounted displays (HMD), which are the main part of AR, can give
rise to various challenges related to the obstruction of the actual object displayed by the HMD lenses due to
their relative weight when used for a long time. In addition, there can also be a number of limitations with
regard to the use of smart mobile applications that affect the user experience[6]. Additionally, the limitations
found in a number of smart devices or tablets in different studies had a negative impact on the user experience
and on their acceptance in dealing with the current experience[6]. The increased screen time required by AR-
enabled guides can be a distraction when it comes to participation and learning about the scholarly material
provided through AR[12]. Furthermore, the evaluation of the landmarks did not focus on the learning aspect,
but simply on the handling of artifacts or the usability of the guidelines of these landmarks. Since smartphones
are already being used by almost everyone, smart glasses can be directly connected to smart devices, thus
providing the freedom to create new solutions without any additional hardware cost. Therefore, the use and
development of modern technologies to aid the user experience are vital. There is a need to create and evaluate
AR based on advanced AI practices, the retrieval of information on archaeological sites, and a rich viewing
mechanism to aid the experience of visitors to historical landmarks.
Therefore, the research question for this study focused on the techniques and algorithms that could be
implemented to display information related to historical landmarks in general through smart glasses, depending
on the AR and amount of information, and number of audio, and video files displayed on a screen that has
been integrated with smart glasses in order to feed the user’s cognition of the two holy cities in the KSA;
namely, Makkah and Madinah.
6
Figure 2. Head mounted displays (HMDs) with augmented reality (AR).
The research focused on the importance of linking smart glasses with AR, which can be worn anywhere,
as the AR content would be created inside the lens of the glasses in front of the user. Thus, when users wear
the smart glasses, they will be able to see their physical surroundings in the same way as in the case of
conventional glasses. In addition to the AR-related smart glasses, Google Glass has been overlaid with
additional content as to what users will see[12]. Furthermore, Samsung offers smart lenses with a built-in camera
and sensors that are controlled by blinking. This technological cluster superimposes additional content for the
wearer. Thus, the informational content and material that are translated into the user’s language through the
smart glasses will help to increase the level of acceptance and ease of use by providing the material presented
through AR[13].
This study has been divided into a number of parts, namely, the introduction, which contains various
information about the scope of the study, the status and the technical aspects, focusing on the outputs of
previous studies in the same field, the methodology used for this study, the proposed technical framework, two
stages of Implementation and testing, discussion, evaluation, and results. These aspects are presented in detail
in the following sections.
2. Literature review
Previous studies focused on a number of basic aspects which are integral to the use of AR. A number of
studies focused on working and dealing with the historical geographic information system (HGIS), which is
considered to be the technological bridge between geography and history in a digital framework. These studies
covered the history of Makkah in the KSA for several thousand years, with many sites representing important
historical landmarks[5–7]. A study by Mirza et al. focused on the maps but faced certain limitations due to the
large geographical coverage required to provide appropriate images for the subject of the study and to display
information based on the place, time, and historical data presented in the HGIS. Therefore, a number of studies
used the HGIS to set coordinates that could provide historical information associated with landmarks, and then,
added multimedia files that helped display the historical information directly related to the coordinates[3]. Baik
et al.[6] indicated the negative aspects of the HGIS and its direct dependence on the coordinates associated with
the places, which are expected to overlap if there are many historical facts within a small area, similar to the
case of Makkah or Medina.
A number of studies focused on the use of multimedia to directly document, archive, and review Islamic
architectural heritage sites, which are tourist attractions in the KSA, for many parties and places. Alsabban and
Fatani[7] focused on the use of multimedia videos, photographs, and VR technology to document the
architectural heritage of the Grand Mosque area over the past decade, which represented a quantum leap in
documenting construction works with the latest modern technologies. Thus, the utilisation of modern
technologies to display places, provide access to information and present it to tourists, is a basic requirement
for documenting the historical depth of the Makkah region and the surrounding areas[4]. Alsabban and Fatani[7]
also referred to some proposed models in the process of designing and creating content using AR technology,
which are being used globally, such as in the Rennes Museum in France, where information is presented from
7
five different aspects and focuses on the description, technology, icon, artist, and theme, which are compatible
with all the selected artifacts. The user can access different multimedia content for each segment. Since the
prototype was implemented with different media such as texts, audios, videos, and slideshows with two-
dimensional (2D) and 3D overlays, the problem that users faced was related to information redundancy as the
information that was presented in the videos or audios was actually presented in the texts as well[7].
Besides, many studies indicated the importance of using AR as one of the solutions in the process of
presenting the informational side of facts and events related to specific places[5–7,9]. Khan et al.[9] indicated the
importance of looking at works of art or landmarks through AR to encourage the visitor to learn more about
the context of artistic production. The study focused on the keenness of users to take advantage of the hidden
details of artworks, where the information shows the AR approach to museums and thus, enables visitors to
discover artifacts in fun and memorable ways[9,14]. Furthermore, the use of AR coincides with the requirements
of VR and AR technologies in many fields such as education, museums, and various artistic fields, which help
to increase the attraction of the listener and viewer to the information used in the relevant field[15]. The virtual
museum experience and its improved online images also constitute attractive factors for young Internet users
to access museum resources and use them interactively through many AR methods[13]. These methods will help
to indicate the level of audience acceptance and emotional resonance through the museum experience scale
(MES), which examines the characteristics of the online community through opinion polls about the acceptance
level of the material presented and the way it is produced and presented to the user[9].
AR technologies focus on a number of methods, namely SLAM, marker, and position. The SLAM-based
method focuses on simultaneous translation and mapping technology. It relies on a set of complex algorithms
and data received from sensors. Thus, SLAM technology in AR requires many predefined map-based
applications to place virtual 3D objects, and SLAM allows computers to understand what is going on around
them using detection based on specific points on the drawn map. This approach is considered to be the main
driver of innovations such as autonomous vehicles, robots, and drones, as shown in Figure 3.
Figure 3. SLAM-based approach in AR.
The Marker-based method in AR depends on a widespread approach to the application of AR technologies.
It can recognise tags in images through the camera of a mobile device to detect a predefined tag, which then
triggers certain computer-generated content. Tags can also be AR codes, physical objects, or printed images
as they appear in the Absolut Vodka augmented reality app. To enable an AR application to recognise a real-
world object as a trigger, a specific tag must be embedded in that object. Once a tag appears within the view
of the camera, the software calculates the position of both the tag and the content displayed for it, and by
changing the position of the object in the real world, the position of the content that is created through the
computer can be affected (Figure 4).
8
Figure 4. Maker-based approach in AR.
The location-based method is used to locate and place additional digital information, based on data
derived from GPS. It is also known as AR based on locations and positions. In order to provide the application
to display the computer-generated information correctly, the devices that view the information must have a
data-based trigger. For example, the best solution in the location-based method is that the geolocation that
links the information in AR to where it is displayed to its true location in the real world is shown within the
field of view of the camera. Thus, AR applications can tell users more about a specific location after pointing
the camera at the restaurant’s sign. Users can check and search for basic information that can be provided on
Google such as opening hours, top photos, and reviews of previous visitors (see Figure 5).
Figure 5. Location-based approach in AR.
To keep up with the significant progress in the current industrial revolution, it highly important that
historical monuments be digitalised and ML-based methods be used to better identify and digitally reconstruct
archaeological sites while enhancing the visitor experience. The responsible authorities in the KSA tend to
consider the importance of preparing Islamic historical sites using the latest technologies such as mixed reality,
including AR and VR, to provide a better user experience for visitors[14]. Since technologies such as AR can
provide museum visitors with an interactive and free environment when it comes to archaeological sites[14],
several projects and prototypes have been developed to achieve an interactive environment that matches the
requirements of the current research field and increases the level of acceptance among visitors to historical and
archaeological sites in the KSA[16].
Through reviewing previous studies as shown in Table 1, the research gap appears by finding and
activating uses of modern technologies in real museums by creating a VR and AR environment for which they
are designed to appear appropriately[13,15]. However, the use of these technologies in the natural environment
faces many difficulties, such as natural lighting of the sun or night, as most of the historical monuments in
Mecca and Medina are monuments in open places, except for the monuments in the two holy mosques. In
addition, this study will use a diversified education methodology to compensate for the loss of information
through the natural environment that is needed by taking advantage of modern technologies in drawing an
appropriate approach to the natural and technical requirements of this study, which were not covered in detail
during previous studies.
9
Table 1. Related work summary.
No.
Reference
Year
Main finding
Main category
1
[5,9]
2020, 2021
Focused on the maps but faced certain limitations due to the large
geographical coverage required to provide appropriate images
Using augmented reality
2
[6]
2015
Negative aspects of the HGIS and its direct dependence on the
coordinates associated with the places
Using HCIS
3
[7]
2018
Using multimedia videos, photographs, and VR technology to
document the architectural heritage of the Grand Mosque area
Using multimedia in VR
4
[9,14]
2021, 2019
Focused on the keenness of users to take advantage of the hidden
details of artworks, where the information shows the AR approach to
museums
User experience through
AR
5
[13,15]
2020, 2020
It is benefit to increase the attraction of the listener and viewer to the
information used in the relevant field
The requirements of VR
& AR technologies
3. Research methodology
In this research, the proposed methodology was used to formulate an experimental setup to evaluate the
collection of samples for the landmarks in this study. Additionally, the results of the evaluation were taken
into account with a focus on the extent of the benefits that could be provided by the participants in the
evaluation of AR methodologies by users through a number of statistical tests, such as a t-test, to compare the
results of the participants in the evaluation.
After identifying the landmarks as samples, a critical stage in this study was the evaluation form, where
the effectiveness of the methodology used was evaluated through the landmark achievement analysis tool.
After the reviewing the literature in the field of landmark evaluation, the place experience scale (PES) was
chosen to examine the experience of visitors to the selected sites. The PES consists of four basic components,
namely participation, knowledge and learning, emotional communication, and purposeful experience[17].
Furthermore, the usability and usefulness of the proposed framework were analysed as additional elements to
know the level of acceptability, and ease of use of the tools and materials presented in this study[18].
The evaluation factors were then used to determine the sample landmarks after obtaining their test results.
Since the target audience needed to have prior knowledge of smart applications and AR, therefore, a small
questionnaire was distributed to a random sample before they were selected for participation in the evaluation
of the parameters used in this study. Then, the samples were handed the application linked to the smart glasses,
and the participants were able to enjoy a tour of a number of the landmarks that had been selected for this
study.
By showing the steps for the selection of the samples, the questionnaire was used to measure the level of
satisfaction with the information provided through the four previous steps of the questionnaire. The two
subscales of the usability and usefulness of the 3D model were also evaluated by this experiment.
After the enjoyable visit, all the participants were analysed in terms of their experience using a
questionnaire that was created as an evaluation tool. The questionnaire was divided into three sections. First,
it focused on identifying the characteristics of the participants. Secondly, it focused on the four components of
the historical landmarks experience scale using PES. Lastly, it focused on the usability and usefulness of the
three-way interaction model[18].
4. Proposed framework
4.1. Technical environment
The following environments were used to develop the framework that was generally applied for this study.
10
Google AR-core: This is a software development kit (SDK) developed by Google for the AR environment.
It provides cross-platform APIs and applications to professionally create new experiences and works in the
smartphone environment such as android and iOS and the web. Through the AR-core, there can be learning to
create a new world of contextual understanding of people, places, and things, where basic tools are provided
to help build experiences of AR. Also, it has tracking movement and environmental understanding, with
aspects that focus on the size and location of all types of surfaces, and on deep understanding, which measures
the distance between surfaces from a certain point. Finally, it provides a lot of information about the average
intensity and colour correction in the virtual environment from the reality of the natural environment[19].
Android platform: This works to develop the android environment application proposed for this study.
The android platform helps to provide a unified approach towards the development of apps for smart devices
based on working across different android devices[20].
Augmented image database: This is a database that contains a list of images that will be detected and
tracked by the AR-core. Additionally, this tool supports databases with up to 1000 images for one landmark.
The required database can be generated by the database generation tool provided in the SDK, or created
dynamically at runtime by adding individual images to the Augmented Image Database, and only one image
database can be active in a session. Any of the images in the currently active image database can be used to
provide the tracking status or pause the tracking if new file storage tools related to the requested feature are
utilised in the application of the database[21].
MySQL database: This is used to save information through a relational database management system that
stores data in separate tables and organises the database structure into physical files optimised for speed[22].
4.2. Architecture
The flowchart diagram, shown in Figure 6, for this study consists of 13 components and they are
presented as the following parts:
⚫ AR-app: The proposed framework is for application in smartphones and can be used in tablets. It helps to
increase the level of interaction between smartphones and the associated required accessories to operate
the proposed framework for this project.
⚫ Identify location: At this stage, as seen in Figure 5, the current location is determined to identify the
landmarks that are within and stored in the database. This feature speeds up the process of identifying
landmarks by reducing the number of possibilities for tourist attractions by making a comparison between
the image captured from the smart glasses with the images stored in the databases of the application. It is
used to identify the required landmark and then, to fully specify the type of information that can be
provided to the user.
⚫ Identify the users: At this stage, the users are required to register in order to determine the basic
information about them, which will help in fetching the required information professionally. For example,
the required information will be age, level of knowledge, and preferred language. Different data will
appear about the landmark if the viewer is a child or a man, who is visiting the landmark for the first time
or who has visited the landmark many times, the proposed language, or the level of knowledge of the user.
All this information will help to determine the level and depth of information that will be presented to the
user later. All these parameters will contribute to bringing customised and appropriate content to the
visitor, along with an easy analysis of his/her information about the teacher.
⚫ Identify the users: At this stage, the users are required to register in order to determine the basic
information about them, which will help in fetching the required information professionally. For example,
the required information will be age, level of knowledge, and preferred language. Different data will
appear about the landmark if the viewer is a child or a man, who is visiting the landmark for the first time
or who has visited the landmark many times, the proposed language, or the level of knowledge of the user.
11
All this information will help to determine the level and depth of information that will be presented to the
user later. All these parameters will contribute to bringing customised and appropriate content to the
visitor, along with an easy analysis of his/her information about the teacher.
Figure 6. Proposed framework.
⚫ Complete the registration: The application forces the user to complete the registration to get the full
benefits of the application. These steps will help to determine the user’s knowledge about the features of
statistical modelling to make a full analysis for the users and their particular needs. After the registration
stage has been completed, then the user’s account will be verified and authenticated through two basic
authentication methods such as by email or mobile number to fully authorise the user account.
⚫ Landmark training module: This part of the application focuses on storing the landmarks in databases,
including the storing of 2D and 3D images for each landmark, with classified information, illustrative
images, links to each landmark in the system, and various images of landmarks from different sides.
Furthermore, the database contains data that has been generated in advance by the system manager
through the Tensor Flow Lite (TFLite) training module, which recognises and processes images using
DL. In this case, a class is built for each feature and then, the captured data is converted to grayscale to
be normalised to form the inputs of the neural network. Both the backpropagation algorithms, which
appear in the above formula, are used to give improved results. Moreover, it is possible, through the
application, to connect to smart glasses while using AR. The images captured through the camera are
analysed and a classified input model is built that is used to fetch the data related to that landmark and
then, translate it according to the user’s language. After that, the information about the neural network is
updated to improve the user’s future experiences.
⚫ Determine user knowledge about landmark: At this stage, the user’s level of knowledge should be known
through a simple questionnaire that appears to the user of the application during the registration stage.
This level of knowledge is determined in terms of the user’s scientific knowledge, and the nature and
depth of the information that he/she wants to know and view. Thus, the customer’s experience, through
the identification of the level of user experience, will be increased according to the type of information
12
that will be provided to him/her, and thus, the desire to benefit from the application will be greater.
⚫ Connect with smart glass: At this stage, the information extracted from the image on the smart glasses
will be linked with the images in the application database, and thus, the attached information will be
recalled based on the location captured from the smart glasses and depending on the level of knowledge
required by determining the level of knowledge of the user.
⚫ Provide the user interface in the app: This step is available for the user and system administrator, as it
includes some services and functions that are common to both sides. Through this stage, the tools and
capabilities of the system are managed, and the important stages for the customers and their experience
are developed frequently to achieve the appropriate model for the user, according to the run PES survey
step.
⚫ Run time for the AR through CNN: Through this sensitive and important stage, the application is learned
through the many additional cognitive aspects of CNN, which are fed through two basic methods; firstly,
through the data that is initially stored in the application database, and secondly, through the data that is
added frequently from the Internet and receives good attention and great satisfaction when presented to
the user through the questionnaire used in the PES step.
⚫ Translate that information into the user’s language: This stage provides access to the translation of the
submitted scientific material into the user’s preferred language, which is chosen when starting the
application. During this stage, the information is translated using electronically accredited translation
tools such as Google Translate.
⚫ Run PES survey: This questionnaire is implemented according to the research methodology mentioned
earlier.
⚫ Update CNN and user profile: This phase of the application is linked to the application database, through
which an update is made on the data stored in the database, which helps in the deep ML of the application.
This part of the application will help to increase the level of knowledge over time in the database of places
and what information is considered more enriching and acceptable to users of the application over time.
⚫ Artifact information retrieval model: This stage is related to the ‘run time for the AR through CNN’ stage,
through which archaeological information and tourist attractions are retrieved and work is done to provide
the educational and knowledge environment needed by the application in all the functions and tools
associated with it.
When the user starts using the application and connects the smart glasses, the user’s authentication is
verified and, in the case of pre-registration, he/she will be asked to register. Then, the user is directed to the
application interface. The architecture of the AR museum application consists of two main units, namely:
⚫ Artifact model training unit: This unit includes image recognition and processing methods. It contains the
proposed landmarks and deals with various processes by which the landmarks are recognised or a message
is shown when the landmark being viewed is not known.
⚫ Artifact detection module unit: This unit deals with the augmentation of multimedia content; such as texts,
images, or videos, in the real world. The user is also instructed to use eye gestures or other methods to
activate information that was not previously fully rendered by CNN, or when the user clicks a button to
enhance information about a specific landmark.
When the user views a particular landmark and requests additional information, the multimedia content
is placed along with the displayed material without interrupting the user or removing the displayed material.
The user can browse through photos, videos, and historical information about a landmark by focusing the smart
glasses on the landmark for several seconds.
5. Implementation and testing
Currently, this study focused on testing the proposed framework. This was an important stage as it would
13
be used by customers directly and would help to increase the capability level of the application in the smart
mobile environment to identify the basic landmarks that were directly related to Hajj and Umrah tourists in
Makkah and Madinah and their surrounding places. It began by feeding the application with the various
historical aspects of the archaeological landmark by defining the general shape of the landmark with a large
number of images that were stored in the supporting database of the application to directly identify the
landmark from any angle or various distances. This was a fundamental stage for the preparation of a list of
required information associated with the main scope of this study for the future. As the identification of
parameters was done through smart glasses, linking the glasses directly with the application was the next basic
requirement for accessing the application in an appropriate manner. Thus, the testing phases could be reduced
to four different types.
• First type: This was represented by defining the basic parameters, and therefore, the application was
expected to have previous information about the identified landmark or more information that needed to
be known if it was a new landmark. In this step, the landmark and its relevant basic information would
be sent to the technical team to work on providing primary information to make it available for all users.
• Second type: This type focused on the presence of preliminary information about religious monuments
that the customer could benefit from. This information was tested by the customer experience through the
PES questionnaire to ensure its suitability for the customer and to develop it in the future to greatly
increase the level of customer satisfaction.
• Third type: This type focused on the testing stages carried out through automatic feeding through DL,
which adopted the search and explore approach for primary information of religious landmarks through
the various available technical media channels, classifying and filtering them, and adding the information
to the database according to a number of criteria. This stage represented one of the most important stages
of DL and testing. It focused on teaching the machine the appropriate way and DL of the religious
landmarks that had been selected and included in the application.
• Fourth type: This involved testing the customer’s experience of the application in general and knowing
their level of satisfaction with the complete application. This part of the test was very important because
it could help to increase the opportunities for the development of the application in the future, and to
know the customer’s future requirements, which would help win customer loyalty for the application and
their experience in the future.
In addition to the above, various statistical tests were conducted, which might help to increase the chances
of designing the knowledge objectively:
• It is believed that designing the study by previous framework requirements might help to provide
information to answer the research questions objectively. The design of the research methodology for the
post-test was considered, which represented the test participants’ acceptance of the presentation method
and its associated knowledge only[23]. A number of statistical tests were also implemented, such as a t-
test for the independent data, and all the questionnaire variables were tested on a five-point Likert scale[23].
• A system was required that worked with smart glasses and was compatible with the android system that
was to be developed using the Google AR-core library to make the experience of AR possible and for
CNN to help identify archaeological sites. The proposed architecture of the android application is
described in Figure 6.
6. Discussion and evaluation
This study highlighted the importance of providing information with temporal and spatial dimensions for
landmarks that possess a diverse cultural heritage to increase the various cognitive opportunities for tourists to
access in-depth historical information so as to develop a diverse historical narration in various fields that would
be useful for refining the customer’s experience, and expanding it over time in the future.
14
The role of the Merciful Guests Programme is to provide the opportunity for the largest possible number
of Muslims to fully perform the Hajj and Umrah rituals and to enrich and deepen their experience by preparing
the two Holy Mosques and cultural landmark sites, and providing the best services before, during, and after
their visit to Makkah al-Madinah and the holy landmarks there. It represents one of the programmes of the
KSA in Vision 2030, which is being worked on as part of vigorous steps to provide a spiritual transformational
experience for the guests of Rahman through a number of programmes, including the preparation of Islamic
historical sites in such a way as to ensure that pilgrims and Umrah performers are enriched in their knowledge
and cultural experience, and to achieve integration between all the relevant parties to improve the system of
services provided through 16 tracks to enable 30 million pilgrims to experience a transformational faith. It
might help to reflect the honourable and civilised image of the KSA in the service of the two Holy Mosques
and the Merciful Guests Programme[12].
Therefore, this project seeks to give a unique and distinct opportunity by providing a lot of information
and enriching knowledge to present the Islamic historical landmarks in a unique and technical manner through
AR using the latest technologies in smart glasses and linking them with AR technologies to display the
information with audios using the method of simultaneous translation by presenting the material directly in the
users’ language.
The objectives of this research were mainly represented by a number of aspects to help increase the level
of knowledge of visitors and tourists in general to the KSA, especially in presenting the tourist attractions in
the city of Makkah, which is one of the main landmarks of tourism in the KSA. Therefore, the knowledge and
technology will help to increase the development of the non-profit economy in the following aspects.
• The presence of a large cultural and historical heritage, which is unique to the holy areas in the KSA, such
as Makkah and Madinah. It is possible to take advantage of this heritage and display it using various AR
and VR methods.
• Highlighting the Islamic landmarks and archaeological sites that have historical value and presenting their
information to visitors to the holy places.
• Facilitating the introduction of different nationalities and languages to the landmarks by presenting the
information in an AR manner, which depends on delivering the material presented in the various basic
languages. Thus, most tourists will benefit from it without needing translators to translate the material
presented into their mother tongue.
• It is possible to benefit from increasing the level of income by focusing on one of the strongest sources
of income for the KSA, which is represented by the services that are provided in relation to the Hajj and
Umrah. The Vision 2030 of the KSA is focused on increasing that income by facilitating many different
programmes and services through the Ministry of Hajj and Umrah to increase the number of pilgrims and
Umrah performers to 30 million by 2030. It represents one of the most important sources of transition to
a non-oil economy in the KSA[10].
• Benefits can be gained from presenting the historical landmarks in a modern manner through AR and VR,
and by taking advantage of multimedia capabilities in disseminating correct news about landmarks and
clarifying distorted historical facts in a modern and reliable manner.
• It is beneficial to use blockchain technology to record and secure landmarks and relevant information to
ensure higher reliability of the results. Thus, it can help to increase the level of user acceptance on the
technological side in dealing with applications, which will greatly enrich the knowledge material in the
tourism field in the KSA.
7. Conclusion
This study focused on linking the alignment of tourism with modern technologies that depend directly on
15
AI, ML, and DL to help increase the level of knowledge and enrich it completely at the level of individuals
and governmental, semi-governmental, and private institutions. It works to guide tourists to present and visit
various landmarks through the knowledge material spread on the Internet, as well as through the digital and
informational content available to the parties interested in tourism, through smart glasses and various technical
tools in the tourism field. Machine learning (ML) and DL algorithms are used to focus on increasing the
opportunities for promoting tourist attractions scattered throughout the KSA through the presence of formal
and informal media in the various means of communication and the digital world in general, to increase the
chances of accessing a large amount of enriching information. Opportunities to increase Saudi tourist
attractions will directly raise the level of tourism, which is one of the goals of the KSA’s Vision 2030.
Therefore, these tourism goals will help to increase the non-oil economy and the knowledge economy, which
will have a direct impact in the future. The development of the DL model and the application of various
advanced algorithms might help to achieve faster and more accurate results to increase the level of user
satisfaction and acceptance of modern technologies in the enrichment of knowledge from a tourism perspective.
Author contributions
Conceptualization, AB and KA; methodology, AA; software, AA; validation, AB, KA and AA; formal
analysis, KA; investigation, AB; resources, AB; data curation, KA; writing—original draft preparation, AA;
writing—review and editing, AA; visualization, AA; supervision, AB; project administration, AB; funding
acquisition, AB. All authors have read and agreed to the published version of the manuscript.
Acknowledgments
This research work was funded by Makkah Digital Gate Initiative under grant no. (MDP-IRI-19-2022).
Therefore, authors gratefully acknowledge technical and financial support from Emirate of Makkah Province
and King Abdulaziz University, Jeddah, Saudi Arabia.
Conflict of interest
The authors declare no conflict of interest
References
1. King R. The pilgrimage to Mecca: some geographical and historical aspects. Erdkunde. 1972; 26(1). doi:
10.3112/erdkunde.1972.01.06
2. Makkah State. Services directory for the Makkah region. Available online:
https://www.stats.gov.sa/sites/default/files/ar-g-serv-2015-makkah.pdf (accessed on 25 August 2023).
3. Mirza MN, Dawod GM, Elzahrany RA, Mirza MM. Building a Historical GIS for the City of Makkah, Saudi
Arabia. In: Regional Conference on Surveying & Development. 2015.
4. GAS. Hajj statistics. Available online: https://www.stats.gov.sa/system/tdf/
file_manger/Hajj%20Report%201444H-AR_compressed.pdf?file=1&type=node&id=145223&force=0 (accessed
on 25 August 2023).
5. Alandijany TA, Faizo AA, Azhar EI. Coronavirus disease of 2019 (COVID-19) in the Gulf Cooperation Council
(GCC) countries: Current status and management practices. Journal of Infection and Public Health. 2020; 13(6):
839-842. doi: 10.1016/j.jiph.2020.05.020
6. Baik A, Yaagoubi R, Boehm J. Integration of Jeddah Historical BIM and 3D GIS for Documentation and
Restoration of Historical Monument. The International Archives of the Photogrammetry, Remote Sensing and
Spatial Information Sciences. 2015; XL-5/W7: 29-34. doi: 10.5194/isprsarchives-xl-5-w7-29-2015
7. Alsabban RF, Fatani MA. Using multimedia in documenting and archiving the islamic architectural heritage of the
holy mosque area in Makkah city, Saudi Arabia. Islamic Heritage Architecture and Art II. Published online April
17, 2018. doi: 10.2495/iha180061
8. Alfelali MA, Garcia-Fuentes JM. Growth of the holy mosque in mecca and the first legislation of heritage
conservation in Saudi Arabia (1955–2010). Islamic Heritage Architecture and Art III. Published online July 28,
2020. doi: 10.2495/iha200061
9. Khan MA, Israr S, S Almogren A, et al. Using augmented reality and deep learning to enhance Taxila Museum
16
experience. Journal of Real-Time Image Processing. 2020; 18(2): 321-332. doi: 10.1007/s11554-020-01038-y
10. Arab News. New e-services launched to aid Hajj and Umrah Available online:
https://www.arabnews.com/node/2324381/saudi-arabia (accessed on 25 August 2023).
11. KSA Embassy in Washington DC. Vision 2030 is built around three primary themes: a vibrant society, a thriving
economy and an ambitious nation. Available online: https://www.saudiembassy.net/vision-2030 (accessed on 25
August 2023).
12. 2030 Vision. Doyof Al Rahman Program (2021). Available online:
https://www.vision2030.gov.sa/v2030/vrps/darp/ (accessed on 4 December 2021).
13. Magomadov VS. Examining the potential of VR and AR technologies for education. Journal of Physics:
Conference Series. 2020; 1691(1): 012160. doi: 10.1088/1742-6596/1691/1/012160
14. Guazzaroni G, Pillai AS. Virtual and Augmented Reality in Education, Art, and Museums. IGI Global; 2020. doi:
10.4018/978-1-7998-1796-3
15. Chen TL, Lai WC, Yu TK. Participating in Online Museum Communities: An Empirical Study of Taiwan’s
Undergraduate Students. Frontiers in Psychology. 2021; 11. doi: 10.3389/fpsyg.2020.565075
16. Ghouaiel N, Garbaya S, Cieutat JM, et al. Mobile Augmented Reality in Museums : Towards Enhancing Visitor’s
Learning Experience. International Journal of Virtual Reality. 2017; 17(1): 21-31. doi:
10.20870/ijvr.2017.17.1.2885
17. Othman MK, Petrie H, Power C. Engaging visitors in museums with technology: scales for the measurement of
visitor and multimedia guide experience. In IFIP Conference on Human-Computer Interaction. Springer, Berlin,
Heidelberg. pp. 92-99.
18. Othman MK, Ee Young N, Aman S. Viewing Islamic Art Museum Exhibits on the SmartPhone: Re-examining
Visitors’ Experiences. Journal of Cognitive Sciences and Human Development. 2015; 1(1): 102-118. doi:
10.33736/jcshd.192.2015
19. AR-core. AR-core over view, Available online: https://developers.google.com/ar (accessed on 25 August 2023).
20. Android platform. SDK Platform release notes Available online:
https://developer.android.com/tools/releases/platforms (accessed on 25 August 2023).
21. AugmentedImageDatabase. AugmentedImageDatabase. Available online:
https://developers.google.com/ar/reference/java/com/google/ar/core/AugmentedImageDatabase (accessed on 25
August 2023).
22. MySQL database. MySQL HeatWave—One MySQL Database service for OLTP, OLAP, ML, and Lakehouse,
Available online: https://www.mysql.com/ (accessed on 25 August 2023).
23. Korur F, Yerdelen-Damar S, Sağlam H. The Development of an Integrated Scale of Technology Use in Physics.
Research in Learning Technology. 2021; 29. doi: 10.25304/rlt.v29.243
