Content uploaded by Ching-Yu Huang
Author content
All content in this area was uploaded by Ching-Yu Huang on Aug 07, 2017
Content may be subject to copyright.
Abstract—In recent years, many of the latest smartphones or
mobile devices currently in the market come with built-in
cameras and GPS systems. These devices can geotag and store
critical geo-information along with the photos. This crucial
metadata information will be used in the development for an
algorithm that will identify the relationship between geotagged
images and facilities (buildings). The development of this
technology generates massive amounts of geo-data which could
be very helpful to public and authorities on large campuses or
cities. This research proposes methods to mine and construct
the relationship between polygon points of buildings and the
geo-information on huge sets of images, and then determine
whether these buildings could be shown on the pictures or not.
In order to test the methods, many geotagged images will be
taken throughout the Kean University campus.
Index Terms—Geotagging, latitude, longitude, orientation,
augmented reality.
I. INTRODUCTION
In the last few years, photos taken with smartphones have
outnumbered photos taken with traditional cameras as more
and more people use the built-in cameras in their phones.
These devices come with special sensors that allow for
additional data such as geographic information via GPS to
be added to the metadata of these photos. Once a picture is
taken, the camera registers the location the photo was taken
in the form of latitude, longitude, altitude, direction as well
as device names, taken date and time, resolution, ISO,
exposure, shutter speed and other EXIF information.
Recently, there has been a dramatic increase in the number
of people using the mobile devices to take pictures and share
with other people. The geo-location information for tagging
pictures can be kept or removed when the images are
transformed such as resizing, or uploaded to social media
websites such as Flickr or Facebook.
The location that is automatically attached to a photo by a
GPS enabled device reflects the position from where the
photo was taken. Modern cameras or smartphones are
equipped with an internal compass, which enriches the
information recorded by including the orientation towards
true north or magnetic north. A geo-semantic modeling was
proposed to provide a powerful capability to learn visually
from historical geo-tagged photos [1]. Geotagging photos
can also be used in Computer Vision and multimedia, and
virtual reality [2], [3].
Manuscript received September 5, 2015; revised March 15, 2016.
Ching-Yu Huang is with the Department of Computer Science, Kean
University, USA (e-mail: hcy100@gmail.com).
Henry Choque is with Costume Supercenter in Edison, New Jersey,
USA.
Brany Suero Benitez is with New Jersey Institute of Technology, USA.
A framework was proposed to estimate the camera
viewing direction of a single geotagged photo in any
environment [4]. For searching and tracking suspects,
authorities can ask people to upload their geotagged photos.
In order to let people know the city or campus better, people
upload and share their pictures online. However, both cases
will require methods that can automatically process a huge
amount of geotagged images and identify the relationship
between images and facilities. In this paper, we make use of
the geo-information in order to develop an algorithm for
identifying the relationship between geotagged images and
facilities (buildings) located inside the Kean University
campus. Overall the accuracy of these coordinates is
approximately 5 to 10 meters [5] and meets our data
requirements.
II. RULE BASED METHODS
For this research, we are using hundreds of geotagged
images which have been uploaded to a database constructed
using MySQL for easy accessing. The database runs under
the Apache web server. For purposes of this research, we are
also assuming that all photos have a 45-degree angle view
and are within the 5 to 10-meter accuracy previously
mentioned. For the campus buildings, GPS coordinates were
obtained through Google Map [6]. PHP is the computer
language used for this project because there free
mathematical functions to retrieve geotagged information
from photos and calculate the geometric data [7].
We proposed four specific algorithms and rules to
identify the relationship between geotagged images and
facilities. The first rule excludes any photos that were too far
away from the target location. The second rule excludes any
photos that were not facing the target location. The third rule
excludes any photos that were blocked by another building.
The fourth and final rule determine whether or not a photo
had an overlap with another photo. Fig. 1 shows some
photos, a partial campus map, and the 4 rules referring to a
particular target location.
A. Rule 1 — Distance
If a photo is too far from a target location, it might not
contain the searching objects or the resolution might not be
good enough to identify the objects. Since the uploaded
photos could be from randomly located places, it is
necessary to filter out unnecessary photos to narrow the
amount of processing photos for next steps. To exclude a
given photo that is N ft (500ft in our example) away from
the Target Location (Willis Hall in our example), we
implemented what is known as the „haversine‟ formula [8].
From this formula, we define distance as computing the
Mining the Relationship between Geotagged Images and
Facilities
Ching-Yu Huang, Henry Choque, and Brany Suero Benitez
International
Journal of Information and Electronics Engineering, Vol. 6, No. 3, May 2016
202
doi: 10.18178/ijiee.2016.6.3.624
lengths of great circle arcs connecting pairs of points on the
surface of a sphere (earth). We compute the azimuths of the
second point in each pair with respect to the first (the angle
where the arch crosses the meridian containing the first
point). Since the input latitudes and longitudes can be
scalars or arrays of equal size, they must be expressed in
degrees.
Fig. 1. The test photos on Kean campus.
Fig. 2. Cartesian plane representation for rule 2.
B. Rule 2 — Direction
The photos will not contain the objects if they are not
facing the target location, even if they are very close to the
target location. So, the second rule was adopted to exclude
any photos passing Rule 1 but not facing target location
(Willis Hall in our example). First, a geotagged photo has a
direction b. The bearing b is defined to be the angle between
two given points on earth‟s surface. Both a and b are
assumed that degree 0 means facing north. If |a - b| < c, the
photo is considered facing the building where c is a half of
camera angle (c = 25 in our example). Two photos shown on
Fig. 1 are examples. From the bearing between photos and
buildings and direction of the photo, we can see the
IMG3837 direction is not facing Willis Hall, and IMG3806
is facing Willis Hall (See Fig. 2).
C. Rule 3 – Blocking
A photo might not contain the searching objects because
other buildings could be blocking the objects even if the
photo meets Rule 1 and 2. The third rule needs to exclude
any pictures that are blocked by other buildings. To
accomplish this, the polygon points (GPS coordinates) of the
building are required and they can manually be retrieved
from Google maps. These points are passed to a polygon
array to form the building. A new algorithm is applied to
determine if the segment (PnPm) of any two polygon points
Pn and Pm intersect with the line (PiPb) between the photo
Pi and building center Pb. As shown in Fig. 3, the line from
photo IMG3820 to Willis Hall has two intersections
(marked in a black dot). Therefore, photo IMG3820 won‟t
contain Willis Hall.
Fig. 3. Cartesian plane representation for rule 3.
D. Rule 4 — Overlapping
When a photo meets Rule 1, 2 and 3, it should contain the
target object. However, the photo only shows an angle of the
object. In many cases, it is very important to have different
views of the objects from different times or angles of other
photos. This will require a new algorithm to identify if any
photos could possibly overlap and have the same objects.
The fourth algorithm is adopted to achieve this goal by
checking if the angles Ai1 and Ai1 intersect each other
where Ai1, Ai2 are the angles of photo i1 and i2 respectively
with the assumption i1 and i2 meet rules 1, 2 and 3. The left
image on Fig. 4 shows that photo IMG5485 is not facing the
same direction as photo IMG3806. So, they are not
overlapped and won‟t possibly have same objects. The
photos IMG3806 and IMG5517 on the right image of Fig. 4
have same objects because the photos are overlapped.
International Journal of Information and Electronics Engineering, Vol. 6, No. 3, May 2016
203
Fig. 4. Rule 4 no angle intersection; Rule 4 match angle intersection found.
III. ANALYSIS OF EXPERIMENTAL RESULTS
In our approach, we used the above rules to obtain our
results. The algorithms were designed around these rules
and were tested to prove their accuracy. For our first goal,
we can observe that any pictures outside of the 500ft range
have been excluded since these pictures are not very
accurate. For our second rule, image IMG 3837 does not
face our primary target which is Willis Hall and is
consequently excluded as well. Image IMG 3820 direction
does face Willis Hall but is being blocked by another
building, in this the case the CAS (Center for Academic
Success) building. For our final rule, we try to determine
whether two photos intersect by finding the point of
intersection between these two images. In our case, we can
see that IMG 5518 and IMG 3806 face the direction and
despite the fact that one photo is much farther away than the
other, an intersection does occur. Several hundreds of
photos were available on the database, but a set of 20 photos
was picked to test rules 1, 2 and 3. The results were
manually verified and shown on Table I.
For Rule 1, we entered two coordinate points and used the
earlier described method to calculate the distance between
these two points. The results were manually verified by
comparing the results with similar algorithms found online
such as Google maps distance feature. For Rule 2, similar
algorithms were not found to compare our results, but we
were able to visually verify our results by comparing our
given method results from the orientation retrieved from the
metadata stored in the picture. In the above picture, for
example, by observing the arrows we can determine whether
it faces the target building and compare this with our
algorithm.
For Rule 3, we discarded any pictures that although were
facing our main target, were also being blocked by another
building. For this method, the polygon points of the
blocking facility were retrieved from Google Maps in order
to create an outline image of the building. If the given angle
direction of the photo intersected any of these points, then
we could safely assume that a blocked had occurred. For
Rule 4, we had to determine whether an intersection
between two given photos had occurred. In order to do this,
we had to find the point of intersection between these two
photos. Once a point of intersection was found, it meant that
an intersection had occurred. There was a flaw in the
algorithm, however, basically due to the fact that certain
intersections could potentially occur behind or even within a
facility. In this case, the two photos are not considered
overlapped. Table II shows the result of the 190
comparisons among 20 photos.
TABLE I: RESULTS OF RULE 1, 2 AND 3
Results
Total # of photos
Total # of errors
Accuracy
Rule 1
20
0
100%
Rule 2
20
2
90%
Rule 3
18
3
83.3%
TABLE II: RESULTS OF RULE 4
Total # of
comparisons
Total # of
overlaps
Total # of
overlap errors
Accuracy pct
190
86
5/24
80%
IV. CONCLUSION
How to identify the relationship between photos and
buildings are very important in mining spatial data and
constructing an Augmented Reality (AR) Information
system. Authorities can use the relationship to track suspects
through a large volume of photos uploaded by the public.
People can use AR App on their mobile device to face a
facility or an open space to see the event's information,
including the multimedia supplements, without typing in
any URL. The 3-dimension geographic information and the
direction the user is facing can be constructed by the photo
relationship to identify the target facility to retrieve the
corresponding events and activities.
This research showed how geotagged information can be
used in a variety of ways, and they are very helpful for new
students, faculty, staff and visitors to become familiar with
the campus as well as for everyone to find out about events
such as open houses, seminars, exhibits, club activities, and
competitions.
All the obtained results were thoroughly checked in an
effort to obtain the highest possible accuracy. This research
is in its beginning stage and sets the foundation for future
students who want to continue researching this type of
technology. Our research will eventually help construct a
campus view for augmented reality or even help authorities
automatically narrow down/identify suspects by collecting
camera photos from the general public.
However, the geotagged information can be also used by
predators to track and invade people, especially the minors
or innocent students. Many people like to use their cellular
phone to take and share pictures on social media. In order to
protect people‟s privacy and safety, we strongly suggest that
all websites should give users options to remove the
geotagged information when they upload and share personal
photos. This option could be also implemented in browser
settings to automatically remove the geo-tagged information
when the browsers upload the photos to any websites so
users will not need to rely on if websites provide the options.
REFERENCES
[1] J. R. Smith, “What happens where?” in Proc. the 2nd ACM
International Workshop on Geotagging and Its Applications in
Multimedia, Barcelona, Catalunya, Spain, October 21-25, 2013.
[2] J. B. Luo, D. Joshi, J. Yu, and A. Gallagher, Geotagging in
Multimedia and Computer Vision – A Survey, 2011, pp. 1-19.
[3] C. Hang, A. Gallagher, and T. Chen, “GPS refinement and camera
orientation estimation from a single image and a 2D map,” in Proc.
2014 IEEE Conference on Computer Vision and Pattern Recognition
Workshops, 2014.
International Journal of Information and Electronics Engineering, Vol. 6, No. 3, May 2016
204
[4] M. Park, J. Luo, R. T. Collins, Y. Liu, “Beyond GPS: Determining the
camera viewing direction of a geotagged image,” in Proc. the
International Conference on Multimedia, Firenze, Italy, October 25-
29, 2010.
[5] B. Christine. On the (In-) Accuracy of GPS Measure of Smartphones:
A Study of Running Tracking Applications, Dec. 2013.
[6] Google Street View Image API, Google Developers, p. 21, May 2015.
[7] Trying to Calculate Distance between Locations Using PHP While
and for Loops, Aug. 2, 2013.
[8] C. Veness. Movable type scripts. Calculate Distance and Bearing
between Two Latitude/Longitude Points Using Haversine Formula in
JavaScript. (March 22, 2015). [Online]. Available:
http://www.movable-type.co.uk/scripts/latlong.html
Ching-Yu Huang is an assistant professor of the
Department of Computer Science at Kean University
since September 2014. Dr. Huang was born in Taiwan
and received a Ph.D. degree in computer and
information science from New Jersey Institute of
Technology, Newark, New Jersey, USA in January
1998.
Prior to joining Kean University, Dr. Huang had
more than 16 years of experience in the industry and
academics in software development and R&D in bioinformatics. His
researches are focus on SNP genotype calling and cluster detection, image
processing and pattern recognition, especially in microarray and fingerprint;
geotagged images and location information reconstruction, database
application development, data processing automation, e-learning,
educational multimedia, methodology, and online tools for secondary
schools and colleges. Dr. Huang has more than 20 publications in journals
and conferences and more than 20 presentations in workshops and invited
lectures.
Henry Choque was born in Lima, Peru. He received
his B.S. degree in computer science from Kean
University, Union, New Jersey in 2015. While in
college, he completed a summer internship at MTF
(Musculoskeletal Transplant Foundation) as an IT
administrator where he gained some valuable
experience in system troubleshooting, computer
networking and system imaging. After graduation, he
joined Costume Supercenter in Edison, New Jersey as
a Jr. IT System Administrator.
He is a member of the LSAMP program and continues to pursue his
interest in computer programming, with emphasis on PHP, Java and
database development.
Brany Suero graduated in May 2015 with a dual
major of a B.S degree in computer Science and B.A.
degree in mathematics from Kean University, Union,
NJ. Currently, he is pursuing his master degree in
computer science at New Jersey Institute of
Technology. Brany is passionate for big data,
software development (involving mathematical
procedures) and new technology. Brany Suero came
to the US in 2008 and joined Kean University in
2009.
International Journal of Information and Electronics Engineering, Vol. 6, No. 3, May 2016
205
