Content uploaded by Tatjana Budimirov
Author content
All content in this area was uploaded by Tatjana Budimirov on May 21, 2019
Content may be subject to copyright.
Importance of Image and Point Cloud Matching of Road Infrastructure
Feature Extraction
Dejan VASIĆ, Mehmed BATILOVIĆ, Tatjana KUZMIĆ, Marina DAVIDOVIĆ, Serbia
Key words: point cloud, matching, extraction, accuracy
SUMMARY
Nowadays, collection procedures are getting faster and simpler, so the requirements of data
processing are higher. This paper will show some of realized projects using the methodology
of mobile laser scanning (MLS). The paper will also cover best practices for working with the
large data sets and creation of final product. Reliable feature extraction from 3D point cloud
data is an important phase in many application domains, such as traffic managing, object
recognition, autonomous navigation, civil engineering and architectural projects, and so on. All
those projects need to rely on quality data for existing conditions in order to be successful, so
the used methodology for data processing has been analyzed in details. Point cloud, as a main
surveying result has to be matched in order to obtain high-quality data. This paper deals with
problems and methodology of matching of point cloud, as well as matching of images and
emphasize the importance of this step at the multiple scans of the same corridors. All the future
work, for example- classification of point cloud, extraction of selected features and creation of
the final product depends on the point cloud matching. The importance of the integration of
images and point cloud in the process of extraction of 3D structural elements of the space of
the road infrastructure, as well as in the process of visualization and presentation of the same,
was emphasized. After this part, the whole process of extraction of road infrastructure data is
done. As there are different solutions for extraction, in this paper is chosen Orbit application.
The usage of Orbit feature extraction and Orbit publisher is explained with all its advantages.
At the end, ideas about further work and conclusions are presented.
Importance of Image and Point Cloud Matching of Road Infrastructure
Feature Extraction
Dejan VASIĆ, Mehmed BATILOVIĆ, Tatjana KUZMIĆ, Marina DAVIDOVIĆ, Serbia
1. INTRODUCTION
LiDAR (Light Detection And Ranging) collects information on the surface of the Earth as
scattered and unorganized thick point cloud (Wang et al 2009). What makes LIDAR particularly
attractive is the high spatial and temporal data resolution, as well as the ability to observe the
atmosphere and cover the altitude from the ground to more than 100 km (Forestry Commission
2012).
The implementation of laser scanning technology combined with a high precision navigation system
enables 3D scanning of roads, buildings and trees on the move (MLS - Mobile Laser Scanning).
The system uses several laser scanners, where each performs about 10000 measurements per second
(Ninkov et al 2014). MLS mapping systems provide a three-dimensional point cloud. They are
particularly applicable in projects involving smaller areas and specific tasks. Surveying is done by
moving the vehicle on the ground, when the laser scanner collects data about the environment, and
the navigation system based on GPS and IMU tracks the vehicle trajectory. It has to be pointed out
significantly lower costs of the MLS system when compared to the air laser scanning. With the
appropriate software solutions, MLS can automate key processes such as: creating or extracting
surface models, road signs, urban trackers, curbstones, track geometry and increase the cost-
effectiveness of the mapping process (Kuzmić et al 2017).
The scanner is constantly on the move, which is recorded as a trajectory. Points in the obtained
cloud are linked to the trajectory through a time stamp, vector laser scanner-point and vector
trajectory-laser scanner. The offset errors and the orientation errors between the overlapping
parts of the point clouds can often occur. In order to improve the accuracy of the point cloud,
the scans taken from different scan lines are matched, where exist several algorithms for
matching of overlapping parts of scans (Bang et al 2009). This step is of great importance
because on it directly depends the future work over the point cloud, such as: creating surface
models of different levels of detail, digitizing traffic signs, curbs, track geometry, etc.
2. MATCHING POINT CLOUD AND PHOTOGRAPHS
The procedure for matching point cloud and photographs involves determining precise
connections between two or more data sets, i.e. point cloud and panoramic photographs
collected during surveying terrain at different moments of time, from different angles or from
different platforms or sensors.
2.1. Problem definition
When surveying terrain with a mobile laser system, as a result is obtained a point cloud or scans
that are recorded by platform moving along different scanning lines. In order to match point
cloud, it is necessary that multiple scans represent the same area, i.e. that there is an appropriate
overlap between the scans. In addition to the laser sensor itself, the mobile scanning system is
equipped with a high-resolution digital camera that collects panoramic photographs throughout
the entire recording period. In addition to matching the point cloud itself, it is also possible to
match the collected images into a point cloud.
The matching of point cloud and photographs is challenging because, first and foremost, in the
very point cloud, the noise and variation of the coordinate system occurs. In order for the
process of matching the point cloud and photographs to take place, it is necessary to find the
appropriate link between the point cloud and photographs, and in this way, each recorded
photograph is linked to the corresponding location in the point cloud.
2.2. Related work
The paper (Miled et al 2016) defines a method based on finding the calibration parameters that
best fit the recorded point cloud onto photographs. The quality of this method was measured
using the Mutual Information (MI) measure between the brightness of the photograph and the
point's reflection. The basic motivation and advantage is to ensure the accurate matching of
photograph’s pixel and point cloud simultaneously collected. A line-based method for
registering point cloud and panoramic photographs collected by a mobile scanning system is
presented in (Cui et al 2017). Its performance was analyzed for two camera models by visual
analysis and quantitative comparison. It has been shown that this method can significantly
improve the matching of photographs and point cloud in the case of an ideal spherical and
rigorous panoramic camera model.
2.3. Working methodology
When surveying the terrain with a mobile laser scanner, coordinates of laser scanner are
recorded at certain time intervals, as well as the inclination angles of the scanner in relation to
all three axes - roll, pitch and heading, using the GNSS (Global Navigation Satellite System)
and INS unit. When surveying, there comes to occasional loss of signal and noise, which causes
errors in the measured position and the slope that needs to be corrected subsequently by taking
corrections from the nearest permanent station. After the trajectory correction is performed, it
is necessary to export a point cloud which will for each point keep x, y and z coordinate exported
from the measured distance from the corrected center of the scanner to the point from which
the laser beam has been reflected.
As noted, when surveying the terrain with a mobile laser system, the digital camera collects
panoramic photographs. For each recorded photograph, the time when the photo was taken is
also recorded. Adjusted, i.e. the corrected trajectory of the laser sensor in addition to the
recorded discrete positions of the sensor and inclination angles in all three axes, contains also
the recorded discrete time of when the trajectory data were measured. Based on the recorded
time, each photo is associated with the appropriate position from the trajectory, and in this way
the photo is matched into the point cloud. Figure 1 shows a folded point cloud with photographs
loaded in the TerraPhoto tool.
Figure 1. Matched panoramic photograph and point cloud
3. FEATURE EXTRACTION METHODOLOGY BU USING ORBIT TOOL
This chapter describes the procedure for extracting characteristic road elements from the point
cloud, not just the digitization of curbs, road edges, road signs, but even defects on the road
surface of any kind, using a particular software package. Extraction is carried out only after the
phase of point cloud matching and largely depends on the quality of that initial step.
3.1. Problem definition
The data obtained by laser scanning of a particular area contains useful information and they
are considered as very valuable data source. Compared to traditional surveying methods, they
are shown as more applicable on different terrains and faster in terms of surveying process and
post-processing as well. Different software are developing and improving, in order to get even
more on the speed and quality of the post-processing. Many of them contain automatic or
semiautomatic recognition algorithms for specific spatial entities, e.g. traffic signs, the edge of
the road, etc. In this paper, Orbit 3DM Publisher was used, which proved to be suitable for the
extraction of spatial elements from the point cloud.
3.2. Related work
Many papers in recent times deals with the extraction of spatial entities from the point cloud.
Thus, in paper (Daniels et al 2007) a robust method that identifies sharp elements in the point
cloud by returning a series of smooth curves set along the edges is presented. This extraction is
an example of the multiphase precision method that uses the concept of robust moving least
squares to locally adjust the surface to the potential elements. Using the Newton method, the
points are projected over multiple surface slices and then penetrate into the polylines through
the projected cloud. After solving the gap and linking the edges, the algorithm returns a set of
complete and smooth curves that define the elements.
(Pu et al 2006) present an approach for automatical extraction of the characteristics of the object
from the point cloud obtained by laser scanning. First, the processing of input data by different
segmentation algorithms is carried out, then several important properties (size, position,
direction, topology, etc.) are selected, and finally the potential elements of the building from
the segments (walls, windows and doors, etc.) with feature constraints, based on the properties
of segments are recognized.
In the paper (Steder et al 2011), the topic of extracting significant elements from the point cloud
for recognition and identification of objects was discussed. A new method of extracting key
points that is working on range images generated from arbitrary 3D point clouds, which
explicitly considers the borders of the objects identified by transitions from foreground to
background. It also presents the descriptor of elements that take into account the same
information.
3.3. Working methodology
As it is already noted, for drawing spatial entities a set of tools for measuring and drawing of
the Orbit 3DM Publisher software package has been used. By matching the photographs to
point cloud and publishing them in the Orbit 3DM Publisher the significant advantage is gained,
since while looking at point cloud, photographs can be observed at the same time and any
possible doubts about the terrain status could be solved. Another advantage of this software is
that it is stored on the server and its easy to access: it is enough to type in the username and
password, and from any computer at any time, it can be accessed. After logging, a map view in
the 2D with the current location and panorama images taken at that location (Figure 2) is
displayed, and the paths where the surveying was done or where it can be moved through Orbit
are marked.
Figure 2. Orbit 3DM Publisher's appearance after logging
After positioning to the specific location, it is possible to include a point cloud, if necessary.
Figure 3 provides a parallel view of the software with and without included point cloud:
Figure 3. Orbit 3DM Publisher without included point cloud (up) and with the displayed point
cloud (bottom)
Regardless of this step, the extraction of the specified elements is done directly on the point
cloud, which means that the corresponding coordinates are assigned to these elements, and then
the drawn element is classified into a certain layer, depending on which type belongs to: Area,
Polyline or Point. In addition to defining a layer, it is possible to automatically define other
attributes, such as the name of an extracted entity, or some geometry data: point coordinates,
line length, polygon surfaces, and so on. All of these data are loaded immediately after the
drawing, and one example is shown in Figure 4:
Figure 4. The extraction of traffic sign into Orbit 3DM Publisher and setting the attributes
An important feature of this software is the possibility of direct export of the extracted elements
into .shp, .kml and .ovf extensions. The Google Earth application opens a kml file, and the
extracted features are positioned directly in this way. Figure 5 shows the extraction of elements
of road infrastructure including curbs, traffic signs and drainage in Orbit 3DM Publisher (left),
and an updated .kml file that is then loaded into Google Earth (right).
Figure 5. Extraction of road infrastructure elements (left), obtained .kml file in Google Earth
(right)
As it is already mentioned, this program proved to be extremely suitable for detection and then
extraction of changes on the road, such as cracks, holes and surface deterioration deeper than 5
mm, patched asphalt etc. By matching the photographs to point cloud, visual inspection is
facilitated, and various types of damage can be detected. After the extraction and export, the
.shp file opens in QGis where it can be manipulated with an attribute table, if all the required
data is not already prepared in the Orbit 3DM Publisher. Figure 6 shows the orthophoto and
polygons with damages. The selected polygon (marked in yellow) contains attributes: type of
damage, code, image, geometry, area, width, and ID.
Figure 6. Polygons of different road defects and the attribute table of one of them
4. CONCLUSION
The extraction of road infrastructure elements based on point cloud recorded by the mobile laser
scanning system is increasingly gaining in importance due to obvious advantages over
traditional surveying methods. By using this method, in less time it is possible to obtain a
significantly larger amount of data, representation of a larger part of the terrain, higher level of
detail and the possibility of subsequent additional analysis and processing of the collected data.
The laser scanning method as such is applicable to a growing number of areas, and
consequently, increasing attention is paid to the further development of this method and
improving its accuracy.
The paper defines the problem and methodology of point cloud and photograph matching, as
well as the possible method of extracting spatial entities of the road infrastructure using Orbit
3DM Publisher. Matching of the point cloud is significant because it minimizes the errors that
occur as a result of the surveying of the terrain. By matching photographs on a point cloud and
by publishing them in the Orbit 3DM Publisher, a significant advantage is gained by viewing
photographs at the same time and seeing the situation on the terrain in order to address possible
concerns, such as, for example, the existence or the absence of certain spatial entities, the
differentiation of advertisements and signs that are important for the traffic flow, the insight
into the traffic light and the identification which direction of movement is regulated by the
traffic. Additionally, the extraction itself is done directly in the photograph and the drawn
entities are directly allocated height from the point cloud. Therefore, there is no need for
additional time spent on creating a digital terrain model and dropping the entities to it in order
to allocate heights. Another important advantage is simultaneously filling the .shp file database
that can be exported and loaded into GIS tools such as QGIS, ArcGIS, and the like.
Note: Project presented in paper is published with kind permission of the contributor. The
original data were provided by DataDEV company, Novi Sad, Republic of Serbia.
REFERENCES
Bang K-I., Kersting A.P., Habib A., Lee D-C., 2009, LiDAR system calibration using point
cloud coordinates in overlapping strips, ASPRS Annual Conference, Baltimore, Maryland
Cui T., Ji S., Shan J., Gong J., Liu K., 2017, Line-Based Registration of Panoramic Images and
LiDAR Point Cloud for Mobile Mapping, Sensors, Vol. 17 (1), 70
Daniels J., Linh K. Ha, Ochotta, T., Silva, C., 2007, Robust Smooth Feature Extraction from
Point Clouds, IEEE International Conference on Shape Modeling and Applications 2007 (SMI
'07), Lyon, France
Forestry Commission, “Introduction to Lidar,” (2012), pp. 1–18.
Kuzmić T., Ninkov T., Bulatović V., Vasić D., Davidović M., 2017, Modern geodetic methods
with application in the environmental management and its protection,7th International
Symposium on Environmental and Material Flow Management –EMFM, pp 59-70, Bor, Serbia
Miled M., Soheilian B., Habets E., Vallet B., 2016, Hybrid online mobile laser scanner
calibration through image alignment by mutual information, ISPRS Annals of the
Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. III-1, pp. 25-31
Ninkov T., Bulatović V., Sušić Z., Vasić D., Marković M., 2014, Modern Acquisition
Technology of Spatial Data as a Basis of Environmental Engineering and Planning Projects,
XXV FIG Congress, pp. 1-9, Kuala Lumpur, Malaysia
Pu S., Vosselman G., 2006, Automatic extraction of building features from terrestrial laser
scanning, International Institute for Geo-information Science and Earth Observation (ITC),
Commission 6
Steder, B., Rusu R., Konolige, K., Burgard, W., 2011, Point feature extraction on 3D range
scans taking into account object boundaries, 2011 IEEE International Conference on Robotics
and Automation, China
Wang J., Shan J., 2009, Segmentation of LiDAR point clouds for building extraction, ASPRS
Annual Conference, Baltimore, Maryland
BIOGRAPHICAL NOTES
Dejan Vasić was born in Sarajevo, Bosnia and Herzegovina, in 1980. He received the Ph.D.
degree in geodesy and geomatics from the Faculty of Technical Sciences (FTS), University of
Novi Sad (UNS), Novi Sad, Serbia in 2018. Currently, he is an Assistant Professor at the FTS,
UNS. His areas of interest are 3D terrestrial and airborne laser scanning, BIM modelling and
Engineering Geodesy.
Mehmed Batilović was born in Tutin, Serbia, in 1992. He received the B.Sc. and M.Sc. degrees
in geodesy and geomatics from the FTS, UNS, Novi Sad, Serbia in 2014 and 2015, respectively.
Currently, he is a Teaching Assistant at the FTS, UNS. His areas of interest are deformation
measurements and deformation analysis of building structures, Earth's crust geodynamic
displacement analysis and 3D terrestrial and airborne laser scanning.
Tatjana Kuzmić was born in Zrenjanin, Serbia, in 1992. She received the B.Sc. and M.Sc.
degrees in geodesy and geomatics from the FTS, UNS, Novi Sad, Serbia in 2015 and 2016,
respectively. Currently, she is an Intern Researcher at the FTS, UNS and works at DataDEV.
Her areas of interest are 3D terrestrial and airborne laser scanning, GIS and GNSS.
Marina Davidović was born in Foča, Bosnia and Herzegovina, in 1992. She received the B.Sc.
and M.Sc. degrees in geodesy and geomatics from the FTS, UNS, Novi Sad, Serbia in 2015 and
2016, respectively. Currently, she is an Intern Researcher at the FTS, UNS and works at
DataDEV. Her areas of interest are 3D terrestrial and airborne laser scanning, 3D cadastre and
land consolidation.
CONTACTS
Assistant Professor Dejan Vasić
University of Novi Sad, Faculty of Technical Sciences
Dositej Obradović Square 6
Novi Sad
Serbia
Tel. +381658097787
Email: dvasic@uns.ac.rs
Web site: http://www.ftn.uns.ac.rs/691618389/fakultet-tehnickih-nauka
Teaching Assistant Mehmed Batilović
University of Novi Sad, Faculty of Technical Sciences
Dositej Obradović Square 6
Novi Sad
Serbia
Tel. +381694441309
Email: mehmed@uns.ac.rs
Intern Researcher Tatjana Kuzmić
University of Novi Sad, Faculty of Technical Sciences
Dositej Obradović Square 6
Novi Sad
Serbia
Tel. +381642627287
Email: tatjana.kuzmic@uns.ac.rs
Intern Researcher Marina Davidović
University of Novi Sad, Faculty of Technical Sciences
Dositej Obradović Square 6
