Content uploaded by Sikandar Ali
Author content
All content in this area was uploaded by Sikandar Ali on Mar 31, 2018
Content may be subject to copyright.
Lane Detection by Combining Trajectory Clustering and Curve
Complexity Computing in Urban Environments
Zhongguo Yang, Hongqi Li*, Sikandar Ali, Yile Ao, Shuang Guo
Key Lab of Petroleum Data Mining, Computer Department
China University of Petroleum, Beijing 102249, China
yangzhongguo@hotmail.com,hq.li@cup.edu.cn,sikandar@uswat.edu.pk,
aoyile@foxmail.com, 2550632954@qq.com
Abstract—Detecting lane automatically from the IP camera is
an important component of the intelligent vision-based traffic
big data system. Many previous studies focus on the main
lanes detection task based clustering algorithm. However,
some left-turning or right-turning lanes are ignored in these
methods due to their seldom happening in real traffic scene.
This paper attempts to address this issue. We try to detect
those lanes which seldom appear in the trajectory lines by
combining main lane detection and curve complexity
computing method. Firstly, vehicles are detected by using
SSD method. Secondly, a vehicle tracking method was
employed to find all the trajectory lines. We recognized the
left-turning lane among the entire main lane trajectory by
computing its trajectory curve complexity. The main lanes
were detected by using fuzzy k-means clustering method and
the similarity computing method. A modified Hausidorff
distance algorithm is incorporated and some experiments are
conducted on an intersection of the urban environment to test
its validity and efficiency.
Keywords-trajectory clustering; lane detection; curve
complexity; Hausidorff distance
I. INTRODUCTION
It is vital to track objects and further analyze their
behaviors with the aim to detect anomalies phenomena
from a video [1, 2]. In traffic applications, there are many
IP cameras in the road but only a small number of them are
used for the vehicle detection purpose as a part of the
automated traffic system. To count the number of going
straight cars, turning left cars, and turning right cars it is
important to detect vehicles generate big traffic data for
further study and analysis of the city’s transport situation.
Moreover, the data could be used to analyze the anomalous
behaviors and to improve the road safety [3-5].
Besides these applications, trajectory detection and
clustering has got popularity in the literature [6, 7]. It is
used for the identification of abnormal or typical events,
such as vehicles changing lanes or illegal maneuvers,
could be detected from the normal patterns.
Lane detection is the basic function of the automated
traffic system. For generating visual data from a camera,
many studies focus on the trajectory clustering method
[8-13]. According to these researchers, choosing a suitable
similarity distance measure is the key challenge using
clustering method. A number of researchers used smoothed
and resampled trajectories as the input data to the cluster
algorithm. Some of them are described below.
Melo et al. [8] first fit the raw trajectory using a
low-degree polynomials function and applied K-means
clustering on the coefficient space to classify lane types.
The highway was classified into four categories: entry, exit,
primary and secondary.
Hu et al. [9] computed the average Euclidean distance
between points on two trajectories and named it as HU
distance. Their distance computing experiment assumes
that the trajectories similar in spatial must own the same
point distribution. They applied a hierarchical spatial-based
clustering algorithm to classify the main lanes.
Bashir et al. [10] worked in reduced dimensionality
subspace instead of the original trajectory coordinate space.
They transformed space by using Principal Components
Analysis (PCA) methods. To capture the temporal relations
and order underlying entities they used, Hidden Markov
Models (HMM) with a data-driven.
Karavasilis et al. [11] introduced a framework to solve
240
2017 13th International Conference on Semantics, Knowledge and Grids
0-7695-6304-X/17/31.00 ©2017 IEEE
DOI 10.1109/SKG.2017.00049
the object motion segmentation and tracking problem
based on clustering trajectories of image key points
extracted from an image sequence. A sparse regression
mixture model that embodies efficient characteristics was
used to handle trajectories with different length.
Hu et al. [12] utilized the spatial and temporal
information of trajectories in a hierarchic clustering model.
Each motion is represented by a series of Gaussian
distributions in their work. Furthermore, the learned
statistical motion patterns are applied to detect anomalies
and predict behaviors. Then, they modified their own work
and proposed a new method to cluster trajectory by using a
Dirichlet process mixture model (DPMM) [13]. Each
trajectory was learned by a time-sensitive DPMM and then
parameterized index is constructed for each.
Many of the previous studies did experiments on the
synthesized data, highway or Australian Sign Language
signs Data Set. However, in the real traffic situation, the
trajectory data set extracted from the video is inadequate
and severely imbalanced. In addition, considering a normal
situation, a car is waiting to turn left while there are many
cars going straight. In this situation, the tracking of the car
may be lost several times before the end of its successful
turn. So the trajectory of the car may be cut into several
short trajectories. On the other hand, these trajectory data
will be severely imbalanced due to the small number of
turning vehicles (left-turning or right-turning) compared
with straight going vehicles, which is the normal
phenomenon. This paper attempts to solve the trajectory
clustering problem for the imbalanced trajectory data set or
noise data sets in the urban environments.
The remaining of the paper is organized as follows. The
framework of the proposed method is introduced in section
2. The curve complexity computing is presented in section
3. In section 4, an example of real urban intersection is
shown and the result is analyzed. The paper is concludes
with a discussion and summary in section 5.
II. PROPOSED SYSTEM
A multistage approach is applied to classify the main
lane and turning lane in the intersection of the real urban
environment. An overview of the proposed system is
shown in Fig. 1.
trajectories for cars
Delete short trajectories
And wrong trajectories
Left-turning
trajectory
detection
Trajectories
clustering
Main lane
estimation
Rare lane
estimation
Output lanes
Online stream
or
Input Video
SSD
Vehicle
detection
Similarity
object
Step 1 Moving Vehicle Detection
Step 2 Vehicle tracking
Adjacent
frame
Curve
complexity
computing
Step 3 Preprocessing trajectories
Step 4
Left-turn lane detection
Step 5 Main lane
detection
track
object
Figure 1. Overview of the proposed system.
Firstly, we trained a vehicle detection model based on
the SSD architecture [20]. SSD is very solid and fast
objects detection architecture which suitable to be used in
the real-time objects detection scenarios. Recently, the
object detection technique has achieved a big perfection;
therefore the tracking of vehicles is easier as compared
with previous studies. The next step is to track the vehicles
by using similarity computing between vehicles detection
in the two adjacent frames. The trajectories of the vehicle
are generated through the tracking of vehicles. Any short
trajectory or noise trajectories are removed based on
criteria such as the number of point in a trajectory should
be more than fifty.
We have constructed a trajectory data set for an urban
road crossing. Fuzzy k-means clustering method [14] is
applied to divide data set into several groups. The distance
used in the fuzzy k-means clustering algorithm is a
modified Hausidorff distance [15-17].
Each group of the trajectories is used to estimate the
main lane. All the trajectories close to the cluster center are
used to estimate the main lane and to remove it from the
original data set. The remaining trajectories are input to the
clustering algorithm to detect other lanes. However, some
seldom happening trajectory means left-turning or
right-turning lanes, which are often ignored by the
clustering algorithm. Different from the previous work, we
employed the curve complexity computing method to
differentiate these trajectories from the original data set.
At the last, all the lanes are detected, the main lanes are
detected by using clustering method and the turning lanes
are detected by using curve complexity computing method.
III. VEHICLE TRACKING AND TRAJECTORY CLUSTERING
The input of the system is the online video stream or the
recorded video. Frist, we employed an object detection
architecture to recognize the vehicle and applied a simple
241
tracking algorithm to track these vehicles and get
trajectories for each vehicle. Then, a fuzzy K-means
clustering method based on a modified Hausidorff distance
as the similarity distance is applied to these trajectories.
A. Vehicle Detection Based on SSD
Object detection is an established filed, there are
abundant of methods proposed to solve this problem and a
big progress has been made in the recent years. Some state
of the art methods consists of Fast RCNN [18], YOLO [19],
and SSD [20]. Considering the performance and speed,
SSD is selected as vehicle detection part in the frame work.
Some examples of vehicles detection are shown in Fig. 2.
Figure 2. The detection of vehicle in the urban environments.
B. Vehicle Tracking
Vehicle tracking is an important step to get trajectory
data from video. In the paper at hand, we use vehicle
matching method to tracking car in the adjacent frame of a
video. The similarity of two objects was detected with the
help of color histogram, Jaccard overlap and cell matching
similarity between two object patches.
Owing to the partial occlusions or suddenly happening
shelter from other vehicles, we often lose the tracking of
vehicles. In such scene, the trajectory would be incomplete
and wrong. So it is necessary to detect such trajectories.
On the other hand, in the real urban road intersection,
the numbers of straight-going vehicles are far more than
the turning vehicles. So the trajectories of an intersection
are severely imbalanced.
1) Vehicle Similarity Model: Lee et al. [21] utilized the
color histogram and cell matching in the car similarity
computing process. In addition to their work, we added the
Jaccard similarity to the final similarity result which
ruminates the little change in the spatial space for a vehicle
in the adjacent frame.
Color histogram_1 Color histogram_2
Histogram
similarity
coordinate coordinate Jaccard
similarity
Similarity computing
Cell_match_score Cell_match_similarity
Figure 3. The procedure of comp uting the similarity between two
objects.
Fig. 5 shows the color histogram having 24 bins with
each RGB channel being spat into 8 bins. The color
histogram is normalized by Eq. (1).
2
()
'( ) (())
i
hist i
hist i hist i
(1)
The correlation of the two histograms is calculated by
the Pearson product moment correlation coefficient
equation shown in Eq. (3).
2222
'() '() '() '()
'() ( '()) '() ( '())
xy x y
xy
xx yy
n hist i hist i hist i hist i
n hist i hist i n hist i hist i
(2)
The Jaccard coefficient measures the similarity between
finite sample sets and is defined as “the size of the
intersection divided by the size of the union of the sample
sets”. . The Equation 5 shows the computing process.
||
(,) ||
AB
JAB AB
(3)
Area of overlap
Area of union
Jaccard similarity =
Figure 4. The meaning of Jaccard similarity.
To compute the cell matching score, we convert the
object image to gray scale and divide them into several sub
regions, in our implementation the number of sub-regions
is 6×8. In each sub region, mean and variance of pixels are
obtained. Comparing mean and variance values with the
other object image, we can set matching cells by Eq. (4).
1, | (i,j) ( , ) |
(, ) | var ( , ) var ( , ) |
0
xym
xy xyv
if mean mean i j
cell i j and ij ij
else
(4)
242
,
_(,)
__ ___
xy
ij
xy
m cell i j
cell match similarity Total number of cells
(5)
The similarity of the two detected objects consists of
three parts: histogram similarity, cell matching similarity,
and Jaccard similarity.
(_ )/3
xy xy xy xy
Similarity m score JaccarSimilarity
(6)
2) Vehicle Tracking Algorithm: The process of tracking
vehicles is shown in Fig. 4. From the process, we can see
that each time we maintain two state variables to record if
a vehicle being matched or not and an another variable to
record the tracking state of vehicles. The tracking state
owns three constant values: 0, 1 and 2 and they mean a
new initialize vehicle, a tracking vehicle and an ended
vehicle correspondingly.
Vehicles
identification
Similarity computing
Similarity > threshold
Update state:
findState=1
Y
N
Iterator all the
vehicles in the list
If the findState=0 and trackingState = 1,
then trackingState=2;
If the findState=1 and trackingState=0,
then trackingState=1
trackingState=2?
Tracking end,
Remove the vehicle
and save trajectory
Read a frame
from video or
online IP camera
Y
Y
Fetch next
frame?
Output trajectories
N
N
Fetch one vehicle
Y
N
Add to list as a new vehicle
findState=0,trackingState=0
Figure 5. Vehicle tracking process.
C. Lane Detection by Trajectory Clustering
1) Preprocessing Trajectories: We attain multiple
trajectories and each one is the representation of a moving
vehicle. There may exist, two short trajectories for one
vehicle. It is due to the disappearance and reappearance of
the vehicle during the tracking process. The reason may lie
in the occlusion, collision, and viewpoint changes and so
on. Especially in dense traffic, a vehicle may stay in a
place for a long time waiting for a red light or waiting for
turning left, the trajectory generated by this vehicle may be
divided into several patches. Assuming some noise data
may lead to singleton cluster. The following rules are taken
into account for clustering before applying the clustering
algorithm to the trajectories:
(i) Number of samples in a trajectory will be greater than
20 (remove short trajectories)
(ii) Average speed of moving object will be greater than 5
pixels per frame (exclude trajectories of slow moving
objects, such as waiting for a red light)
(iii) The distance between the start point and the end point
will be greater than 10(exclude noisy trajectories)
To use fuzzy K-means clustering method in the
trajectory data set. First of all, we resample the trajectory
to a certain length with two hundred points. The trajectory
consists of n coordinates
(, )
tt
xy
and
n
F
is expressed as
follows.
11 22 33 1 1
{( , ),( , ),( , ),...,( , ),..., ( , ),( , )}
nttnnnn
Fxyxyxy xy xyxy
We
let
(, )
ttt
fxy
And
12 1
{ , ,..., ,..., , }
ntnn
Fff fff
The trajectory data set can be presented as S=
12 ,
{ , ,..., ..., }
lM
FF F F
2) Curve Complexity Computing: The expected cluster
lane including going straight lane, turning left lane, turning
right lane are marked with a different colors. An overview
of the lanes distribution is shown in Fig. 6. As we can see
in Fig. 6 that the turn left lane have a bigger curve slope in
the entire and a longer path compared with other
trajectories. Due to the severely imbalanced trajectory data,
it is necessary to distinguish the turning left trajectory, as
for as possible, before employing the clustering algorithm
to the data set.
Figure 6. The expected lanes in an intersection.
243
To differentiate the turn left or turn right trajectory from
the multiple trajectories, a curve complexity computing
method are used. An example of curve complexity is
shown in Fig. 7.
Figure 7. The diagram of computing curve complexity.
In Fig. 7, the
,
and
denote three randomly
selected points and they decided a circle. In the circle, C is
the center point and R means the length of the radius. A
straight line owns an infinite radius and a curve owns a
smaller radius. So it could be used to show the extent of
curvature of a trajectory which could be seeing as a curve.
The curve complexity could be computed as equation 7.
__curve complexity length radius
(7)
Here, we choose a threshold to distinguish the turn left
trajectory from other trajectories. In this way, we leave the
main lane trajectories in the data set which are used as
input to the fuzzy K-means algorithm to find the main
lanes.
3) Fuzzy K-means Algorithm: Before applying a
clustering algorithm to the trajectories, each trajectory is
resampled to a vector with equal number of points. Let
11 22 33 1 1
{( , ),( , ), ( , ),..., ( , ),...,( , ), ( , )}
nttnnnn
Fxyxyxy xy xyxy
be
one of the trajectory and S=
12 ,
{ , ,..., ..., }
lM
FF F F
is the
trajectory data set. Fuzzy K-means clustering method is
applied to find K cluster in the trajectories. Each cluster
centroid is represented by a vector whose dimension is the
same as any other trajectory. The cluster centroid vectors
are initialized randomly and adjusted gradually to form a
mapping from input to output which keeps the distribution
features of the intermediate trajectories.
Let K denote the number of cluster centroids and N the
dimension of cluster centroid vectors. Cluster centroid j is
denoted by an N-dimensional vector
j
V
. For simplicity,
each intermediate trajectory
*
l
T
is represented by an
N-dimensional vector
l
X
. The fuzzy membership
ij
R
of
each sample
l
to each cluster centroid j
(
( 1,2,..., , 1,2,..., )lMjK
is computed using equation 8.
Cluster centroid vectors are updated iteratively by using
the following Equation:
1
1
()*( ())
(1) ()
()
M
lj l j
l
M
jj
lj
l
Rt X Vt
Vt Vt
Rt
(8)
Where
lj
R
is the degree of membership of each sample
l
to
each cluster centroid
j
and it is computed by equation 9.
2
2
1
1/ || || ,1 ,1
(1/ | | || )
lj
K
lj
lm
m
XV
RlMjK
XV
(9)
The end condition is shown below:
1
max{|| ( 1) ( )
jj
jK
Vt Vt
(10)
The distance used is the modified Hausidorff distance
[17] and it is computed using below equations.
(, ) min( (, ), ( ,))
Hij hij hji
DFF DFFDFF
,,
,
,
( , ) max(min ( , )), ,
hij Eikil
f
fik
ik
DFF d f f kl
,,
,
,
( , ) max(min ( , )), ,
hij Ejlik
f
fik
jl
DFF d f f lk
Where
,,
(,)
Ejlik
df f
is the Euclidean distance for two
points
,jl
f
and
,ik
f
coming from trajectory
j
F
and
i
F
correspondingly.
IV. EXPERIMENTAL RESULTS
To evaluate the proposed system, we choose a typical
four intersection crossing roads. We get a half hour’s road
cross video recorded at urban intersection to test our
method. The time is in the rush hour, from nine o’clockto
twenty minutes past nine o’clock. During this time, many
cars, buses, and trucks passed by the crossing road. A
numbers of trajectories are recorded to test the lane
distribution and detection method. Some details of the
trajectories are shown Fig. 8.
Figure 8. The original trajectory detected.
As the Fig. 8 illustrates there are 1883 trajectories in
y
x
P
1
P
2
P
3
C
R
244
twenty minutes. The rules discussed in section 2 are
applied to the original data set. Given the curvature of
left-turning trajectories, we could use the value to
distinguish the kind of trajectories. Moreover, we would
divide the original trajectories into two different groups.
Each group means a different type of trajectories. In
addition, the clustering algorithm will show their effect on
these trajectories.
Figure 9. The turning left lanes.
As depicted in Fig. 9 the left-turning trajectories have a
bigger value of curvature compared to the going straight
trajectories and the turning right trajectories.
Considering the trajectories coming from the
intersection, the K used in clustering algorithm is four. The
result of clustering trajectories is shown in Fig. 10. The
dashes lines are the four main lanes estimated.
Figure 10. The clustering result.
V. CONCLUSIONS
This paper proposed a framework to cluster trajectories
in the intersection into several groups. There are three
main stages in the framework, target detection, trajectories
collection, and trajectories clustering. In the last stage, we
first use the curve complexity computing method to
distinguish the turn left trajectories from the rest of the
data set. The reason lies in the left-turning trajectories
holding a bigger curve complexity than the average
curvature. After removing the left-turning trajectories, the
remaining trajectories are used for future handling. In
addition, the fuzzy K-means clustering algorithm is applied
to the data set. In the clustering process, four main groups
are detected and four main lanes are estimated. In addition,
the trajectories near to the four main groups are removed
from the data set. The rest of trajectories are mainly the
right-turning trajectories. The clustering method is used
again to detect the turn right lanes. To demonstrate the
effectiveness of the proposed method, experiment was
conducted taking one intersection data is an input. The
experimental results confirm that our framework correctly
detects the turn left, turn right, going straight lanes
automatically. Besides, there is only one parameter need to
input, and it is decided by the main lanes number in the
intersection.
Future work will focus on the automatically clustering
parameter selection and make it suitable to a different
intersection.
REFERENCES
[1]Hu, W., Xiao, X., Fu, Z., et al.: ‘A system for learning statistical
motion patterns’,IEEE Trans. Pattern Anal. Mach. Intell., 2006, 28, (9),
pp. 1450–1464.
[2]Hampapur, A., Brown, L., Connell, J., et al.: ‘Smart video surveillance’,
IEEE Trans. Signal Process. Magazine, 2005, 22, (2), pp. 38–51.
[3]Jung, C.R., Hennemann, L., Raupp Musse, S.: ‘Event detection using
trajectory clustering and 4-D histograms’, IEEE Trans. Circ. Syst. Video
Technol., 2008,18, (11), pp. 1565–1575.
[4]Wang, J., Li, X., Liao, S.S., et al.: ‘A hybrid approach for automatic
incident detection’, IEEE Trans. Intell. Transp. Syst., 2013, 14, (3), pp.
1176–1185 .
[5]Pan, T.L., Sumalee, A., Zhong, R.X., et al.: ‘Short-term traffic state
prediction based on temporal-spatial correlation’, IEEE Trans. Intell.
Transp. Syst., 2013,14, (3), pp. 1242–1254.
[6]P. Sadeghi-Tehran and P. Angelov, “A real-time approach for novelty
detection and trajectories analysis for anomaly recognition in video
surveillance system,” IEEE Conference on Evolving and Adaptive
Intelligent Systems, pp. 108 – 113, 2012.
[7]B. Morris and M. Trivedi, “ Trajectory learning for active
understanding: unsupervised, multilevel, and long-term adaptive
approach,” IEEE Transactions on Pattern Analysis and Machine
Intelligence, Vol. 33, No. 11, pp. 2287 – 2301, 2011.
[8]J. Melo, A. Naftel, A. Bernardino and J. Santos-Victor, “Detection and
classification of highway lanes using vehicle motion trajectories,” IEEE
Transactions on Intelligent Transportation Systems, Vol. 7, No. 2, pp. 188
– 200, 2006.
[9]W. Hu, D. Xie, Z. Fu, W. Zeng and S. Maybank, “Sematic-based
245
surveillance video retrieval,” IEEE Transactions on Image Processing, Vol.
16, No. 4, pp. 1168 – 1181, 2007.
[10]F. I. Bash ir, A. A. Khokhar and D. Schonfeld, “Object trajectory
based activity classification and recognition using Hidden Markov
models,” IEEE transactions on Image Processing, vol. 16, No. 7, pp.
1912-1919, 2007.
[11]Karavasilis, V., Blekas, K., Nikou, C.: ‘A novel framework for motion
segmentation and tracking by clustering incomplete trajectories’, Comput.
Vis. Image Understand., 2012, 116, (11), pp. 1135–1148.
[12]Hu, W., Xiao, X., Xie, D., et al.: ‘Traffic accident prediction using
3-Dmodel-based vehicle tracking’, IEEE Trans. Veh. Technol., 2004, 53,
(3),pp. 677–694.
[13]Hu, W., Li, X., Tian, G., et al.: ‘An incremental DPMM-based
method for trajectory clustering, modeling, and retrieval’, IEEE Trans.
Pattern Anal. Mach. Intell., 2013, 35, (5), pp. 1051–1065.
[14]Hu, Weiming, et al. "A system for learning statistical motion
patterns." IEEE transactions on pattern analysis and machine intelligence
28.9 (2006): 1450-1464.
[15]Sun Zongyuan, Fang Shouen. Modeling of vehicle motion trajectories
based on Fuzzy clustering. Journal of Tongji University (natural science).
45.5(2017):699-704.
[16]Atev, Stefan, Osama Masoud, and Nikos Papanikolopoulos.
"Learning traffic patterns at intersections by spectral clustering of motion
trajectories." Intelligent Robots and Systems, 2006 IEEE/RSJ
International Conference on. IEEE, 2006.
[17]Hu, H. Y. "Research on Methods for Traffic Event Recognition Based
on Video Processing." Jilin University, Jilin, China (2010).
[18]Girshick, Ross. "Fast r-cnn." Proceedings of the IEEE international
conference on computer vision.
[19]Redmon, Joseph, et al. "You only look once: Unified, real-time object
detection." Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition. 2016.
[20]Liu, Wei, et al. "Ssd: Single shot multibox det ector." European
conference on computer vision. Springer, Cham, 2016.
[21]Lee, Wono, et al. "Intelligent video surveillance system using
dynamic saliency map and boosted Gaussian mixture model." Neural
Information Processing. Springer Berlin/Heidelberg, 2011.
246
