A Simple and Fast Surveillance System for Human Tracking and Behavior Analysis

Abstract

In this paper, we designed a simple and fast visual surveillance system to track human position and to determine if any abnormal behavior like wall climbing and falling happened. By taking both time and background difference into considerations, illumination effects could be greatly reduced while calculating motion masks. Refinements including holes filling, shadow removal, and noise reduction are done to obtain much more reliable motion masks. However, motion masks corresponding to occluded moving people, greater than a given width, are segmented recursively into smaller ones by bi-modal thresholding. Meanwhile, background could also be updated by the refined motion masks. Integrated location-based and weighted block-based matching is done for object tracking. A similarity is defined from these weighted matched block for object classification. Finally, a couple of criterions are defined to analyze whether objects stop, disappear, climb, or fall. Experimental results are given to demonstrate the robustness of our system.

Full text

A Simple and Fast Surveillance System for Human Tracking and Behavior
Analysis
Chen-Chiung Hsieh and Shu-Shuo Hsu
Department of Computer Science and Engineering, Tatung University, Taipei, Taiwan, R.O.C.
cchsieh@ttu.edu.tw
Abstract
In this paper, we designed a simple and fast visual
surveillance system to track human position and to
determine if any abnormal behavior like wall climbing
and falling happened. By taking both time and
background difference into considerations,
illumination effects could be greatly reduced while
calculating motion masks. Refinements including holes
filling, shadow removal, and noise reduction are done
to obtain much more reliable motion masks. However,
motion masks corresponding to occluded moving
people, greater than a given width, are segmented
recursively into smaller ones by bi-modal thresholding.
Meanwhile, background could also be updated by the
refined motion masks. Integrated location-based and
weighted block-based matching is done for object
tracking. A similarity is defined from these weighted
matched block for object classification. Finally, a
couple of criterions are defined to analyze whether
objects stop, disappear, climb, or fall. Experimental
results are given to demonstrate the robustness of our
system.
KEY WORDS
Video Surveillance; Motion Detection; Object
Tracking; Behavior Analysis;
1. Introduction
More and more people pay attention to visual
surveillance systems for the purpose of security. There
have been many surveillance systems applied to our
surrounding environments, such as airports, train
stations, shopping malls, and even private residential
areas. Motion detection and object tracking are the
most significant tasks in a video surveillance system.
Meanwhile, many motion detection and object tracking
schemes have been proposed. W4[1], a real time
surveillance system, detects and locates people through
a combination of shape analysis and tracking. A video
monitoring system designed by Kim and Kim [2]
utilizes a method for region-based motion segmentation
to extract each moving object. Sakbot proposed by
Cucchiara et al. [3] adopts statistics and knowledge of
segmented objects to improve background modeling
and moving object detection. Moreover, Derek
Anderson et al. [4] presented a fall detection system
achieved by silhouette analysis. A fence climbing
detection system [5] was also proposed to deal with
climbing situations by decoding the state sequence of
the block based HMM.
There are two typical approaches for motion
detection: background subtraction and temporal
differencing [6,7]. Background subtraction refers to a
robust background model while temporal differencing
focuses on two consecutive frames. Background
subtraction can extract complete motion masks, but it
usually takes much time to maintain the background
model. On the contrary, the drawback of temporal
differencing is the incomplete motion masks. We
integrate these two methods for moving regions
detection. Source frame referencing is utilized to fill
the holes. For each motion mask, vertical projection
analysis is applied to segment each moving object. A
fast object tracking method based on location
estimation and weighted block-based similarity
measurement is proposed to track all the moving
objects. Finally, segmented motion mask corresponding
to each moving object will be analyzed by size,
location, and horizontal projection to classify its
behavior such as stopping, disappearing, climbing or
falling. The overall system architecture is as shown in
Fig. 1.
Section 2 focuses on extraction and refinement of
the motion masks. Object extraction by recursive bi-
modal thresholding of the vertical projection is
discussed in Section 3. Object tracking by location
estimation and weighted block matching is described in
Section 4. A couple of criteria are defined in Section 5

for behavior analysis. Experiments are then given in
Section 6 to demonstrate the feasibility and robustness
of the proposed system. Finally, conclusions are made
in the last section.
Figure 1.System block diagram.
2. Motion Mask Refinement
Before start tracking objects, possible moving
regions are extracted by the following process: frame
differencing, holes filling, shadow removal, connected
components labeling, and noise removal. Raw motion
masks as shown in Fig. 2(c) are firstly produced by
intersecting both time difference and background
difference [2]. However, there are always a couple of
vacant areas appearing in the motion masks due to
uniform regions within objects.
2.1. Holes Filling
This problem is frequently encountered especially in
the situation that people are dressed in a uniform color.
Holes exist in motion masks because some motion
pixels are misjudged as non-motion ones. In this paper,
source frame referencing is utilized to fill these holes.
Non-motion pixels or holes adjacent to explicit motion
pixels would be re-classified if they have the same
intensity as the explicit ones. The algorithm is stated as
follows:
Input: Raw motion mask, P
Output: Refined motion mask
Step1. For each motion pixel P(x, y) in the motion
mask, check all the adjacent pixels of P(x, y),
which are denoted as Padj(i, j). Here, the
eight connected pixels are used.
Step2. Padjt(i, j) is set as a motion pixel if Padj(i, j) is
a non-motion pixel and | Padj(i, j
)－
P(i, j) | is
less than a specified threshold.
Step3. Repeat Step2 until all Padj(i, j) are visited
and re-classified.
Step4. Repeat Step1 until no new motion pixels are
found.
The hole-filling procedure stops when no new
motion pixels are added to the motion masks. The
experimental result is shown in Fig. 2 (d). Most holes
are successfully filled. Some regions are not recovered
because their sizes are too small.
(a) Frame at time
t-1
. (b) Frame at time
t
.
(c) Raw motion mask. (d) Refined motion mask.
Figure 2.Motion masks refinement.
2.2. Shadow Removal
The appearance of shadows is due to the light being
cut off by objects. It is frequently encountered,
especially in outdoor environments. However, the
shadows will make it difficult to extract exact motion
masks. Here, we adopt a shadow detection algorithm
proposed by Cucchiara et al [8]. Because Red-Green-
Blue (RGB) color space is less sensitive to brightness
changes, the Hue-Saturation-Value (HSV) color space
is used instead. Luminance (V component) and
chrominance information (H and S components) are
more powerful for detection of brightness changes.
Assume the HSV color of each pixel in the current
frame are PH(x, y), PS(x, y), and PV(x, y), respectively
and the pixels in background model are BH(x, y), BS(x,
Time/Background
Frame Difference
Motion Mask Refinement
Motion Object Extraction
Object Tracking
Object Behavior Analysis
Output Abnormal
Events
Me
ssages

y), and BV(x, y). The shadow mask SD is defined as
follows:













Otherwise
TyxByxPand
TyxByxPand
yxB
yxP
if
yxSD
H
HH
s
ss
v
v
0
),(),(
)),(),(((
),(
),(
1
),(

(1)
The value of α depends on the strength of the light
source. On the other hand, β isalways less than one
providing the flexibility to avoid the small changes in
the background. According to Eq. (1), motion masks
with SD=1 are excluded. Fig. 3 demonstrates that the
algorithm is effective to remove shadows.
(a)
(b) (c)
Figure 3. Motion masks refinement. (a) Original frame.
(b) Without shadow removal. (c) With shadow removal.
2.3. Noise Removal
Frame subtraction produces motion mask as well as
noises caused by illumination changes. The noises
should be removed in order to obtain more accurate
motion masks. Morphological opening and closing
operations, corresponding to erosion and dilation, are
performed to remove noises. However, morphological
operations are not guaranteed to remove all the noises.
To be more robust, the size filter is applied to remove
all small size noises.
3. Object Extraction
3.1. Recursive Bi-modal Thresholding
Motion masks, corresponding to one or more moving
objects, are connected if objects are occluded. Thus, a
bounding region of the motion mask probably includes
multiple moving objects. For example, two people walk
passing by each other. To tackle this problem, vertical
projection analysis is developed to extract each
individual moving object. The vertical projection,
formed by projecting a motion mask vertically, is
assumed to be bell-shaped for a single person. By
formulating as a normal distribution, the standard
deviation could be used to define the boundary for each
object. If the standard deviation of a peak is greater
than a threshold, the area is regarded as a moving
region containing multiple objects. The standard
deviation is defined as follows:
 
,
1
DeviationStandard
1
2


 n
i
iMP
n
(2)
where Mis the mean value of all pi.
Referring to the proposed bi-modal thresholding [9],
the original vertical projection His divided into several
sub-intervals Hx by sliding a window of size 2k+1. The
total pixel number S(Hx) within each Hx is computed.
If S(Hx) is smaller than S(Hx-k) and S(Hx+k),xis
where a valley or local minimum will be located.
Multiple objects can be separated into individuals when
a local minimum is found.
,
otherwise,not valley0
Nto1kallfor),()(and
)()(if,valley1









kxx
kxx
xHSHS
HSHS
H
(3)
where S(Hx)is the total pixel number within Hxand N,
the maximum of k, is approximating half of the interval
number. In Fact, the parameter kdetermines the level
of fault tolerance. False valley resulted from aliasing
noises can be eliminated by giving a proper N. Fig. 4
illustrates the vertical projection analysis for individual
object segmentation.
Figure 4. Vertical projection analysis.
S(Hx)
is the total
pixel number within
Hx
In real situations, the vertical projection of a motion
mask may contain more than two moving objects.
Therefore, multiple objects could be extracted by
applying the bi-modal thresholding recursively if the
object width is known. A real example is also given in
Fig. 5 to show its feasibility. Each extracted object is
described by its minimum bounding rectangle (MBR)
as shown in Fig. 5(d).
S(Hx)
H
x
+2
H
x
+1
H
x
-
1
H
x
H
x
-
2
Valley
……
……

(a) (b)
(c) (d)
Figure 5. Object segmentation by analyzing vertical
projection of motion masks. (a) The source image. (b)
Extracted motion masks. (c) Vertical projection
corresponding to (b). (d) The final result of multiple
objects segmentation.
3.2. Background Updating
In real situations, the background image changes
over time. For example, tree branches swing slightly in
the wind or a stationary object starts moving. As time
varies, the original background image will become less
and less powerful. Thus, it is necessary to update the
background over time. The main concept is to find the
scene changes in the non-motion areas and to update
the current intensity to the background image. If the
refined mask indicates a pixel is a non-motion one but
the criterion Db=),(),( yxByxP
t

a given threshold,
the pixel is regarded as a scene change. The intensity
value will be then updated to the background image to
form a new one.
Different from the scheme proposed by Kim and
Kim [2] which updates its background immediately
after background subtraction, our system updates the
background image after the extracted motion masks are
refined completely. A complete and accurate motion
mask can be combined with the criterion defined above
to form the background update function. Assume each
pixel within the refined motion mask is denoted as Rt(x,
y). The updated background B’(x, y)can be constructed
by the following equation:



otherwiseyxB
yxDyxRifyxP
yxB btt
),(
1),(),(),,(
),(
', (4)
where Db(x,y) is the calculated scene change. Figure 6
illustrates how the system updates the background
image over time. In the test video streams, a person lay
on the ground for a long time and was regarded as a
part of the background. Then, the person started
moving again as shown in Fig. 6(a). Figure 6(b) shows
the recovery of the background image as time varies.
Eventually, the background image was updated
correctly as shown in the last picture of Fig. 6(b).
4. Object Tracking
Each extracted moving object would be recorded
and matched with all existing models for the purpose of
tracking [10,11]. In order to match objects, the
similarity between a moving object and each recorded
object model is calculated. The moving object is
identified as the model with the largest similarity. Here,
we proposed a weighted block-based similarity
measurement. However, object tracking could be quite
simple if there is only one moving object found in the
previous frame near that location, these two objects X
and Y are recognized as the same one.
(a)
(b)
Figure 6. Demonstration of background updates. (a) A
person lay on the ground for a long time and then
moved away; (b) The updated background over time.
4.1. Weighted Block-based Similarity
Measurement
An unlabelled moving object is firstly divided into
blocks of size 8 × 8. There are some well-known
distance measurement methods, such as MSE (Mean
Square Error), MAD (Mean Absolute Difference), and
NCCF (Normalized Cross Correlation Function). To
consider both computational cost and correctness, we

adopted NMAD (Normalized Mean Absolute
Difference) for the distance measurement.
 
,255),(),(
1
NMAD
1 1
2211

 
 m
i
n
j
jyixPjyixP
mn (5)
where P(x1, y1)and P(x2, y2)are the intensity of the
pixel located in (x1, y1)and (x2, y2), respectively. The
parameters mand n, denote the block size.
Experimental results showed that the corresponding
blocks would distribute uniformly if we search in the
correct model. On the contrary, if we search the
corresponding blocks in a wrong model, most of the
blocks found would distribute disorderly and overlap
each other. Therefore, each block is given a weight to
represent its reliability. The area of the overlapped
pixels is counted for each matched block. The greater
the overlapped area is, the greater the weight is as
shown in Eq. (6):
blockaofArea
pixelsoverlappedofArea
wxy (6)
The similarity is defined by Eq. (7). Each extracted
moving object is assigned as the model with the largest
similarity.
,%100)),(1(
1
Similarity
1 1
















 
 
M
x
N
y
xy yxDw
MN
(7)
where D(x, y) represents the minimum NMAD between
a block of a moving object and the corresponding
block in the model. Fig. 7 gives an example that both
moving objects were accurately labelled by this
weighted block-based matching method.
(a) (b)
Figure 7. Similarity measurement for object 1. (a) The
correct one with larger similarity of 49.03 (b) The
incorrect one with smaller similarity of 42.12.
4.2. Occlusion Detection
The object tracking proposed in the pervious section
works even occlusion occurs because we have saved
the object models before occlusion. However, the
significant issue is how to determine the exact time for
saving objects as models. The models must be saved
before the objects overlap. Therefore, an occlusion
detector described by Eq. (7) is developed. An alarm
will be triggered when two objects overlap. In the
equation, the MBRs of the objects at time t are
compared with those at time t-1.
,
),()(if1
initially0,inobjecteachFor
11






x
tttx
x
xt
C
frameinyobjectyMBRxMBRC
C
Cframex (8)
If the value of Cxis greater than 2, occlusion occurs
within object x. When occlusion is detected, the
tracking system still keeps the model of each object and
turns to track with a temporary overlapping object
model. As soon as the overlapping objects separate
again, each of them can be detected and tracked
accurately by the saved models in the tracking system.
5. Behavior Analysis
In order to recognize abnormal behaviors, a couple
of criteria are defined in our system. Once the
suspicious behaviors are detected, the system will set
off an alarm to the security officers. Several typically
abnormal behaviors, including wall climbing, stopping,
disappearing, and falling, are discussed in this section.
5.1. Stopping and Disappearing
In real situations, pedestrians are not always moving.
They possibly stop in a sudden. However, frame
difference does not work well when objects keep still.
In our system, each moving object’s location is
recorded. If a location is occupied by object ibut
released in the next frame, object iis recognized as a
stationary object. Moreover, if object ikeeps still for a
while and its location is close to the boundaries, the
system would consider object idisappeared.









,,Lif,Normal0
otherwise,Stopping1
)(
otherwise,Normal
and1if,ngDisappeari
)(
1-t ill
iS
S(i)MBR(i)S
iD
t
BDtt
t(9)
(10)
where Ltis a set of existing label number at time tand
SBD is a set of predefined boundary areas.
5.2. Wall Climbing
Obviously, the motion vectors tend to be upward
when an object is climbing. The center of an object iat
time tis defined as Ct(i). By comparing Ct(i) with the
previous ones, the motion vector MVt(i) could be
calculated. If MVt(i) is upward, object iwould be

judged as a climbing object. To differentiate the wall
climbing from a small jump, MVt(i) must bigger than a
threshold, TSc.
,
otherwise,Normal
)(if,Climbing
)( 


ctt-kt
t
TS(i).y(i).y - CC(i)MV
iCL (11)
where Ct(i).y is the ycoordinate of object i’s center at
time t.
5.3. Falling
As mentioned in the previous section, vertical
projection could be used to extract multiple moving
objects. Likewise, horizontal projection could be
applied as well to detect whether a monitored person
falls. The horizontal projection of a falling person,
formed by projecting the motion mask horizontally, is
assumed to be bell-shaped and could be considered as
normal distribution which means the standard deviation
is less than a threshold.
,
otherwise,Normal
if,Falling
)( 


fdt
t
TS(i)SD
iFD (12)
where SDt(i) is the standard deviation according to the
horizontal histogram of object iat time t.
We tested three different abnormal behaviors
including stopping, falling, and wall climbing. As
shown in Fig. 6.3, all abnormal behaviors were
detected by the proposed algorithms. The warning
messages were shown on the screen when abnormal
behaviors were detected.
(a) (b) (c)
Figure 8. Abnormal behavior detection. (a) Two
people walked toward each other, and then stopped to
shake hands; (b) A person was walking and fell all a
sudden; (c) Someone climbed over a fence.
6. Experimental Results
A series of scenarios were tested in order to
demonstrate the robustness of the proposed system.
Videos were captured outdoors with resolution of
352x240 in several different environments. Our
program ran on Intel® Pentium® 4 3.4 GHz processor
with 512MB RAM and 60GB hard disk drive.
The proposed system is designed to detect and track
moving objects in real-time. In order to verify the
accuracy and efficiency of the proposed system, five
video sequences were tested. Table 1 shows the
successful matching rates of location based estimation
and weighted block-based matching. The errors for
location estimation were caused by un-detected
occlusion near edges and wrong motion masks. As to
the errors for weighted block-based matching, they
were caused by similar objects like people wearing
clothes in same color. Table 2 shows the elapsed
processing time and the processing frame rate.
Table 1:Accuracy verification.
The above experimental results show that our system
can successfully detect and track moving objects for
most situations. The accuracy would be higher if object
behaviors are taken into consideration. On the other
hand, the frame rate is around 6 frames per second. It
can be used in most normal environments. However,
the frame rate can still be raised by upgrading the
hardware equipment.
Table 2: Efficiency verification.
Frame
Number
Execution
Time(s)
Frame
Rate(fps)
Vedio1 439 38 11.55
Vedio2 877 85.6 10.62
Vedio3 1017 97.6 10.42
Vedio4 2790 258 10.8
Vedio5 4012 400 10.03
7. Conclusions and Future Works
In this paper, we designed a simple and fast visual
surveillance system which successfully detects moving
objects and continuously tracks and locates them. A
novel approach by source frame reference was used to
refine the binary motion mask. Most vacant areas in the
raw motion masks were patched well by this step. Then,
multiple-object segmentation was accomplished by
analyzing the vertical projection of motion masks.
Finally, all extracted moving objects were accurately
tracked by the integration of location estimation and
weighted block-based similarity measurement among
saved models in the tracking system.

To demonstrate the feasibility and robustness of our
system, video data were captured outdoors in several
environments such as pathway, corridor, and entrance.
Experimental results showed that the system efficiently
detected and tracked multiple moving objects even if
occlusion occurred. Illumination variations and small
changes in the background were tolerable as well.
Improper behaviors like wall climbing, falling,
stopping, and disappearing were also recognized
correctly.
However, the system can be extended with more
future work to deal with people and vehicle counting,
suspicious behavior analysis, theft detection, and so on.
Tracking rigid objects like cars differs significantly
from tracking semi-rigid objects. Appearance-adaptive
models proposed by Zhou et al. [12] can be applied to
achieve this goal. If all rigid and semi-rigid objects are
detected and tracked, the system can count people and
vehicles respectively. With this possible, traffic control
could be realized. The system could be improved to
recognize more suspicious behaviors in the future, such
as car stealing, bank robbery, firearm activities, or
abnormal wandering. More intelligent schemes can be
applied to our system to make it more powerful in
behavior analysis. A human star skeletonization motion
analysis scheme proposed by Fujiyoshi et al. [13] could
be utilized to recognize abnormal behaviors. With
more useful features added into the system, it will bring
forth benefits of a more complete system.
Acknowledgments
The authors would like to thank the National Science
Foundation (NSC) of the Republic of China (ROC) for
financially supporting this research under project No:
NSC 95－2221－E－036－003.
References
[1] I. Haritaoglu, D. Harwood, and L. S. Davis, “W4: Real-
Time Surveillance of People and Their Activities”,IEEE
Transactions on Pattern Analysis and Machine Intelligence,
Vol. 22, No. 8, pp. 809-830, August 2000.
[2] J. B. Kim and H. J. Kim, “Efficient Region-Based Motion
Segmentation for a Video Monitoring System”,Pattern
Recognition Letters, Vol. 24, pp. 113-128, 2003.
[3] R. Cucchiara, C. Grana, M. Piccardi, and A. Preti.,
“Detecting Moving Objects, Ghosts, and Shadows in Video
Streams”,IEEE Transactions on Pattern Analysis and
Machine Intelligence, Vol. 25, No. 10, pp. 1337-1342,
October 2003.
[4] D. Anderson, J. M. Keller, M. Skubic, X. Chen, and Z.
He, “Recognizing Falls from Silhouettes”,Proceedings of
the 28th IEEE EMBS Annual International Conference, pp.
6388-6391, New York, Sep., 2006.
[5] E. Yu, and J. K. Aggarwal, “Detection of Fence Climbing
from Monocular Video”,Proceedings of the 18th
International Conference on Pattern Recognition, pp. 375-
378, 2006.
[6] S. Joo and Q. Zheng, “A Temporal Variance-Based
Moving Target Detector”,IEEE VS-PETS, Jan. 2005.
[7] Q. Wu, H. Cheng, and B. Jeng, “Motion Detection via
Change-Point Detection for Cumulative Histograms of Ratio
Images”,Pattern Recogmition Letters, Vol. 26, pp. 555-563,
2005.
[8] R. Cucchiara, C. Grana, M.Piccardi, A. Prati, and S.
Sirotti , “Improving Shadow Suppression in Moving Object
Detection with HSV Color Information”,Proceedings of
IEEE Intelligent Transportation System Conference (ITSC
2001), Oakland, CA, USA, pp.334-339, Aug., 2001.
[9] H. Shen and C. R. Johnson, “Semi-Automatic Image
Segmentation: A Bimodal Thresholding Approach”,
Technical Report UUCS-94-019, Un. of Utah, Dept. of
Comp. Science, 1994.
[10] M. F. Abdelkader, R. Chellappa, and Q. Zheng,
“Intergrated Motion Detection and Tracking for Visual
Surveillance”,The Fourth IEEE Conference on Computer
Vision Systems, pp. 28-34, Jan. 2006.
[11] A. Gyaourova, C. Kamath and S. Cheung, “Block
Matching for Object Tracking”,Tech. Rep. UCRL-TR-.
200271, Lawrence Livermore National Laboratory, Oct.
2003.
[12] S. K. Zhou and R. Chellappa, “Visual Tracking and
Recognition Using Appearance-Adaptive Models in Particle
Filters”,IEEE Transactions on Image Processing, Vol. 13,
No. 11, pp. 1491-1506, November 2004.
[13] H. Fujiyoshi and A. J. Lipton, “Real-time Human
Motion Analysis by Image Skeletonization”, Proc of IEEE
Workshop on Application of Computer Vision, pp. 15-21,
October 1998.