Content uploaded by Shabbir N Merchant
Author content
All content in this area was uploaded by Shabbir N Merchant on May 31, 2013
Content may be subject to copyright.
IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—II: EXPRESS BRIEFS, VOL. 53, NO. 10, OCTOBER 2006 1039
Interpolated Candidate Motion Vectors for Boundary
Matching Error Concealment Technique in Video
S. Garg and S. N. Merchant
Abstract—The pertinency of error concealment (EC) schemes
to the encoder–decoder model makes them an attractive choice to
conceal the effects of transmission errors in a compressed video bit-
stream. Among various EC techniques, boundary matching (BM)
has been the most popular due to its distortion measure criterion
to select the most appropriate motion vector (MV) for damaged
macroblock among a set of candidate MVs (CMVs). In a scenario
where error bursts do not extend to more than one frame of digital
video, the performance of BM can be highly enhanced if the CMVs
depend upon an interpolation of surrounding MVs. This interpola-
tion needs to be done, considering the change in their pattern from
the reference to the current erroneous frame. This brief proposes a
technique to build this novel set of CMVs and compares the results
of BM with traditional CMVs qualitatively and quantitatively for
MPEG test sequences.
Index Terms—Boundary matching (BM), candidate motion vec-
tors (CMVs), error concealment (EC), video transmission.
I. INTRODUCTION
T
O achieve efficient video transmission over error-prone
channels, error resilience techniques [1], [2] and en-
coder–decoder interactive techniques [3] try to mitigate the
effects of errors by adding redundant information while
encoding. Error concealment (EC) methods, which are im-
plemented at the decoder side, present another convenient
way of dealing with this problem, as they do not require any
change in the encoder–decoder structure. In EC techniques, the
decoder attempts to conceal the effects of errors by providing
a subjectively acceptable approximation to the original data.
This is achieved by exploiting the limitations of the human
visual system and the high temporal and/or spatial correlation
of video sequences.
A classification of the EC techniques can be done as spatial
interpolation [4]–[8], temporal interpolation [6], [8]–[11], and
motion-compensated temporal interpolation. Neither spatial nor
temporal interpolation techniques account for the motion be-
tween the current erroneous frame and the previous frame. This
is taken into consideration by motion-compensated EC tech-
niques. Motion-compensated concealment can be implemented
in various ways, of which averaging (AV) [3], [12] and boundary
matching (BM) [13]–[16] are the most popular. Al-Mualla
et al.
[3] provide simulation results to display the superior perfor-
mance of BM over AV and attributes it to the distortion mea-
sure criterion of BM to find a concealed motion vector (MV) for
Manuscript received August 12, 2005; revised January 18, 2006. This paper
was recommended by Associate Editor A. Loui.
The authors are with the Signal Processing and Artificial Neural Net-
works Laboratory, Department of Electrical Engineering, Indian Institute of
Technology Bombay, Mumbai 400076, India (e-mail: shanks@ee.iitb.ac.in;
merchant@ee.iitb.ac.in).
Digital Object Identifier 10.1109/TCSII.2006.882205
a damaged macroblock (MB). There are a variety of EC tech-
niques in which BM is used. Lam et al. [13] use the displaced
frame difference (DFD) from the top of the missing MB and
then employed BM to estimate the missing MV. This algorithm
is called extended BM (EBM). Although [17] provides the case
for failure of BM due to the presence of slanting edges and high
gradient of gray levels, [17] and [18] present an improved BM
to palliate their effects. Lee et al. [19] present a multiframe re-
covery principle based on BM to minimize the effect of errors,
not only in the current decoded frame but also in the succeeding
dependent frames. BM dependency is also present in [20] for
forward and backward motion tracking, [21] for concealment
using long-term and short-term reference frame, [22] for gra-
dient-based BM (GBM), [23] for bitstream level error correc-
tion mechanism, and [24] for concealment in object-based video
coding. All the cited works on BM establish that it scores over
other EC techniques, but a little insight into its application pro-
cedure can further enhance the results. The principle behind BM
is that continuity of objects passing through a damaged MB shall
be maintained. The continuity measure is called side-match dis-
tortion (SMD), which is defined in [3] as
SMD
SMD SMD SMD SMD (1)
where SMD
is the sum of absolute, or squared, differences
across the left boundary of the damaged block, when con-
cealed using the
th MV of the candidate MV (CMV) bank
. Likewise, SMD , SMD , and SMD are
the differences across the right, top, and bottom edges of the
damaged MB, respectively. Based on the smoothness property
of the video signals, the CMV that achieves the minimum SMD
is chosen as the concealed MV.
This brief focuses on this aspect of BM and defines a new in-
terpolated CMV (ICMV) bank in Section II. Section III presents
the simulations and testing results carried out to study the com-
parisons in the concealed frames. The results include perceived
video quality and the peak signal-to-noise ratio (PSNR) values
of concealed frames.
II. ICMV
S
The authors in [13]–[24] use the CMVs for BM as a combi-
nation of some or all of the following parameters:
1) zero MV;
2) MV of the spatially corresponding MB in the previous
frame;
3) MVs of the available neighboring MBs;
4) mean of the available neighboring MVs;
5) median of the available neighboring MVs.
1057-7130/$20.00 © 2006 IEEE
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on December 1, 2008 at 04:35 from IEEE Xplore. Restrictions apply.
1040 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—II: EXPRESS BRIEFS, VOL. 53, NO. 10, OCTOBER 2006
Fig. 1. Interpolation procedure of CMVs.
The basis for these choices is high temporal correlation of video
sequences and high correlation of MVs of neighboring MBs
(high spatial correlation). However, BM might fail for areas with
low spatial correlation, e.g., at the boundaries of moving objects
as illustrated with results in [3]. Such anomalies are also easy
to notice and affect the perceived video quality considerably.
BM performs poorly because the aforementioned CMVs are not
representatives of the true motion type for such areas. If the pat-
tern of change in MVs from the reference (correctly received or
concealed) frame to the current erroneous frame is included in
CMVs, BM would have much more meaningful CMVs to esti-
mate a damaged MV.
This technique can be explained with the help of Fig. 1. Using
the notation where
, , , and denote the MVs for
the MBs to the left of, to the right of, above, and below the
damaged block in the current erroneous frame. Let
, , ,
and
denote the MVs for the spatially corresponding MBs
in the previous frame. In addition,
denotes the MV of MB
in the previous frame at the spatial position of damaged block.
Further, any MV can be broken down as
, where
and are a part of the video bitstream.
The procedure starts by gathering the relationship of dam-
aged MB’s neighboring MVs with respective MVs of spatially
corresponding MBs in the reference frame. Then, this relation-
ship is used on
to compute an ICMV for BM to estimate a
concealed MV for damaged MB. In the simplest form, let there
be four ICMVs denoted by icmv
, icmv , icmv , and icmv .
First, the relationship of
, , , and is computed with
, , , and , respectively. Then, these relationships are
applied on
to estimate icmv , icmv , icmv , and icmv ,
respectively.
Since MVs are vectors, they can be treated as complex num-
bers in polar form, i.e.,
. Furthermore, ICMVs,
which are also represented in polar form, can be calculated by
icmv
(2)
The benefit of polar-form representation is that it separates the
magnitude part and the direction part of an MV. This helps in
applying the relationships in such a way that the first part in
(2) represents translation (magnitude scaling) and the second
part represents rotation. ICMVs now contain the information of
possibly how the MVs of neighboring MBs could have changed
their pattern for the skip from the reference frame to the current
frame, and then what the potential MVs are if the same patterns
are applied on
. In this way, there would be more information
available in CMVs, because along with the exploitation of tem-
poral and spatial correlation properties of the video signals, the
interpolation of motion pattern is also infused in the relation.
Developing the idea, ICMVs can be calculated using MVs
not constrained to the immediate neighboring MVs only. Due
to VLC representation of MVs, even a single bit error can lead
to synchronization loss in video decoding, and a series of MBs
can be lost. In such a case, left and/or right MVs may not be
available. For this, ICMVs are computed using nearest available
neighbors, and two indices indicate the position of MB with
respect to the damaged MB, whose MV is used. In this extended
scenario, ICMVs have the following notation:
ICMV
(3)
where
and indicate the location of available neighboring
MBs with respect to the damaged MB.
and can take pos-
itive or negative integer values and signify which corresponding
MBs to use in reference and current frames to estimate possible
relationships of MVs.
The overhead for computing ICMVs is also nominal in terms
of computational complexity and memory requirements. The
extra computation effort would be to transform MVs to polar-
form coordinates and then compute ICMVs. Each MV would re-
quire three multiplication/division and one addition/subtraction
operation for polar-form transformation. Further, each ICMV
calculation consumes two multiplication/division and two ad-
dition/subtraction operations. Therefore, for a set of
ICMVs,
which successively requires
MVs in polar form, the
total extra computations would be
multiplication/divi-
sion and
addition/subtraction operations. For four basic
ICMVs, these values are 23 and 13, respectively. Some memory
would also be required to store the MVs of the reference frame,
which are otherwise erased from the memory as soon as a new
frame arrives for decoding. Considering an MB size of
,a
frame size of
, and bytes for each component of MVs;
the extra memory required would be
bytes. For CIF
resolution (352
288), 32-bit MV components, and 16 16
MB size, this value is around 3 kB, which can easily be made
available in any application.
Now, a new set of CMVs can be proposed to have a combi-
nation of the following parameters:
1) zero MV;
2) MV of the spatially corresponding MB in the previous
frames;
3) ICMVs;
4) mean of ICMVs;
5) median of ICMVs.
CMVs described in 1) and 2) have been taken from the tradi-
tional set of CMVs mentioned at the start of this section. These
two CMVs take care of the most simple case of no motion in
erroneous areas, such as background or temporally constant re-
gions for the reference and current frames. It should also be
noted that CMVs mentioned in 3) of earlier set are just a special
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on December 1, 2008 at 04:35 from IEEE Xplore. Restrictions apply.
GARG AND MERCHANT: ICMVs FOR BOUNDARY MATCHING ERROR CONCEALMENT TECHNIQUE IN VIDEO 1041
TABLE I
P
ERFORMANCE OF
BM
FOR
SUSIE
TABLE II
P
ERFORMANCE OF BM FOR SILENT
case of ICMVs. This case would be evident if and are to-
tally correlated. Then, icmv
would simply reduce to in
(2), which is indeed 3) earlier. In fact, the use of available neigh-
boring MVs as a part of traditional CMVs is based upon the
assumption of this total correlation, which may not be present
in practical video sequences. With regard to temporal motion
pattern information, interpolation procedure applied in ICMVs
computes and incorporates the magnitude of actual correlation.
Thus, this new set of CMVs would provide improved inputs
to BM for majority of the damaged MBs. It is important to note
that considering both sets of CMVs would further increase the
performance of BM, but it would also result in a considerable
increase of computational complexity. This performance–com-
plexity tradeoff has been factored in the following section along
with simulation results.
III. S
IMULATION AND TEST RESULTS
The MPEG test sequences SUSIE,SILENT,FOREMAN,
S
TEFAN, and T
ABLE TENNIS were obtained from MPEG test
bitstreams source (http://www.mpeg.org) for simulations. The
criteria to test the performance are perceived video frame
quality and PSNR. Since BM falls under the category of mo-
tion-compensated concealment, the performance was tested
taking interframe MB-type mode for all MBs. Interframe
interval was increased to 60 frames for prolonged judgement of
results with the proposed ICMVs.
A. Performance Evaluation
The objective quality was tested with CMV banks containing
traditional CMVs for classical (BM) and EBM algorithms.
ICMVs were employed for BM+, and a combination of both
CMVs and ICMVs were considered for BM++. Tables I–IV
list the PSNR values in decibels for various frames of test
sequences for a 20% MB loss rate. For every sequence, the
first row represents the frame in which the error was initially
introduced. The following rows indicate how the PSNR values
varied for subsequent frames after some intervals. The last
row indicates the average PSNRs calculated over the entire 30
frames.
The BM+ performance is always better than the BM for all
the test sequences, and BM++ performs even better. In gen-
eral, the performance of BM for both CMV banks went down
TABLE III
P
ERFORMANCE OF
BM FOR FOREMAN
TABLE IV
P
ERFORMANCE OF BM FOR STEFAN
Fig. 2. Original tenth frame of T
ABLE
TENNIS.
for increasing frames in all the test sequences, which is ex-
pected. For sequences F
OREMAN and S
TEFAN, which have high
motion magnitude, the average improvement is around 1 dB.
For low-motion-magnitude clips, i.e., S
USIE and SILENT, the av-
erage gain is around 1.5 dB. An interesting feature to note is
that for increasing subsequent frames, the difference in BM and
BM+ performance increases. BM++ also follows the same trend
with BM+, but the increase in performance was very less as
compared to the magnitude of computational complexity over-
head. From BM to BM+, the percentage increase in performance
was 5%–10% for a complexity increase of approximately 10%.
From BM+ to BM++ performance, increase was 1%–5% for
a complexity increase of around 40%, because the number of
CMVs for BM++ increased considerably. This is in accordance
with our analysis that ICMVs would contribute better MVs for
BM for majority of the damaged MBs.
The subjective study was done on the tenth frame of T
ABLE
TENNIS as shown in Fig. 2. The corresponding erroneous frame
is shown in Fig. 3 with a 20% MB loss rate. Thereafter, Figs. 4
and 5 present the perceived quality of the concealed frames for
BM with traditional CMVs and ICMVs, respectively.
There are some noticeable improvements by the proposed
technique at the right elbow of the player—its shadow on the
wall and left thigh. These have been enlarged and displayed in
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on December 1, 2008 at 04:35 from IEEE Xplore. Restrictions apply.
1042 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—II: EXPRESS BRIEFS, VOL. 53, NO. 10, OCTOBER 2006
Fig. 3. Erroneous frame with a 20% macroblock loss rate.
Fig. 4. Concealed by BM with traditional CMVs’ bank. PSNR
dB.
Fig. 5. Concealed by BM with ICMVs’ bank. PSNR dB.
Figs. 6–8, respectively. For these figures, (a) shows the orig-
inal, (b) shows the BM performance with traditional CMVs, and
(c) shows the BM performance with ICMVs. As the figures il-
lustrate, ICMVs must have provided BM with better estimated
MVs as the concealed part in (c) is much closer to the original
part in (a). More part of elbow has been reconstructed well, the
Fig. 6. Comparison of the elbow region. (a) Original. (b) Concealed by BM
with traditional CMVs’ bank. (c) Concealed by BM with ICMVs’ bank.
Fig. 7. Comparison of the shadow region. (a) Original. (b) Concealed by BM
with traditional CMVs’ bank. (c) Concealed by BM with ICMVs’ bank.
Fig. 8. Comparison of the thigh region. (a) Original. (b) Concealed by BM with
traditional CMVs’ bank. (c) Concealed by BM with ICMVs’ bank.
shape of shadow resembles much to the original and the thigh
part is closer to the leg. However, there are still some deformi-
ties in the upper arm and shadow, which must have been present
due to unavailability of neighboring MBs belonging to the ob-
ject area.
IV. C
ONCLUSION
The performance of BM in video EC is highly superior than
the other techniques in its category. However, the results pro-
duced by BM are dependent upon the CMVs supplied to it for
computing least distortion in the concealed frame. A new class
of ICMVs that uses the information about motion pattern along
with temporal and spatial correlation property of MVs is pro-
posed. The principle of ICMVs is to estimate the relationship
between MVs around the damaged block from the reference and
current frames and estimate possible MVs for damaged block if
the relationships persisted.
Simulation results displayed that with ICMVs forming a part
of the CMV bank, the performance was better than the tradi-
tional CMVs for a variety of test sequences. In perceived video
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on December 1, 2008 at 04:35 from IEEE Xplore. Restrictions apply.
GARG AND MERCHANT: ICMVs FOR BOUNDARY MATCHING ERROR CONCEALMENT TECHNIQUE IN VIDEO 1043
quality, the improvement was also visible at critical (high mo-
tion) boundary regions.
The proposed strategy can be easily inducted into existing
framework as it does not force a change in other existing mod-
ules. Considering that BM forms an underlying principle for
a large number of concealment techniques, improvement in its
basic inputs would result in better performance of all these tech-
niques.
R
EFERENCES
[1] Y. Wang, S. Wegner, J. Wen, and A. K. Kastaggelos, “Error resilient
video coding techniques,”
IEEE Signal Process. Mag., vol. 17, no. 4,
pp. 61–82, Jul. 2000.
[2] R. Talluri, “Error resilient video coding in the ISO MPEG-4 standard,”
IEEE Commun. Mag., vol. 36, no. 6, pp. 112–119, Jun. 1998.
[3] M. E. Al-Mualla, C. N. Canagarajah, and D. R. Bull, Video Coding for
Mobile Communications. New York: Elsevier, 2002.
[4] M. Ghanbari and V. Seferdis, “Cell loss concealment in ATM video
codecs,” IEEE Trans. Circuits Syst. Video Technol., vol. 3, no. 3, pp.
238–247, Jun. 1993.
[5] P. Salama, N. B. Shroff, E. J. Coyle, and E. J. Delp, “Error concealment
techniques for encoded video streams,” in Proc. IEEE ICIP, 1995, vol.
I, pp. 9–12.
[6] S. Aign and K. Fazel, “Temporal & spatial error concealment
techniques for hierarchical MPEG-2 video codec,” in Proc. IEEE
GLOBECOM, 1995, pp. 1778–1783.
[7] Y. Zhao, D. Tian, M. M. Hannukasela, and M. Gabbouj, “Spatial error
concealment based on directional decision and intra prediction,” in
Proc. ISCAS, 2005, vol. 3, pp. 2899–2902.
[8] M. Chen, C. Cho, and M. Chi, “Spatial and temporal error concealment
algorithms of shape information for mpeg-4 video,” IEEE Trans. Cir-
cuits Syst. Video Technol., vol. 15, no. 6, pp. 778–783, Jun. 2005.
[9] Y. Wang and Q. F. Zhu, “Error control and concealment for video com-
munication: A review,” Proc. IEEE, vol. 86, no. 5, pp. 974–997, May
1998.
[10] A. Narula and J. Lim, “Error concealment techniques for an all-digital
high definition television system,” in Proc. SPIE Conf. VCIP, 1993,
vol. 2094, pp. 304–315.
[11] M. Chen, C. Chen, and M. Chi, “Temporal error concealment algo-
rithm by recursive block matching principle,” IEEE Trans. Circuits
Syst. Video Technol., vol. 15, no. 11, pp. 1385–1393, Nov. 2005.
[12] Q. F. Zhu, Y. Wang, and L. Shaw, “Coding and cell-loss recovery in
DCT-based packet video,” IEEE Trans. Circuits Syst. Video Technol.,
vol. 3, no. 3, pp. 248–258, Jun. 1993.
[13] W. M. Lam, A. R. Reibman, and B. Liu, “Recovery of lost or erro-
neously received motion vectors,” in Proc. IEEE ICASSP, 1993, vol.
V, pp. 417–420.
[14] K. W. Kang, S. H. Lee, and T. Kim, “Recovery of coded video se-
quences from channel errors,” in Proc. SPIE Conf. VCIP, 1995, vol.
2501, pp. 19–27.
[15] M. J. Chen, L. G. Chen, and R. M. Weng, “Error concealment of lost
motion vectors with overlapped motion compensation,” IEEE Trans.
Circuits Syst. Video Technol., vol. 7, no. 3, pp. 560–563, Jun. 1997.
[16] L. Atzori, F. D. Natale, and C. Perra, “A spatio-temporal conceal-
ment technique using boundary matching algorithm and mesh-based
warping (BMA-MBW),” IEEE Trans. Multimedia, vol. 3, no. 3, pp.
326–328, Sep. 2001.
[17] C. S. Park, J. Ye, and S. U. Lee, “Lost motion vector recovery algo-
rithm,” in Proc. IEEE ISCAS, 1994, vol. 3, pp. 229–232.
[18] J. Feng, K. T. Lo, H. Mehrpour, and A. E. Karbowiak, “Cell loss con-
cealment method for MPEG video in ATM networks,” in Proc. IEEE
GLOBECOM, 1995, vol. 3, pp. 1925–1929.
[19] Y. Lee, Y. Altunbasak, and R. Mersereau, “Multiframe error conceal-
ment for MPEG-coded video over error-prone networks,” IEEE Trans.
Image Process., vol. 11, no. 11, pp. 1314–1321, Nov. 2002.
[20] J. Pyun, J. Lee, J. Jeong, J. Jeong, and S. Ko, “Robust error concealment
for visual communications in burst-packet-loss networks,” IEEE Trans.
Consum. Electron., vol. 49, no. 4, pp. 1013–1019, Nov. 2003.
[21] V. Chellappa, P. Cosman, and G. Voelker, “Error concealment for dual
frame video coding with uneven quality,” in Proc. IEEE Data Com-
press. Conf., 2005, pp. 319–328.
[22] E. M. S. Belfiore, M. Grangetto, and G. Olmo, “Spatio-temporal video
error concealment with perceptually optimized mode selection,” in
Proc. IEEE ICASSP, 2003, vol. 5, pp. 748–751.
[23] X. Ding and K. Roy, “A novel bitstream level joint channel error con-
cealment scheme for realtime video over wireless networks,” in Proc.
IEEE INFOCOM, 2004, pp. 2163–2173.
[24] T. H. Tsai, Y. X. Lee, and Y. F. Lin, “Video error concealment using
progressive interpolation and boundary matching algorithm,” in Proc.
Int. Symp. Circuits and Syst., 2004, vol. 5, pp. 443–446.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on December 1, 2008 at 04:35 from IEEE Xplore. Restrictions apply.
