Content uploaded by Shintaro Arai
Author content
All content in this area was uploaded by Shintaro Arai on Apr 16, 2015
Content may be subject to copyright.
IEEE Communications Magazine • July 2014
88 0163-6804/14/$25.00 © 2014 IEEE
Takaya Yamazato,
Hiraku Okada, and
Toshiaki Fujii are with
Nagoya University.
Isamu Takai is with Toy-
ota Central R&D Labs.,
Inc. and Shizuoka Uni-
versity.
Tomohiro Yendo is with
Nagaoka University of
Technology.
Shintaro Arai is with
Kagawa National College
of Technology.
Michinori and Tomohisa
Harada are with Toyota
Central R&D Labs., Inc.
Keita Yasutomi, Keiichiro
Kagawa, and Shoji
Kawahito are with
Shizuoka University.
INTRODUCTION
Visible light communication (VLC) is an optical
wireless communication technology that uses
low-power light-emitting diodes (LEDs) to not
only provide light, but also broadcast data [1–3].
LEDs are extremely energy-efficient and are
expected to become widespread in general light-
ing applications. Because LEDs are solid-state
lighting devices, they can be modulated at high
speed compared to other lighting sources. VLC
uses LEDs to send data by flashing light at
speeds that are undetectable to the human eye.
The widespread use of LEDs in traffic appli-
cations and the growing interest in intelligent
transport systems (ITS) presents a number of
opportunities for VLC applications. Data trans-
mission using LED traffic lights and LED brake
lights is a typical application [3, 4]. While previ-
ous studies have documented the effectiveness
of vehicle-to-infrastructure (V2I) and vehicle-to-
vehicle (V2V) communication using radio tech-
nology in terms of improving automotive safety
[5, 6], in the present article, we introduce the
concepts of V2I-VLC and V2V-VLC.
In V2I and V2V systems, VLC offers several
advantages. Since VLC links are visible, installa-
tion of roadside equipment is much easier. Addi-
tionally, previously installed facilities, such as
LED traffic lights or LED sign boards, can be
used. Furthermore, since the transmitters, or
LED light sources, are designed for lighting pur-
poses (and thus generally have high radiation
power), the signal-to-noise ratio (SNR) is high
for VLC, while eye safety is maintained for dual-
use lighting, defined here as VLC incorporated
with LED illumination. The visible light spec-
trum is not regulated globally, and its bandwidth
extends from 400 up to 790 THz. The large (390
THz) available bandwidth provides attractive
opportunities for ITS applications. Furthermore,
V2I and V2V communications using radio tech-
nology can be used simultaneously with VLC,
with each using a different spectrum.
VLC provides an additional feature if the
receiver incorporates an image sensor or a cam-
era. Specifically, by using image or video pro-
cessing to detect and recognize moving vehicles,
safety applications can be integrated. For exam-
ple, as methods of enhancing driving safety,
adaptive cruise control, collision warning, pedes-
trian detection, and providing range estimations
for nearby vehicles are potential candidates for
incorporation into VLC systems.
In this article, we introduce VLC application
to ITS while focusing on an image sensor as a
reception device. After starting with an overview
of an image sensor as a VLC reception device,
ABSTRACT
The present article introduces VLC for auto-
motive applications using an image sensor. In
particular, V2I-VLC and V2V-VLC are pre-
sented. While previous studies have document-
ed the effectiveness of V2I and V2V
communication using radio technology in terms
of improving automotive safety, in the present
article, we identify characteristics unique to
image-sensor-based VLC as compared to radio
wave technology. The two primary advantages
of a VLC system are its line-of-sight feature
and an image sensor that not only provides
VLC functions, but also the potential vehicle
safety applications made possible by image and
video processing. Herein, we present two ongo-
ing image-sensor-based V2I-VLC and V2V-
VLC projects. In the first, a transmitter using
an LED array (which is assumed to be an LED
traffic light) and a receiver using a high-frame-
rate CMOS image sensor camera is introduced
as a potential V2I-VLC system. For this system,
real-time transmission of the audio signal has
been confirmed through a field trial. In the sec-
ond project, we introduce a newly developed
CMOS image sensor capable of receiving high-
speed optical signals and demonstrate its effec-
tiveness through a V2V communication field
trial. In experiments, due to the high-speed sig-
nal reception capability of the camera receiver
using the developed image sensor, a data trans-
mission rate of 10 Mb/s has been achieved, and
image (320 ×240, color) reception has been
confirmed together with simultaneous reception
of various internal vehicle data, such as vehicle
ID and speed.
VISIBLE LIGHT COMMUNICATIONS
Takaya Yamazato, Isamu Takai, Hiraku Okada, Toshiaki Fujii, Tomohiro Yendo, Shintaro Arai,
Michinori Andoh, Tomohisa Harada, Keita Yasutomi, Keiichiro Kagawa, and Shoji Kawahito
Image-Sensor-Based
Visible Light Communication for
Automotive Applications
YAMAZATO_LAYOUT_Layout 7/2/14 3:25 PM Page 88
IEEE Communications Magazine • July 2014 89
we then introduce a safety support prototype
using V2I-VLC and describe the results of a
field trial. We then introduce a special comple-
mentary metal oxide semiconductor (CMOS)
image sensor designed for receiving high-speed
optical signals, and describe the result of a field
trial of our first-step V2V communication system
using the developed CMOS image sensor.
V2I VLC AND V2V VLC
To date, a variety of experimental and prototype
VLC systems have been demonstrated for trans-
portation applications. These include a 4.8 kb/s
visible light ID used as a mobile application
infrastructure and a lighthouse-based maritime
navigation VLC system that transmits signals
from buoys over ranges of up to 2 km, both
demonstrated by the Visible Light Communica-
tions Consortium (VLCC). VLCC also demon-
strated the effectiveness of visible light
identification (ID) systems in aircraft takeoff
and landing operations. In [7], Haruyama et al.
demonstrated successful 1 Gb/s free-space opti-
cal (FSO) communication transmissions to high-
speed trains from trackside stations. Of course,
needless to say, a significant amount of interest
is currently focused on VLC automotive applica-
tions.
Figure 1 shows an overview of a V2I-VLC
and V2V-VLC traffic safety network environ-
ment that includes an LED traffic light, LED
headlights, and LED brake lights.
V2I-VLC involves the wireless exchange of
critical safety and operational data between
moving vehicles and roadway infrastructure, and
is identical in concept to V2I using radio wave
technology, except that it uses light transmission
instead of radio wave technology. By creating a
networked environment between vehicles and
infrastructure, V2I-VLC facilitates safe driving
by adaptive traffic signal control, intersection
movement assistance, speed management, and
so on. This is also true for V2V-VLC, which
involves wireless exchanges of data between
moving vehicles traveling in the same area.
Potential applications of V2V systems include
emergency brake light warnings, forward colli-
sion warnings, and control loss warnings.
It is generally accepted that VLC links
depend on the existence of an uninterrupted line
of sight (LOS) path between the transmitter and
the receiver. In contrast, radio links are typically
susceptible to large fluctuations in received sig-
nal amplitude and phase. Unlike radio waves,
VLC does not suffer from multipath fading,
which significantly simplifies the design of VLC
links. Because VLC signals travel in a straight
line between a transmitter and a receiver, they
can easily be blocked by vehicles, walls, or other
opaque barriers. This signal confinement makes
it easy to limit transmissions to vehicles nearby.
Figure 1. Vehicle-to-infrastructure visible light communication (V2I-VLC) using an LED traffic light
and vehicle-to-vehicle visible light communications (V2V-VLC) using LED headlights or LED brake
lights.
LED traffic light
Camera
Camera
LED traffic signal to vehicle
(V2I-VLC)
LED brake lights to vehicle
(V2V-VLC)
A variety of experi-
mental and proto-
type VLC systems
have been
demonstrated for
transportation
applications. These
include a 4.8 kbps
visible light ID used
as a mobile applica-
tion infrastructure,
and a lighthouse-
based maritime navi-
gation VLC system
that transmits signals
from buoys over
ranges of up to
2 km.
YAMAZATO_LAYOUT_Layout 7/2/14 3:25 PM Page 89
IEEE Communications Magazine • July 2014
90
Thus, VLC networks can potentially achieve
remarkably high aggregate capacity and simpli-
fied design because transmissions outside com-
munication range need not be coordinated. In
other words, it is not necessary to consider
sources outside visual range.
It should be noted, however, that VLC has
several potential drawbacks. First, since visible
light cannot penetrate walls or buildings, VLC
coverage is restricted to small areas, and some
applications, such as blind spot warning, will
require installation of access points that must be
interconnected via a wired backbone. Further-
more, in addition to outright physical blocks,
thick fog or smoke can blur visible light links
and decrease system performance.
In short-range VLC applications, the SNR of
a direct detection receiver is proportional to the
square of the received optical power. Therefore,
VLC links can tolerate only a comparatively lim-
ited amount of signal path loss.
USE OF AN IMAGE SENSOR FOR VLC
ANIMAGE SENSOR AS A
RECEPTION DEVICE FOR VLC
CMOS image sensors have gained popularity in
recent years because of advances in multi-func-
tionalization, low manufacturing costs, and low
power consumption. The key element of a
CMOS image sensor is the photo diode (PD),
which is one component of a pixel. PDs are typi-
cally organized in an orthogonal grid. In opera-
tion, light passing through a lens strikes a PD,
where it is converted into a voltage signal and
then passed through an analog-to-digital con-
verter. The converter output is often referred to
as a luminance. Since a CMOS image sensor is
composed of a PD array, PD outputs, specifically
light intensity or luminance values, are arranged
in a square matrix to form a digital electronic
representation of the scene.
A CMOS image sensor can also be used as a
VLC reception device. Figure 2 shows examples
for a V2I-VLC system in which LEDs in a traffic
light transmit data, and a V2V-VLC system in
which the LED brake lights in a lead vehicle
transmit data to a following vehicle [4].
A particular advantage of CMOS image sen-
sor usage is, due to the massive number of pixels
available, its ability to spatially separate sources.
Here, the sources include both noise sources,
such as the Sun, streetlights, and other ambient
lights, and LED transmission sources.
If a single-element PD is used as a VLC
reception device, the VLC system cannot be
used in direct sunlight. This is because direct
sunlight is typically strong, and can often be
received at an average power that is much high-
er than that of the desired signal. Furthermore,
it is very difficult to reduce the enormous
amount of noise signals summing all background
lights in the field of view (FOV) to the optical
signal level, even if an optical band-pass filter
(OBPF) is used. Therefore, when a single-ele-
ment PD is used outdoors, directed linkage with
small optical beam divergence is necessary. Oth-
erwise, the PD cannot be used in direct sunlight.
In some cases, a receiver equipped with a tele-
photo lens that achieves an extremely narrow
FOV can be used [7, 8], but this type of receiver
is unsuitable for mobile usage because it limits
the angle of incidence and requires complex
mechanical tracking. In contrast, because of its
ability to separate sources spatially, a VLC
receiver utilizes only the pixels that it recognizes
as LED transmission sources and discards all
other pixels, including those detecting noise
sources.
The ability to spatially separate sources also
provides an additional feature to VLC, specifi-
cally, the ability to receive and process multiple
transmitting sources. As shown in Fig. 2, data
transmitted from an LED traffic light sign and
data transmitted from LED brake lights of a
vehicle ahead can be captured simultaneously.
Furthermore, if a source is composed of multiple
LEDs, as in the case of LED traffic lights or
brake lights, parallel data transmission can be
accomplished by modulating each LED indepen-
dently.
The output of the CMOS image sensor form-
Figure 2. Advantage of image-sensor-based VLC.
Photo diode
Noise(X3, Y3)
Data B(X2, Y2)
Data A
Image sensor
(X1, Y1)
Noise sources
(the sun, streetlights, etc.)
LED of vehicle
data B
LED of infrastructure
data A
CMOS image sensors
have gained popular-
ity in recent years
because of advances
in multi-functional-
ization, low manu-
facturing costs, and
low power con-
sumption. The key
element of a CMOS
image sensor is the
photo diode, which
is one component of
a pixel.
YAMAZATO_LAYOUT_Layout 7/2/14 3:25 PM Page 90
IEEE Communications Magazine • July 2014 91
ing a digital electronic representation of the
scene further provides unique opportunities that
cannot be realized by a single-element PD or
radio wave technology, specifically, the ability to
utilize a multitude of image or video processing
technologies, such as position estimation, object
detection, and moving target detection, simulta-
neously via the data reception capability of a
VLC.
For example, let us consider a situation where
a vehicle is equipped with a CMOS image sen-
sor. To begin, a VLC signal transmitted from a
vehicle nearby is captured along with its spatial
position (X, Y), or the actual row and column
position of a pixel. This means that the VLC sig-
nal can be represented not only by a time
domain signal, but also by the direction of the
incoming vector from the transmitter to the
receiver [9]. Consequently, requisite position
data, which can be obtained by GPS or some
other position estimation system that does not
need to be transmitted, are already available.
HIGH-SPEED OPTICAL SIGNAL RECEPTION
Most CMOS image sensors are designed in con-
sideration of the characteristics of the human
eye. A typical example is the frame rate of com-
mercially available CMOS image sensors, which
are generally limited to 30 fps or several multi-
ples of 30 fps. Despite this, the demand for real-
time high-frame-rate (HFR) image processing
technology is increasing in various applications,
including robotics, factory automation, multime-
dia, and biomedical fields [10]. Needless to say,
VLC for automotive applications requires HFR
image-processing technology.
Assuming a CMOS image sensor with a frame
rate of 30 fps, the transmission rate, or equiva-
lently the “blink” rate of an LED, must be less
than or equal to 15 Hz to satisfy the Nyquist fre-
quency. However, this may be too slow, and
humans may be able to recognize (and be dis-
tracted by) such blinking. Additionally, if a vehi-
cle is moving at a speed of 36 km/h, or 10 m/s,
the vehicle will receive a 15 b/s data signal every
10 m. This rate exceeds the permissible level.
Furthermore, this rate makes the development
of image processing technology for detecting and
recognizing moving vehicles, or a transmitter
located on the road, far more difficult because
the vehicle moves 0.67 m during a single frame.
In contrast, if the frame rate is boosted to
1000 fps, the vehicle moves just 0.01 m per
frame. This not only simplifies the image pro-
cessing required for detecting and recognizing
moving vehicles, it also increases the data rate to
500 b/s. Accordingly, HFR CMOS image sensors
are mandatory for VLC systems intended for use
in automotive applications.
An alternate approach is to integrate a receiv-
er function into a conventional image sensor.
Because of spectacular advances in CMOS pro-
cess technology, the development of such a
multi-purpose CMOS image sensor is straight-
forward. Furthermore, such a sensor not only
maximizes data reception performance, but it
also provides electric images suitable for image
and video processing.
Next, we describe a newly developed CMOS
image sensor in which communication pixels that
have been specialized for receiving high-speed
optical signals are integrated with ordinary
image pixels into a pixel array.
VEHICLE-TO-INFRASTRUCTURE
VISIBLE LIGHT COMMUNICATIONS
In this subsection, we introduce a V2I-VLC sys-
tem using an LED array transmitter, which is
assumed to be an LED traffic light, and an in-
vehicle receiver equipped with an HFR CMOS
image sensor camera, or high-speed camera [11].
SYSTEM OVERVIEW
Figure 3 shows a block diagram of the system.
The transmitter consists of an encoder, an invert-
ed LED pattern insertion unit, a pulse-width
modulator (PWM), and 1024 LEDs arranged in
a 32 ×32 square matrix. The LEDs are the same
as those used in LED traffic lights in Japan.
Input data is first fed into the encoder that
processes R= 1/2 turbo coding. The tracking
LED pattern insertion unit then generates an
inverted signal that is used for LED array track-
ing. Next, in order to assign different luminance
levels to the LEDs, the signal is fed to a PWM.
Finally, the PWM signal is converted into a 2D
signal, and each LED transmits data in parallel
by modulating its luminance individually. In
other words, we transmit data as a 2D LED pat-
tern. The LED blink rate is once per 2 ms.
The packet format is shown below the trans-
mitter. A Baker code sequence of length 11 is
selected for LED array detection. The data part
includes the data signal and the inverted signal
used for tracking. The receiver consists of a
high-speed camera, a header image processing
unit, a data image processing unit, and a
decoder.
The captured images are fed to the header
image processing unit, which identifies the LED
array from the captured images. Thanks to the
Baker code, we achieve robust time synchroniza-
tion as well as robust LED array detection.
From the header part, images are captured at a
frame rate of 1000 fps (1 ms intervals), while the
LED array blinks at 2 ms intervals. This means
that most of the background, notably everything
except the LED array, remains static. Thus,
using interframe differential decoding, the LED
array is stands out in the processed image and
can easily be identified. At the LED array detec-
tion unit, we further perform a time-domain cor-
relation of a Baker sequence in order to
eliminate misdetections.
After the header image processing unit, the
signal is fed to the data image processing unit,
which consists of an LED array tracking unit, an
LED position estimation unit, and a luminance
extraction unit. Tracking is performed by simple
template matching. However, since the captured
LED patterns change one by one for the data
portion of a packet, accurate tracking is difficult.
Accordingly, we insert an inverted pattern at the
transmitter in order to make an all-LEDs-on
pattern at the receiver [10].
After LED array tracking, LED position esti-
mation is performed to output the position of
each LED, based on the pixel row and column
Most CMOS image
sensors are designed
in consideration of
the characteristics of
the human eye. A
typical example is
the frame rate of
commercially avail-
able CMOS image
sensors, which are
generally limited to
30 fps or several
multiples of 30 fps.
YAMAZATO_LAYOUT_Layout 7/2/14 3:25 PM Page 91
IEEE Communications Magazine • July 2014
92
values, and its luminance value. This is only pos-
sible if accurate LED array detection is achieved
because it is necessary to output both the shape
of the LED array and the LED array tracking.
We then calculate the luminance by normaliza-
tion based on the mean and variance of the
extracted luminance. The effect of normalization
is essential for accurate signal demodulation.
Finally, the output of the data image process-
ing unit is fed to a decoder, which performs
turbo decoding and outputs retrieved data.
FIELD TRIALS
This subsection provides an overview of the field
trials conducted using our V2I-VLC system.
These field trials have been performed in
order to confirm the effectiveness of the pro-
posed system under actual driving conditions. In
our experimental setup, we place an LED array
on horizontal ground and mounted the high-
speed camera on the dashboard of the vehicle.
During the experiments, the vehicle is driven
directly toward the LED array at a speed of 30
km/h, as shown in Fig. 4. The communication
distance in these field trials ranges from 70 m to
30 m.
As mentioned previously, the transmitter
consists of 1024 LEDs arranged in a 32 ×32
square matrix. The LED spacing is 15 mm, and
its half value angle is 26˚. In order to compen-
sate for the vibration of the car, we represent
one data bit using four LEDs (a 2 ×2 LED
array). Each LED blinks at 500 Hz, whereas the
PWM is processed at 4 kHz. We use R= 1/2
turbo code for error correction and inverted
LED patterns for tracking. Accordingly, the
overall data rate is 32 kb/s (= 500 b/s ×256 ×
1/2 ×1/2). The input data is audio data and is
assumed to be driving safety information trans-
mitted from LED traffic lights.
For the receiver, we use an in-vehicle high-
speed camera (HSC) (Photoron FASTCAM
1024PCI 100k) with a frame rate of 1000 fps and
a resolution of 512 ×1024 pixels connected to a
PC. The focal length of the lens is 35 mm. Gen-
erally speaking, the light sensitivity of high-speed
image sensors is set high to provide rapid expo-
sure time, which also means we can set a rela-
tively small lens diaphragm. For example, ISO
sensitivity of the HSC is set at 10,000, and the
lens diaphragm is set to 11. Additionally, since
autofocusing is difficult when a vehicle is mov-
ing, the focus is set to infinity.
The header image processing, data image
processing, and decoding are performed in real
time by the PC. In the upper part of the figure,
we present a display showing the PC output. As
shown, we also record and display a grayscale
video obtained using the high-speed camera as a
drive recorder, which simultaneously records the
view in front of the vehicle and the data trans-
mitted from the LED array. In the right window
of the display, we also show the results of LED
array detection and tracking. We confirm robust
detection and tracking of the LED array with
respect to the camera vibration along with a lack
of error in LED array detection and tracking.
Next, we have confirmed clear audio signal
reception for distances of up to 45 m and have
achieved error-free performance.
We also conduct an experiment on the simul-
taneous transmission of text information. In this
case, the data rate is 2 kb/s, and error-free per-
formance has been achieved from 110 m to 20
m, which is deemed to be a suitable range for
intersection safety applications.
Figure 3. A V2I-VLC system.
Header LED array
High-speed camera
1 packet Time
Packet format
Data
Baker sequence
(1111100110101)
Input
data Encoder
Output
data Decoder
Receiver
Data
image
processing
Header
image
processing
High-speed
camera
LED position
estimation
Luminance
extraction
LED array
tracking
LED array
detection
Search for
LED array
Inverted
LED pattern
insertion
Transmitter
Pulse
width
modulation
32 32
LEDs
Lighting pattern
(data)
Optical
channel
After LED array
tracking, LED posi-
tion estimation is
performed to output
the position of each
LED, based on the
pixel row and col-
umn values, and its
luminance value. This
is only possible if
accurate LED array
detection is achieved
because it is neces-
sary to output both
the shape of the LED
array and the LED
array tracking.
YAMAZATO_LAYOUT_Layout 7/2/14 3:25 PM Page 92
IEEE Communications Magazine • July 2014 93
VEHICLE-TO-VEHICLE VISIBLE LIGHT
COMMUNICATION
In this subsection, we provide an overview of our
first-step V2V communication system, which con-
sists of LED transmitters and a camera receiver
equipped with a newly developed CMOS image
sensor. Two requirements were stipulated when
designing this system. One is high-speed optical
signal reception (i.e., over 10 Mb/s per pixel) to
allow color image (video) data and various vehi-
cle internal data to be obtained simultaneously.
The other is accurate and real-time LED detec-
tion using images in order to communicate with
constantly moving LED targets.
In response to these requirements, a new
CMOS image sensor, an optical communication
image sensor (OCI), has been developed [12]. In
the OCI, two new technologies, specifically a
communication pixel (CPx) and a 1-bit flag
image output function, have been employed to
ensure high-speed optical signal reception and
real-time LED detection.
Currently, to confirm the maximum reception
performance and potential of the OCI, we are
focusing on the development and evaluation of
the receiver system. Accordingly, in our current
experiments, 870-nm near-infrared LEDs capa-
ble of being modulated at high speed (over 50
MHz) are used.
OVERVIEW OF THE OCI CHIP
Figure 5 shows a block diagram of the OCI and
an overview of its operations, including the
external processing circuits.
The CPx for a high-speed optical signal recep-
tion and image pixel (IPx) for an image capture
are set alternately in the OCI pixel array. The
IPx is a commonly used four transistor type. The
CPx is designed using the pinned PD technology
and is specialized for communication (i.e., opti-
cal signal reception). More specifically, to
achieve prompt response to optical intensity
variations, the CPx capacitance is significantly
reduced. Hence, its reception performance is
more than 10 times higher than that of the pixels
of conventional specialized image sensors
designed for LED optical signal reception
[13–16].
However, the CPx cannot capture image sig-
nals because it loses the function that accumu-
lates charges generated by photoelectric
conversion when the capacitance is reduced.
Therefore, a hybrid pixel structure is incorporat-
ed into the OCI, in which the image signal is
captured by the IPx array and the optical signal
is received via the CPx array. The light wave-
length range that the IPx and CPx can sense is
visible light to near-infrared light.
The 1-bit flag image for LED detection is
output from the readout circuits with a gray
image at up to 60 fps (in a period of up to 16.6
ms). The gray image that is output via correlated
double sampling (CDS) circuits is conventional.
On the other hand, the exposure time of the flag
image is reduced to approximately 1/100 that of
a gray image. In addition, the flag image is bina-
rized by a comparator circuit to completely elim-
inate low-light-intensity objects from the flag
images. Conversely, only high-intensity objects,
such as LEDs, streetlights, and the Sun, will reg-
Figure 4. Experimental equipment and results.
Vehicle speed: 30 km/h
LED array transmitting
audio and text data
Audio data
Text data
32 kb/s
2 kb/s
Frame rate
Focal of lens
Resolution
1000 fps
35 mm
1024 512 pixels
LED array
tracking
Error free
Distance (m)
High-speed camera
3530
10–2
10–3
BER of audio data
10–1
100
40 45 50 55 60 65
Grayscale video
obtained by the
high-speed camera
To confirm the maxi-
mum reception per-
formance and
potential of the OCI,
we are focusing on
the development
and evaluation of
the receiver system.
Accordingly, in our
current experiments,
870-nm near-
infrared LEDs capa-
ble of being
modulated at high
speed (over 50 MHz)
are used.
YAMAZATO_LAYOUT_Layout 7/2/14 3:25 PM Page 93
IEEE Communications Magazine • July 2014
94
ister in the flag image as “1”s. When this flag
image is used for LED detection, the calculation
time and misdetection are greatly reduced in
comparison to methods using gray images,
because most unnecessary objects have been
removed. Incidentally, in this system, the gray
image is only used for display purposes. Howev-
er, in the near future, we intend to use it for
image-processing-based safety applications, such
as lane and pedestrian detection.
The OCI operation is initiated by the output
of the flag image. After the flag image is deliv-
ered to an image processor on an external digital
signal processing (DSP) unit, the LED regions
are detected, and the central coordinates of the
LED area are obtained by basic image process-
ing techniques. Next, the obtained (x, y)-coordi-
nates are set to the control circuits of the OCI
address generator. Then the CPx corresponding
to the (x, y)-coordinates is selected by x- and y-
address generators, and the selected CPx is acti-
vated. The signal received by the selected CPx is
output through a readout amplifier. Finally, the
output signal is converted to a digital signal and
is equalized on an external unit. This entire
operation repeats in a period of up to 16.6 ms,
which allows this system to receive high-speed
optical signals while the LED is detected in real
time.
The eye diagram in the upper right of the fig-
ure shows the reception result of an optical sig-
nal coded by the 10 Mb/s Manchester code.
A photograph and specifications of the OCI
chip are shown in the lower part of the figure.
This 7.5 ×8.0 mm2OCI chip has been fabricat-
ed using 0.18-mm CMOS image sensor technolo-
Figure 5. Overview of an optical communication image sensor.
X-address generator
y
Y-address generator
Technology 0.18 µm CMOS process
Pixel size 7.5 µm × 7.5 µm
Frame rate Up to 60 fps (gray and flag)
Receivable
wavelength
Visible to near-infrared band
(approx. 400–900 nm)
Chip size 7.5 mm × 8.0 mm
Pixel array
size
Total of 642 × 480
- CPx: 321 × 480
-IPx: 321 × 480
1-bit
LED
Chip specifications
OCI chip photograph
Gray image
Flag image
LED detection
(x, y)
Image processor
100 ns
Equalizer
External DSP unit
A/D
Comparator
Image
outputs
at 60 fps
Readout
circuits
H. scanner
V. scanner
CDS and amp.
Image pixel (IPx)
Communication pixel (CPx)
Received signal outputs
Circuits for image signals
Circuits for communication signals
Readout amplifier
X
Pixel array
Selecting CPx
using (x,y)
x
y
The OCI operation is
initiated by the out-
put of the flag
image. After the flag
image is delivered to
an image processor
on an external digital
signal processing
(DSP) unit, the LED
regions are detected
and the central coor-
dinates of the LED
area are obtained by
basic image process-
ing techniques.
YAMAZATO_LAYOUT_Layout 7/2/14 3:26 PM Page 94
IEEE Communications Magazine • July 2014 95
gy, and has a 321 ×240 IPx array and a 321 ×
240 CPx array set in a 642 ×480 pixel array. The
IPx and CPx pixel size is 7.5 ×7.5 mm2.
FIELD TRIAL
Figure 6 shows the developed V2V communica-
tion system and experimental results. As shown
in the upper part of the figure, a lead vehicle
sends optical signals to a following vehicle.
The lead vehicle has two LED transmitters, a
front-view camera for taking front-view images,
and a control unit. The two LED transmitters
are attached to the left and right sides of the
rear window and send 4 W optical signals at an
angle of 40°.The control unit collects the front-
view image and various vehicle internal data
such as vehicle ID and speed, assembles packets,
encodes send data, and drives the LED transmit-
ters.
The following vehicle has a camera receiver
using the OCI and a control unit that includes a
DSP unit. The FOV of the camera receiver is 22
(H) ×16 (V) degrees. The control unit collects
its own vehicle internal data, detects LEDs using
the flag image, sets the (x, y)-coordinates to the
OCI, equalizes the received signals, disassembles
packets, and displays the obtained data on a PC
monitor.
Sent data is encoded by 10 Mb/s Manchester
code, and errors are corrected by Bose-Chaud-
huri-Hocquenghem (BCH) code. The packet
size is 2464 bits, and each packet consists of a
32-bit preamble, a 32-bit unique word, a 2392-bit
payload, and an 8-bit postamble. Currently, a
non-standard format of our own design is used.
At the start of this system operation, the
front-view camera on the lead vehicle captures
the front-view image (quarter video graphics
array, QVGA: 320 ×240, color) data. This image
is collected by the control unit along with vari-
ous vehicle internal data. Then the front-view
image is compressed. Subsequently, the collected
vehicle internal data and the image data are
assembled into the payload of a packet. The
generated packets are encoded, and the encoded
data is sent to the following vehicle by the LED
transmitters.
At the same time, using the procedures
described in the preceding subsection, the DPS
unit on the following vehicle searches LEDs
from flag images in real time, while the camera
receiver receives optical signals using the select-
ed CPx. The received signal is equalized on the
DPS unit, after which the DSP catches the
preamble of a packet and performs synchroniza-
tion between the LED transmitter and the cam-
era receiver. Next, the content data in the
payload is retrieved and decoded. Finally, the
obtained content data is organized and displayed
on the PC monitor.
The experimental result of the front-view
image reception is shown at the bottom of the
figure. This experiment is conducted under out-
door lighting conditions and under constantly
varying intervehicular distances and vehicle
speeds limited to less than 20 m and 25 km/h,
respectively. As shown in the flag image, almost
all unnecessary objects have been eliminated,
and accurate and real-time LED detection has
been achieved despite the difficult outdoor envi-
ronments. Additionally, as shown in the received
front-view images, a QVGA-sized color image is
successfully received at approximately 10 fps
together with various vehicle internal data even
when each vehicle is constantly moving.
We believe that this experimental result
makes a significant contribution toward the real-
ization of an automotive VLC system, and that it
Figure 6. Experimental equipment and results.
Gray image Flag image
Received front-view images
t2
t1
t4
t3
Results of
LED detection
Results of front-view image reception
Encoding
method
Manchester code
with BCH code
Data
contents
Vehicle internal data:
ID, speed, etc.
Front-view images:
- QVGA, color 10, fps
Packet size 2,464 bits
Data rate 10 Mb/s
System specifications
Lead vehicle
Equalizer
Following vehicle
LED transmitters
(2 units) Front-view camera
Control unit
with DSP unit Control unit
Camera receiver
that equips OCI
Decoder
OVGA, color
10 fps
Front-view image
of lead vehicle
for transmission
to backward
YAMAZATO_LAYOUT_Layout 7/2/14 3:26 PM Page 95
IEEE Communications Magazine • July 2014
96
supports advances in both standardization and
commercialization. Moreover, since the camera
receiver is already capable of achieving high-
speed image-sensor-based VLC, we strongly
anticipate the launch of high-speed visible light
LEDs.
CONCLUSIONS
In the present article, we have introduced an
image sensor based VLC system that is intended
for automotive applications. Furthermore, we
have demonstrated that the use of an HFR
CMOS image sensor or a specialized CMOS
image sensor is essential. Such sensors can pro-
vide not only VLC functions, but also useful
safety applications using image or video process-
ing technologies, including moving object detec-
tion, tracking, and ranging.
In our study, a transmitter using an LED
array, which is assumed to be an LED traffic
light, and a receiver using a HFR CMOS image
sensor camera have been introduced for V2I-
VLC system use. Real-time audio signal trans-
mission with a data rate of 32 kb/s has been
confirmed through field trials. We have also pre-
sented the OCI for realizing the high-speed
VLC system and demonstrated its effectiveness
through the V2V system field trial, during which
a data rate of 10 Mb/s and correct and real-time
LED detection have been confirmed. As a result
of this superior reception capability, we have
demonstrated QVGA-sized color image trans-
mission at approximately 10 fps.
However, it should be noted that the systems
we have introduced are not based on VLC stan-
dards IEEE 802.15.7 (released in September
2011) or JEITA CP-1222 (released in 2013).
Although potential applications of those stan-
dards include ITS and information broadcasting,
no specifications released to date cover the
details necessary for V2I or V2V system applica-
tion. We feel, however, that a slight modification
of current standards may satisfy the V2I and
V2V specifications because a standard for V2V
and V2I communication, known as dedicated
short-range communication (DSRC), is already
in place as an extension of the IEEE 802.11a
wireless local area network (WLAN) standard.
Since backbone networks and roadside equip-
ment network modules can be shared between
DSRC and VLC, it is readily apparent that the
V2I-VLC and V2V-VLC systems shown in the
present article can easily adopt the current VLC
standards.
Furthermore, as we have demonstrated, VLC
offers several advantages for V2I and V2V sys-
tem use, and VLC can accommodate various
application demands. One key feature is that
VLC networks are limited to the LOS region,
and vehicles or other opaque barriers can easily
block VLC signals, which is not the case for
radio waves. Therefore, as one option, it is
important to explore the combined use of VLC
and radio wave technologies, because depending
on the applications in use, VLC technology may
not be able to provide sufficient link quality. If
both link types are available, link quality can be
improved, thus yielding a diversity effect. This
would be especially beneficial in safety applica-
tions. Accordingly, further research into the
combined use of VLC and radio wave technolo-
gy will be conducted in an effort to improve the
reliability of V2I and V2V links.
ACKNOWLEDGMENT
This work was partly supported by the Knowl-
edge Cluster Initiative of the Ministry of Educa-
tion, Culture, Sports, Science and Technology
(MEXT) of Japan.
REFERENCES
[1] T. Komine and M. Nakagawa: “Fundamental Analysis
for Visible-Light Communication System Using LED
Lightings, IEEE Trans. Consumer Electronics, vol. 50, no.
1, 2004, pp. 100–07.
[2] D. C. O’Brien et al., “Visible Light Communication: Chal-
lenges and Possibilities,” IEEE 19th Int’l. Symp. PIMRC,
Sept. 2008.
[3] M. Nakagawa, “Visible Light Communications,” Proc.
CLEO ’07, paper CTuI5, 2007.
[4] H. B. C. Wook, S. Haruyama, and M. Nakagawa, “Visi-
ble Light Communication with LED Traffic Lights Using
2-Dimensional Image Sensor,” IEICE Trans. Fundamen-
tals, vol. E89-A, no. 3, Mar. 2006, pp. 654–50.
[5] R. Kandarpa et al., “Final Report: Vehicle Infrastructure
Integration (VII) Proof of Concept (POC) – Executive
Summary,” Research and Innovative Technology Admin.
(RITA), U.S. Dept. of Transportation, Feb. 2009.
[6] M. L. Sichitiu and M. Kihl, “Inter-Vehicle Communica-
tion Systems: A Survey,” IEEE Commun. Surveys & Tuto-
rials, 2nd qtr., Oct. 2008.
[7] H. Urabe et al., “High Data Rate Ground-to-Train Free-
Space Optical Communication System,” Optical Engi-
neering, Special Section on Free-Space Laser
Communications, vol. 51, no. 3, Mar. 2012.
[8] S. Okada et al., “On-Vehicle Receiver for Distant Visible
Light Road-to-Vehicle Communication,” 2009 IEEE Intel-
ligent Vehicles Symp., June 2009, pp. 1033–38.
[9] T. Yamazato and S. Haruyama, “Image-Sensor-Based
Visible Light Communication and Its Application to
Pose, Position, and Range Estimations,” to appear,
IEICE Trans. Commun., Sept. 2014.
[10] I. Ishii et al., “High-Frame-Rate Optical Flow System,”
IEEE Trans. Circuits and Systems for Video Tech., vol.
22, no. 1, Jan. 2012.
[11] T. Nagura et al., “Tracking an LED Array Transmitter
for Visible Light Communications in the Driving Situa-
tion,” 2010 7th Int’l. Symp. Wireless Commun. Sys-
tems, Sept. 2010, pp. 765–69.
[12] I. Takai et al., “LED and CMOS Image Sensor Based
Optical Wireless Communication System for Automotive
Applications,” IEEE Photonics J., vol. 5, no. 5, Oct.
2013, p. 6,801,418.
[13] Y. Oike, M. Ikeda, and K. Asada, “A Smart Image Sen-
sor with High-Speed Feeble ID-Beacon Detection for
Augmented Reality System,” Proc. 29th Euro. Solid-
State Circuits Conf., Sept. 2003, pp. 125–28.
[14] S. Yoshimura et al., “A 48kframe/s CMOS Image Sen-
sor for Real-Time 3-D Sensing and Motion Detection,”
IEEE Int’l. Solid-State Circuits Conf. Dig. Tech. Papers,
San Francisco, CA, Feb. 2001, pp. 94–95.
[15] D. Yamanaka, S. Haruyama, and M. Nakagawa, “The
Design of High-Speed Image Sensor Ship for Receiving
the Data of Visible-Light ID System” (in Japanese), IEICE
tech. rep., vol. 107, no. 300, 2007 pp. 97–102.
[16] M. S. Z. Sarker et al., “A CMOS Imager and 2-D light
Pulse Receiver Array for Spatial Optical Communica-
tion,” Proc. IEEE Asian Solid-State Circuits Conf., Nov.
2009, pp. 113–16.
BIOGRAPHIES
TAKAYA YAMAZATO [M] is a professor at the Institute of Liberal
Arts and Sciences, Nagoya University, Japan. He received his
Ph.D. degree from the Department of Electrical Engineering,
Keio University, Yokohama, Japan, in 1993. From 1993 to
1998, he was an assistant professor in the Department of
Information Electronics, Nagoya University. From 1997 to
1998, he was a visiting researcher of the Research Group for
RF Communications, Department of Electrical Engineering
and Information Technology, University of Kaiserslautern,
Germany. In 1998, he gave a 1/2 day tutorial entitled “Intro-
We believe that this
experimental result
makes a significant
contribution toward
the realization of an
automotive VLC sys-
tem, and that it sup-
ports advances in
both standardization
and commercializa-
tion. Moreover, since
the camera receiver
is already capable of
achieving high-speed
image sensor based
VLC, we strongly
anticipate the launch
of high-speed visible
light LEDs.
YAMAZATO_LAYOUT_Layout 7/2/14 3:26 PM Page 96
IEEE Communications Magazine • July 2014 97
duction to CDMA ALOHA” at IEEE GLOBECOM held in Syd-
ney Australia. Since then, he has been serving as a Technical
Program Committee (TPC) member of IEEE GLOBECOM and
ICC. In 2006, he received the IEEE Communication Society
2006 Best Tutorial Paper Award. He served as a Co-Chair of
the Wireless Communication Symposium of ICC ’09 and the
Selected Areas in Communication Symposium of ICC’11.
From 2008 to 2010, he served as Chair of the Satellite Space
and Communication Technical Committee. In 2011, he gave
a half-day tutorial entitled “Visible Light Communication” at
ICC’01 held in Kyoto, Japan. He was an Editor-in-Chief of the
Japanese Section of IEICE Transaction on Communications
from 2009 to 2011. His research interests include visible
light communication, satellite and mobile communication
systems, and ITS.
ISAMU TAKAI [M] received his B.S. and M.S. degrees in infor-
mation science from Gifu University, Japan, in 2002 and
2004, respectively. Since 2004, he has worked in Toyota
Central R&D Labs., Inc. His research interests include CMOS
image sensors, optical sensing devices, and optical commu-
nication systems for automotive applications. He is a mem-
ber of the Institute of Image Information and Television
Engineers of Japan.
HIRAKU OKADA [M] received his B.S., M.S., and Ph.D. degrees
in information electronics engineering from Nagoya Univer-
sity, Japan, in 1995, 1997, and 1999, respectively. From
1997 to 2000, he was a research fellow of the Japan Soci-
ety for the Promotion of Science. He was an assistant pro-
fessor at Nagoya University from 2000 to 2006, an
associate professor at Niigata University from 2006 to
2009, and an associate professor at Saitama University
from 2009 to 2011. Since 2011, he has been an associate
professor of the EcoTopia Science Institute at Nagoya Uni-
versity. His current research interests include the packet
radio communications, wireless multihop networks, inter-
vehicle communications, and CDMA technologies. He
received the Inose Science Award in 1996 and the IEICE
Young Engineer Award in 1998. He is a member of ACM
and the Institute of Electronics, Information and Communi-
cation Engineers (IEICE).
TOSHIAKI FUJII [M] received his Dr.E. degree in electrical engi-
neering from the University of Tokyo in 1995. From 1995
to 2007, he was with the Graduate School of Engineering,
Nagoya University. From 2008 to 2010, he was with the
Graduate School of Science and Engineering, Tokyo Insti-
tute of Technology. He is currently a professor in the Grad-
uate School of Engineering, Nagoya University. He was a
sub-leader of the Advanced 3D Tele-Vision Project estab-
lished by the Telecommunications Advancement Organiza-
tion of Japan from 1998 to 2002. Now he serves as a
Vice-President of the Image Engineering Technical Group of
The Institute of Electronics, Information and Communica-
tion Engineers (IEICE), Japan. He received an Academic
Encouragement Award from the IEICE in 1996 and Best
Paper Award from 3-D Image Conference several times
during 2001 and 2009. He is known for his work on 3-D
image processing and 3-D visual communications, based
on Ray-based representation. His current research interests
include multi-dimensional signal processing, large-scale
multi-camera systems, multi-view video coding and trans-
mission, free-viewpoint television, and their applications
for Intelligent Transport Systems. He is a member of the
IEICE and the Institute of Image Information and Television
Engineers (ITE) of Japan. He serves as an Associate Editor
of IEEE TCSVT.
TOMOHIRO YENDO [M] received his B.Eng., M.Eng., and Ph.D.
degrees from Tokyo Institute of Technology, Japan, in
1996, 1998 and 2001, respectively. He was a researcher at
the Telecommunications Advancement Organization (TAO)
of Japan from 1998 to 2002 and a research fellow at
Japan Science and Technology Agency (JST) from 2002 to
2004. From 2004 to 2011, he was an assistant professor at
Nagoya University. Since 2011, he has been an associate
professor at Nagaoka University of Technology. His current
research interests include visible light communication, and
3D image display and capturing.
SHINTARO ARAI [M] received the B.E., M.E., and D.E. degrees
from Tokushima University, Japan, in 2004, 2006, and
2009, respectively. From January 2007 to December 2008,
he was a special research student at Nagoya University.
From April 2009 to March 2011, he worked as a postdoc-
toral fellow of ITS Laboratory, Aichi University of Technolo-
gy, Japan. Since April 2011, he has been a research
associate at Kagawa National College of Technology,
Japan. His research interests include visible light communi-
cation systems, chaos-based communication systems, and
stochastic resonance phenomena. He is a member of the
IEICE.
MICHINORI ANDO received his B.S. and M.S. degrees in elec-
tronics engineering from Nagoya Institute of Technology in
1983 and 1985, respectively. Since 1985, he has worked at
Toyota Central R&D Labs, Inc. His research interests include
image sensing devices and semiconductor device physics.
TOMOHISA HARADA received his B.S. and M.S. degrees in elec-
tronics engineering from Nagoya Institute of Technology in
1982 and 1984, respectively. Since 1988, he has worked at
Toyota Central R&D Labs, Inc. His research interests include
wireless communication systems and digital control sys-
tems for automotive power converters.
KEITA YASUTOMI [M] received his B.E. and M.E. degrees in
electrical and electronic engineering and his Ph.D. degree
from Shizuoka University, Hamamatsu, Japan, in 2006,
2008, and 2011, respectively. He is currently an assistant
professor with the Research Institute of Electronics, Shizuo-
ka University. His research interests include time-resolved
CMOS image sensors and low-noise imagers. He is a mem-
ber of the IEEE, IEICE, and Institute of Image Information
and Television Engineers of Japan.
KEIICHIRO KAGAWA [M] received his B.E., M.E., and Ph.D.
degrees from Osaka University in 1996, 1998, and 2001,
respectively. From 2001 to 2007, he was an assistant pro-
fessor at the Nara Institute of Science and Technology,
Japan. From 2007 to 2010, he was an associate professor
at Osaka University. Since 2011 he has been an associate
professor at Shizuoka University. His research interests
include CMOS image sensors, high-performance or com-
pact multi-functional multi-aperture imaging systems, and
their applications.
SHOJI KAWAHITO [F] received his Ph. D. degree from Tohoku
University, Sendai, Japan, in 1988. In 1988, he was a
research associate at the same university. From 1989 to
1999, he was with Toyohashi University of Technology,
Japan. From 1996 to 1997, he was a visiting professor with
the Swiss Federal Institute of Technology (ETH) Zurich,
Switzerland. Since 1999, he has been a professor with the
Research Institute of Electronics, Shizuoka University. He
has published more than 250 papers in refereed journals
and conference proceedings. His research interests include
CMOS imaging devices, sensor interface circuits, and mixed
analog/digital circuit designs. He has been a recipient of
many awards, including the Outstanding Paper Award at
the 1987 IEEE International Symposium on Multiple-Valued
Logic, the Special Feature Award in the Large-Scale Integra-
tion Design Contest at the 1998 Asia and South Pacific
Design Automation Conference, the Beatrice Winner Award
for Editorial Excellence at the 2005 IEEE International Solid-
State Circuits Conference, the IEICE Electronics Society
Award in 2010, the 24th Takayanagi Memorial Award in
2010, and the Walter Kosonocky Award in 2013. He is a
Fellow of the ITE, and a member of IEICE and SPIE.
If both link types are
available, link quality
can be improved,
thus yielding a diver-
sity effect. This
would be especially
beneficial in safety
applications. Accord-
ingly, further
research into the
combined use of
VLC and radio wave
technology will be
conducted in an
effort to improve the
reliability of V2I and
V2V links.
YAMAZATO_LAYOUT_Layout 7/2/14 3:26 PM Page 97
