Recent advances in high dynamic range imaging technology

Abstract

Recently, visual representations using high dynamic range (HDR) images become increasingly popular, with advancement of technologies for increasing the dynamic range of image. HDR image is expected to be used in wide-ranging applications such as digital cinema, digital photography and next generation broadcast, because of its high quality and its powerful expression ability. HDR imaging technologies will spread its sphere of influence in imaging industry. In this paper, we review the state-of-the-art studies and the trends of the HDR imaging, in terms of the following three points: (1) HDR imaging sensor and HDR image generation techniques as image acquisition technologies, (2) encode method of HDR images for efficient transmission and storage, (3) human visual system issues associated with reproduction of HDR image.

Full text

RECENT ADVANCES IN HIGH DYNAMIC RANGE IMAGING TECHNOLOGY
Yukihiro BANDOH†, Guoping QIU‡, Masahiro OKUDA††, Scott DALY‡‡, Til Aach††† and Oscar C. AU‡‡‡
†NTT Cyber Space Laboratories, NTT Corporation
‡The University of Nottingham, School of Computer Science
†† University of Kitakyushu
‡‡ Sharp Laboratories of America
††† Institute of Imaging and Computer Vision, RWTH Aachen University
‡‡‡ Hong Kong University of Science and Technology
ABSTRACT
Recently, visual representations using high dynamic range
(HDR) images become increasingly popular, with advance-
ment of technologies for increasing the dynamic range of
image. HDR image is expected to be used in wide-ranging
applications such as digital cinema, digital photography and
next generation broadcast, because of its high quality and its
powerful expression ability. HDR imaging technologies will
spread its sphere of influence in imaging industry. In this
paper, we review the state-of-the-art studies and the trends
of the HDR imaging, in terms of the following three points:
(1) HDR imaging sensor and HDR image generation tech-
niques as image acquisition technologies, (2) encode method
of HDR images for efficient transmission and storage, (3)
human visual system issues associated with reproduction of
HDR image.
Index Terms—high dynamic range image, high bit-depth
image, multiple exposure principle, human visual system,
tone mapping
1. INTRODUCTION
Realistic representations using high quality images are be-
coming increasingly popular. The realistic representations
demand the following four elements: high spatial resolu-
tion, high temporal resolution, reproducing accurate color,
and large dynamic range. For example, digital cinema offers
digital images with high-resolution. In order to represent a
smooth movement, an high-speed HDTV camera that can
shoot at 300 [frames/sec] is developed. The advanced ef-
forts to reproduce accurate color are made. High dynamic
range imaging (HDRI) is a new imaging technology that has
emerged in recent years and it has the promise of bringing a
new revolution to digital imaging [1].
The real world scenes humans experience every day have
far higher luminance dynamic ranges. For instance, a scene
showing both shadows and sunlit areas will have a dynamic
range exceeding 100,000:1. Human visual system is capa-
ble of perceiving light intensities over a range of 4 orders
of magnitude, and with adaptation, its sensitivity can stretch
to 10 orders of magnitude. However, conventional computer
monitors and other reproduction media such as printing pa-
pers have limited dynamic ranges, often less than 2 orders
of magnitude. On the image capture side a similar argument
can be made. Most cameras limit their outputs to eight bits
per colour channel. Therefore, it is clear that the dynamic
range of current display, camera and image file formats are
not enough to represent real scenes and only record a fraction
of the contrast that humans are capable of perceiving.
In HDRI, the image files record the actual colour and
dynamic range of the original scene rather than the limited
gamut and dynamic range of the monitor or other reproduc-
tion media. This means that image processing, manipulation,
display and other operations will no longer be limited by the
number of bits used to represent each pixel. Thus, HDRI will
have widespread applications in digital cinema, digital pho-
tography, computer games, etc., and will open up many new
possibilities, including dramatically improving the visual re-
alism of digital photographs and videos, enabling the devel-
opment of more accurate computational vision techniques,
etc. In the near future, the imaging industry will inevitably
move to HDRI which will affect all components of the digital
imaging pipeline including capturing (sensor, camera), stor-
age (compression, coding) and reproduction (rendering, tone-
mapping, printing and display).
In this survey paper, we discuss technical issues, recent
development and future directions of this very promising and
exciting imaging technology. This paper is composed as fol-
lows. Section 2explains the technologies for acquisition and
generation of HDRI. Then, section 3treats with reproducing
technologies of HDRI. Section 4shows HDRI coding which
is needed for efficient storage and transmission.

2. ACQUISITION AND GENERATION OF HDR
IMAGE
2.1. Recent developments on HDRI sensors
The dynamic range of a camera CCD or CMOS sensor is de-
fined as the ratio of the full well capacity (FWC), i.e. the
maximum measurable signal, and the root-mean-square dark
noise, i.e. the lowest signal differentiable from the noise floor
(NF). Sensor manufacturers can increase the dynamic range
by either decreasing the NF or increasing the FWC and thus
the saturation level. An overview of the different solutions to
increase the sensor’s dynamic range is given in [2] and has
later been extended in [3].
Solutions to increase the dynamic range by changing from
a linear to a logarithmic response for the photon-electron con-
version have been suggested [4]. These sensors, however suf-
fer from increased fixed-pattern-noise (FPN) which becomes
most critical for low light conditions. Therefore combined
log-linear sensors [5] adapt the response curve to the lighting
conditions on a per pixel basis. Both methods reduce, how-
ever, the resolution of detectable light changes due to their
logarithmic responses.
Instead of increasing the FWC the sensor could as well
provide the time to saturation (TTS) as the signal correspond-
ing to the incident light flux. Sensors following this TTS ap-
proach have been presented in [6]. Other approaches deal
with pixel specific adaptive integration time (AIP) [7], which
has been combined with the TTS approach to improve the
low-light behaviour [8]. Furthermore it has been investigated
to include the concept of multiple exposures on the chip in
form of a Bayer-pattern like spatially varying neutral density
filter [9] as well as continually varying filters [10].
2.2. HDR image generation based on multiple exposure
principle
In the past decade, it had been widely agreed in the CG com-
munity that the dynamic range of the traditional imaging is
inadequate for Image-based lighting (IBL)[11], [12]. IBL is
the rendering process of illuminating objects with images of
light taken from the real world. IBL can provide realistic ap-
pearances when the image has a high dynamic range and is
radiometrically calibrated. Debevec first introduced a method
to acquire an omnidirectional HDR image called Light Probe
in [13] and applied it to IBL. Since then, the HDR image has
been widely used and is now available in many graphics pro-
cessors.
Several methods have been proposed to improve the dy-
namic range of general photographs based on a multiple ex-
posure principle. Mann et al. first attempted to construct the
HDR [14]. This algorithm is composed of the four steps: (1)
Photographs of still objects are taken off line by a camera at a
fixed location, (2) A camera response curve is estimated from
the multiple exposure set by self calibration, (3) Linearize the
images by applying the inverse of the response curve, and (4)
Merge the linearized images. These four steps is a basic pro-
cedure for the HDR acquisition and many of conventional al-
gorithms follow it. There are several variations on the camera
response curve estimation. While Mann et al. approximates
the response curve by a simple gamma function [14], Debevec
et al. [13] describes the curve by a set of exposure values for
more precise approximation. Mitsunaga et al. express the
curve by a low order polynomial [15], which provides a more
flexible radiometric model.
These techniques based on the multiple exposures have a
disadvantage. It is assumed that a scene is completely still
during taking photographs. Therefore if there is any motion
of objects or camera shake, ghosting artifacts will appear af-
ter combining the images. Motion compensation is a classical
problem and there exist many methods for optical flow esti-
mation and image registration. Ward [16] presented a method
to align the images and applied it to multiple exposure fusion.
Kang et al. [17] used gradient-based optical flow estimation
to remove the ghosting artifacts of the HDR video.
3. REPRODUCTION OF HDR IMAGE
3.1. Perceptual HDR image quality
Systems for HDR can be dissected into image capture, im-
age/video path (including compression) and displays. The
displays are most intimately related to the human visual sys-
tem (HVS) since there are no intervening unknown elements
between the display and the viewer. Since CRTs and film
have been able to achieve approximately 3 log units of dy-
namic range for many decades, that range forms a convenient
distinction between standard dynamic range (SDR) displays
and HDR displays. An early HDR display by combining 2
digital film images was developed by Greg Ward [18]. This
idea was extended to video by projecting a digital image as
an LCD backlight [19]. Using such equipment, several psy-
chophysical studies began assessing the advantages of HDR
displays for preferences, such as comparing SDR and HDR
imagery [20], comparing HDR images against the real-world
[21], and comparing preferred tonescales for mapping SDR
to HDR, [22].
In addition, the advantages in terms of functionality have
been studied, such as for medical images [23]. Some stud-
ies have been done casting doubt on the ability of the visual
system to be able to distinguish HDR from SDR images due
to optical flare, at least for static images [24], [25]. Regard-
ing video processing and tonescale manipulation, algorithms
have been designed to take into the account the extra range af-
forded by specular highlights and tested with observers [26],
or used HVS models in their design, such as spatial frequency
channels [27].

3.2. Tone Mapping
One of the problems of high dynamic range imaging is the
display of high dynamic range radiance maps on conventional
reproduction media such as LCD panels. One solution to this
problem is to compress the dynamic range of the radiance
maps such that the mapped image can be fitted into the dy-
namic range of the display devices. This mapping is called
tone mapping [28]. Several tone mapping methods have ap-
peared in the literature in recent years.
These methods can be divided into two broad categories.
The global tone mapping techniques [29, 30, 31] using a sin-
gle appropriately designed spatially invariant mapping func-
tion for all pixels in the image; the local mapping techniques
[32, 33, 34] adapt the mapping functions to local pixel statis-
tics and local pixel contexts. Global tone mapping is simpler
to implement but tend to lose details. Local tone mapping
is much more computationally intensive and harder to get it
right since there are often a number of parameters in the al-
gorithms which have to be set empirically. Given the nature
of the problem, it is not possible to have one method fits all,
i.e., it may not possible to have one method that will solve the
problem once and for all. What are needed are multiple meth-
ods and depending on the particular requirements of the users,
one method will be better suited than others, or a combination
of methods will be necessary.
4. EFFICIENT CODING METHOD FOR HDR IMAGE
Increase in dynamic range requires increase in bit depth in or-
der to represent natural quality with smooth gradation. Since
it leads to the amount of image data, we need the efficient en-
coding algorithm. As the studies for video coding with high
bit depth over 10 [bits/channel], layered scalable coding ap-
proaches are considered. These approaches feature to offer a
bit-depth scalability that can support multi bit-depth, and the
compatibility of the base layer with JPEG[35], MPEG-4[36]
and AVC/H.264[37]. Mantiuk et al. study a high dynamic
range video encoding that optimizes luminance quantization
based on the contrast threshold perception in the human vi-
sual system [38]. Ito et al. study a encoder design for high bit
depth sequences based on the optimization of tone mapping
curve [39]. This method features to design a tone mapping
curve that can minimize bit-depth transform error.
Recent inter-national standards of image/video coding
also support HDRI through increasing processable bit-depth.
AVC/H.264 includes three profiles (High 4:4:4 predictive,
High 4:4:4 intra and CAVLC 4:4:4 intra) [40] that support bit
depth up to 14 [bits/channel] and 4:4:4 color format, though
conventional video codecs mainly consider sequences with
4:2:0 format of 8 [bits/channel]. JPEG2000 supports bit depth
up to 12 [bits/channel] and 4:4:4 color format. Furthermore,
JPEG-XR can handle bit depth up to 32 [bits/channel] and
4:4:4 color format. AVC/H.264 also offer a SIE message [41]
that transmits information for describing tone mapping oper-
ation which adjusts the bit depth of each pixel values in the
reconstructed images. Using this SIE message, the decoder
of AVC/H.264 can display is given a post-processing func-
tionality that can map a higher bit depth sequences images to
a lower bit depth display.
5. CONCLUSION
Research in high dynamic range imaging has started more
than a decade ago, and it is a relatively young field. Already,
we have seen examples of dramatic visual quality improve-
ments that HDR image can achieve over traditional low dy-
namic range image. Currently, this new technology is still
in its early stage of development. In order to achieve its full
potential, for example, to implement HDR imaging technol-
ogy in consumer level digital camera to enable ordinary users
to take high quality photographs and videos under any light-
ing conditions, there are still many technical hurdles to over-
come, including, image sensing, coding and storage and dis-
play. This paper briefly surveyed what have been achieved in
HDR imaging which also highlighted that much needs to be
done in order to make this promising imaging technology the
mainstay of digital imaging.
6. REFERENCES
[1] G. Ward E. Reinhard, S.Pattanaik and P. Debevec, High Dy-
namic Range Imaging: Acquisition, Display, and Image-Based
Lighting, Morgan Kaufmann Publisher, 2005.
[2] O. Yadid-Pecht, “Wide-dynamic-range sensors,” Optical En-
gineering, vol. 38, no. 10, pp. 1650–16602007, 1999.
[3] A. Spivak, A. Belenky, A. Fish, and O. Yadid-Pecht, “Wide-
dynamic-range CMOS image sensors—comparative perfor-
mance analysis,” IEEE Trans. on Electron Devices, vol. 56,
no. 11, pp. 2446–2461, Nov. 2009.
[4] H.-Y. Cheng, B. Choubey, and S. Collins, “A high-dynamic-
range integrating pixel with an adaptive logarithmic response,”
IEEE Photonics Technology Letters, vol. 19, no. 15, pp. 1169–
1171, Aug. 2007.
[5] J. Guo and S. Sonkusale, “A high dynamic range CMOS im-
age sensor for scientific imaging applications,” IEEE Sensors
Journal, vol. 9, no. 10, pp. 1209–1218, Oct. 2009.
[6] X. Guo, X. Qi, and J. G. Harris, “A time-to-first-spike CMOS
image sensor,” IEEE Sensors Journal, vol. 7, no. 8, pp. 1165–
1175, Aug. 2007.
[7] P. M. Acosta-Serafini, I. Masaki, and C.G.Sodini, “A 1/3”
VGA linear wide dynamic range CMOS image sensor imple-
menting a predictive multiple sampling algorithm with over-
lapping integration intervals,” IEEE Journal of Solid-State Cir-
cuits, vol. 39, no. 9, pp. 1487–1496, Sept. 2004.
[8] D. Stoppa, M. Vatteroni, D. Covi, A. Baschirotto, A. Sartori,
and A. Simoni, “A 120-dB dynamic range CMOS image sen-
sor with programmable power responsivity,” IEEE Journal of
Solid-State Circuits, vol. 42, no. 7, pp. 1555–1563, July 2007.

[9] S. K. Nayar and T. Mitsunaga, “High dynamic range imaging:
spatially varying pixel exposures,” in Proc. IEEE Conf. on
Computer Vision and Pattern Recognition, 2000, vol. 1, pp.
472–479 vol.1.
[10] S. K .Nayar and V. Branzoi, “Adaptive dynamic range imaging:
optical control of pixel exposures over space and time,” in
Proc. IEEE Int. Conf. on Computer Vision, Oct. 2003, vol. 1,
pp. 1168–1175.
[11] G. Ward, “Real pixels,” in Graphic Gems II, pp. 80–83. Cam-
bridge University Press, 1991.
[12] P. Debevec, “Image-based lighting,” IEEE Computer Graphics
and Applications, vol. March/April, pp. 26–34, 2002.
[13] P. E. Debevec and J. Malik, “Recovering high dynamic range
radiance maps from photographs,” Proc. of SIGGRAPH 97,
Computer Graphics Proc., pp. 369–378, 1997.
[14] S. Mann and R. Picard, “On being ’undigital’ with digital
cameras: Extending dynamic range by combining differently
exposed pictures,” IS&T’s Annual Conf.; Society for Imaging
Science and Technology, vol. 56, pp. 2446–2461, May 1995.
[15] T. Mitsunaga and S. K. Nayer, “Radiometric self calibration,”
IEEE Conf. on Computer Vision and Pattern Recognition, vol.
1, pp. 374–380, Jun. 1999.
[16] G. Ward, “Fast, robust image registration for compositing high
dynamic range photographs from hand-held exposures,” Jour-
nal of Graphics Tools, vol. 8, pp. 17–30, 2003.
[17] S. Winder S. Kang, M. Uyttendaele and R. Szeliski, “High
dynamic range video,” Trans. on Graphics 2003, pp. 319–325,
2003.
[18] G .Ward, “A wide field, high dynamic range, stereographic
viewer,” Proc. of PICS 2002, April 2002.
[19] H. Seetzen, W. Heidrich, W. Stuerzlinger G. Ward, L. White-
head, M. Trentacoste, A. Ghosh, and A. Vorozcovs, “High
dynamic range display systems,” ACM SIGGRAPH, 2004.
[20] P. Ledda, A. Chalmers, T. Troscianko, and H. Seetzen, “Evalu-
ation of tone mapping operators using a high dynamic range
display,” in Proc. of ACM SIGGRAPH/EUROGRAPHICS
Symp. on Graphics hardware, 2007, vol. 24.
[21] A. Yoshida, V. Blanz, K. Myszkowski, and H. Seidel, “Per-
ceptual evaluation of tone mapping operators with real-world
scenes,” in Proc. of Human Vision and Electronic Imaging X,
IS&T/SPIE Annual Symp. Electronic Imaging, 2005, pp. 192 –
203.
[22] A. Aykuz, R. Fleming, B. Rieche, E. Rheinhard, and
H. Bultoff, “Do HDR displays support LDR content ? A psy-
chophysical evaluation,” Trans.of the ACM, 2007.
[23] A. Badano, G. Guarnieri, G. Ramponi, and L. Albani, “Quanti-
zation in medical imaging displays: initial observer results for
a high-luminance range dual-layer LCD,” Proc. SID, vol. 39,
pp. 923–926, 2009.
[24] A. Rizzi and J. McCann, “Glare-limited appearances in HDR
images,” Journal of the Society for Information Display, vol.
17, no. 1, pp. 3–12, 2009.
[25] J. McCann and A. Rizzi, “Retinal HDR images: Intraocular
glare and object size,” Journal of the Society for Information
Display, vol. 17, no. 11, pp. 913–920, 2009.
[26] L. Meylan, S. Daly, and S. S¨
usstrunk, “The reproduction
of specular highlights on high dynamic range displays,” in
IS&T/SID Color Imaging Conf. (CIC), 2006.
[27] R. Mantiuk, S. Daly, and L. Kerofsky, “Display adaptive tone
mapping,” ACM Trans. on Graphics (Proc. of SIGGRAPH),
vol. 27, no. 3, 2008.
[28] J. DiCarlo and B. Wandell, “Rendering high dynamic range
images,” in Proc. SPIE, 2001, vol. Proc. SPIE, pp. 392–401.
[29] G. W. Larson, H. Rushmeier, and C. Piatko, “A visibility
matching tone reproduction operator for high dynamic range
scenes,” IEEE Trans. on Visualization and Computer Graph-
ics, pp. 291–306, 1997.
[30] G. Qiu, J. Guan, J. Duan, and M. Chen, “Tone mapping for
HDR image using optimization - a new closed form solution,”
in Int. Conf. on Pattern Recognition, 2006, vol. 1, pp. 996–999.
[31] J. Duan, M. Bressan, C. Dance, and G. Qiu, “Tone-mapping
high dynamic range images by novel histogram adjustment,”
Pattern Recognition, vol. 43, no. 5, pp. 1847–1862, May 2010.
[32] F. Durand and J. Dorsey, “Fast bilateral filtering for the display
of high-dynamic- range images,” ACM Trans. on Graphics, pp.
257–266, 2002.
[33] E. Rheinhard, M. Stark, P. Shirley, and J. Ferwerda, “Pho-
tographic tone reproduction for digital images,” ACM SIG-
GRAPH, 2002.
[34] R. Fattal, D. Lischinski, and M. Werman, “Gradient domain
high dynamic range compression,” ACM Trans. on Graphics,
pp. 249–256, 2002.
[35] G. Ward and M. Simmons, “JPEG-HDR: A backwardscom-
patible, high dynamic range extension to JPEG,” in IEEE Int.
Conf. on Image Processing, 2005, pp. 283–290.
[36] R. Mantiuk, A. Efremov, K. Myszkowski, and H. Seidel,
“Backward compatible high dynamic range mpeg video com-
pression,” ACM Trans. Graph., 2006.
[37] M. Okuda and N. Adami, “Two-layer coding algorithm for
high dynamic range images based on luminance compensa-
tion,” Journal of Visual Communication and Image Represen-
tation, vol. 18, no. 5, pp. 377–386, Oct. 2007.
[38] R. Mantiuk, G. Krawczyk, K. Myszkowski, and H. Seidel,
“Perception-motivated high dynamic range video encoding,”
ACM Trans. Graph., vol. 23, no. 3, pp. 733–741, 2004.
[39] T. Ito, Y. Bandoh, S.Takamura, and H. Jozawa, “Coding algo-
rithm for high bit depth video using bit depth transform,” in
IEEE Int. Conf. on Image Processing, in this special session,
2010.
[40] ISO/IEC SC 29/WG 11 (MPEG) 14496-10:2004, Coding of
audio visual objects – Part 10: Advanced Video Coding 3rd
Ed., Int. Standard, Nov. 2007.
[41] A. Segall, A. Kerofsky, and S. Lei, “Tone mapping SEI mes-
sage,” JVT-T060. Klagenfurt, Austria, July 2007.