ArticlePDF Available

Single-pixel three-dimensional imaging with time-based depth resolution

Authors:

Abstract and Figures

Time-of-flight three-dimensional imaging is an important tool for applications such as object recognition and remote sensing. Conventional time-of-flight three-dimensional imaging systems frequently use a raster scanned laser to measure the range of each pixel in the scene sequentially. Here we show a modified time-of-flight three-dimensional imaging system, which can use compressed sensing techniques to reduce acquisition times, whilst distributing the optical illumination over the full field of view. Our system is based on a single-pixel camera using short-pulsed structured illumination and a high-speed photodiode, and is capable of reconstructing 128 × 128-pixel resolution three-dimensional scenes to an accuracy of ~3 mm at a range of ~5 m. Furthermore, by using a compressive sampling strategy, we demonstrate continuous real-time three-dimensional video with a frame-rate up to 12 Hz. The simplicity of the system hardware could enable low-cost three-dimensional imaging devices for precision ranging at wavelengths beyond the visible spectrum.
Content may be subject to copyright.
ARTICLE
Received 30 Sep 2015 |Accepted 20 May 2016 |Published 5 Jul 2016
Single-pixel three-dimensional imaging with
time-based depth resolution
Ming-Jie Sun1,2, Matthew P. Edgar2, Graham M. Gibson2, Baoqing Sun2, Neal Radwell2, Robert Lamb3
& Miles J. Padgett2
Time-of-flight three-dimensional imaging is an important tool for applications such as object
recognition and remote sensing. Conventional time-of-flight three-dimensional imaging
systems frequently use a raster scanned laser to measure the range of each pixel in the scene
sequentially. Here we show a modified time-of-flight three-dimensional imaging system,
which can use compressed sensing techniques to reduce acquisition times, whilst distributing
the optical illumination over the full field of view. Our system is based on a single-pixel
camera using short-pulsed structured illumination and a high-speed photodiode, and is
capable of reconstructing 128 128-pixel resolution three-dimensional scenes to an accuracy
of B3 mm at a range of B5 m. Furthermore, by using a compressive sampling strategy, we
demonstrate continuous real-time three-dimensional video with a frame-rate up to 12 Hz.
The simplicity of the system hardware could enable low-cost three-dimensional imaging
devices for precision ranging at wavelengths beyond the visible spectrum.
DOI: 10.1038/ncomms12010 OPEN
1Department of Opto-electronic Engineering, Beihang University, Beijing 100191, China. 2SUPA, School of Physics and Astronomy, University of Glasgow,
Glasgow G12 8QQ, UK. 3Selex ES, Edinburgh EH5 2XS UK. Correspondence and requests for materials should be addressed to M.-J.S. (email:
mingjie.sun@buaa.edu.cn) or M.J.P. (email: miles.padgett@glasgow.ac.uk).
NATURE COMMUNICATIONS | 7:12010 | DOI: 10.1038/ncomms12010 | www.nature.com/naturecommunications 1
Whilst a variety of three-dimensional (3D) imaging
technologies are suited for different applications,
time-of-flight (TOF) systems have set the benchmark
for performance with regards to a combination of accuracy and
operating range. TOF imaging is performed by illuminating a
scene with a pulsed light source and observing the back-scattered
light. Correlating the detection time of the back-scattered light
with the time of the illumination pulse allows the distance dto
objects within the scene to be estimated by d¼tc/2, where tis the
TOF and cis the propagation speed of light.
The transverse spatial resolution of the image obtained is
retrieved either by using a pixelated array or by using a
single-pixel detector with a scanning approach1–8. In both
cases, the inherent speed of light demands the use of detectors
with a fast response time and high-speed electronic read-out to
obtain high precision depth resolution. Advances in sensor
development have enabled the first TOF single-photon avalanche
detector array cameras to enter the commercial market, having a
resolution of 32 32 pixels. However, such devices are still in
their infancy9–11. On the contrary, there are mature single-pixel
detectors on the market, which provide stable time-resolved
measurements, and by using compressed sensing principles
for image reconstruction, which takes advantage of the sparsity
in natural scenes, the acquisition times of the scanning approach
is largely reduced12–16.
Recently there have been some interesting developments in 3D
imaging utilizing single-pixel detectors. One technique utilizes
structured illumination and spatially separated photodiodes to
obtain multiple images with different shading properties from
which 3D images can be reconstructed via photometric stereo17.
Another scheme scans a scene, pixel by pixel, using a pulsed
illumination source and measures the reflected light using an
avalanche photodiode, whereon the first detected photon is
used to recover depth and reflectivity via TOF18. An alternative
method for scanning a scene and recovering depth and
reflectivity via TOF has also been demonstrated utilizing
structured pulsed illumination19–22.
Among the mentioned demonstrations, many5,18,20–22 used
photon-counting detection (that is, Geiger mode), which is well
suited for low-light-level imaging. However, one limitation of
photon-counting detectors is the inherent electronic dead time
between successive measurements, often 10s of nanoseconds,
which prohibits the retrieval of short-range timing information
from a single illumination pulse. Instead, an accurate temporal
response from a 3D scene requires summing the data over many
back-scattered photons and hence many illumination pulses
(usually several hundreds or thousands21,22). In contrast, as
first demonstrated by Kirmani et al.19, a high-speed photodiode
can retrieve the temporal response from a single illumination
pulse, which can be advantageous in certain circumstances, for
instance when the reflected light intensity is comparatively large.
Incidentally, photon counting cannot operate under such
conditions since the detection will always be triggered by back-
scattered photons from the nearest, or most reflective, object,
rendering more distant objects invisible (Supplementary Note 1).
In this paper, we present a single-pixel 3D imaging system using
pulsed structured illumination and a high-speed (short response
time) photodiode for sampling the time-varying intensity of the
back-scattered light from a scene. We show that by using an
analogue photodiode to record the full temporal form of the back-
scattered light, along with our original 3D reconstruction
algorithm, it is possible to recover surface profiles of objects with
an accuracy much better than that implied by the finite temporal
bandwidth of the detector and digitization electronics. At distances
of B5 m we demonstrate a range profile accuracy of B3mmwith
image resolutions of 128 128 pixels, whilst simultaneously
recovering reflectivity information of the object. This accuracy is
achieved despite a detection bandwidth and a digitization interval
corresponding to distances of 150 and 60 mm, respectively. We
further demonstrate that by using a compressive sampling scheme,
the system is capable of performing continuous real-time 3D video
with a frame-rate up to 12 Hz.
Results
Experimental set-up. The single-pixel 3D imaging system,
illustrated in Fig. 1, consists of a pulsed laser and a digital
micromirror device (DMD) to provide time-varying structured
illumination. A high-speed photodiode is used in conjunction
with a fresnel lens condenser system to measure the back-
scattered intensity resulting from each pattern. The analogue
photodiode output is passed through a low-noise amplifier and
sampled using a high-speed digitizer. In our work we chose to use
the Hadamard matrices23 for providing structured illumination.
To remove sources of noise, such as fluctuations in ambient light
levels, we obtain differential signals by displaying each Hadamard
pattern, followed by its inverse pattern, and taking the difference
in the measured intensities24,25. Detailed information about
experimental set-up is provided in the Methods section.
Pulsed
laser
Beam
expander
DMD
Projection
lens
Scene
Collecting
lens
Single-pixel
photodiode
Laser
pulse
Broadened
signal
Ball
Head
Screen
v
v
t
t
Figure 1 | Single-pixel 3D imaging system. A pulsed laser uniformly illuminates a DMD, used to provide structured illumination onto a scene, and the
back-scattered light is collected onto a photodiode. The measured light intensities are used in a 3D reconstruction algorithm to reconstruct both depth and
reflectivity images.
ARTICLE NATURE COMMUNICATIONS | DOI: 10.1038/ncomms12010
2NATURE COMMUNICATIONS | 7:12010 | DOI: 10.1038/ncomms12010 | www.nature.com/naturecommunications
A 3D image of the scene is reconstructed utilizing the time-
varying back-scattered intensities (measured for each output pulse
of the laser) and the associated set of Npatterns used to provide
the structured illumination. An overview of the reconstruction
algorithm is shown in Fig. 2 (the result in this diagram represents a
scene of three objects B0.5 m apart in depth). An incident laser
pulse (Fig. 2a) is back-scattered from a scene. The high-speed
digitizer converts the amplified analogue signals (Fig. 2b) into
discrete data points (Fig. 2c), which are subsequently processed by
the computer algorithm. Whereas typical single-pixel imaging
schemes use the integrated signal for each illumination pattern to
reconstruct a two-dimensional image, our algorithm utilizes M
discretely sampled intensity points from the time-varying signal to
reconstruct Mtwo-dimensional images, resulting in an x,y,z
image cube (Fig. 2d). In the image cube, each transverse pixel (x,y)
has an intensity distribution (Fig. 2e) along the longitudinal axis
(z), which is related to the temporal shape of the pulse, the detector
response, the read-out digitization and the pixel depth and
reflectivity information.
To enhance the range precision beyond the limits imposed by
the sampling rate of the system, methods such as parametric
deconvolution19,21 and curve fitting can be used. However, often
these methods can be computationally intensive, which makes
them unsuitable for real-time applications. Instead we choose to
apply cubic spline interpolation to the reconstructed temporal
signal at each pixel location, which introduces minimal
computational overhead. The depth map of the scene (Fig. 2f)
is subsequently determined by finding the maximum in these
interpolated signals. In addition, the scene reflectivity (Fig. 2g)
can be calculated by averaging the image cube along the
longitudinal axis. Utilizing both the depth and reflectivity
information, a 3D image of the scene is then reconstructed.
It is worth mentioning that, with the assumption that there is
only one surface at each transverse pixel location, our depth
estimation (see detail in Methods) works well for scenes that
have smooth features, such as the mannequin head and ball, since
the reconstructed temporal signals should be slowly varying
between sample points. We note that the depth accuracy
is limited by the amplitude noise of the data points, over-
interpolation only increases the depth precision, but not
necessarily the accuracy. In addition, more interpolation adds
more processing time, therefore, in our experiments we chose to
interpolate by a factor of five times when investigating the
static scenes with 20 pulses per pattern (Figs 3–5), and four
times when investigating scenes with motion (Fig. 6), balancing
the computational overhead and 3D image quality.
High-resolution imaging. In one experiment, a scene containing
a 140-mm diameter polystyrene soccer ball, a life-size skin-tone
mannequin head and a screen was located at a distance of B5.5 m
from the imaging system (Fig. 3a). The objects were closely
separated in distance such that the total depth of the scene was
B360 mm. A complete Hadamard set of 16,384 patterns, and
their inverse, was used as the structured illumination, and the
back-scattered intensities (Fig. 3b) measured for reconstructing
128 128-pixel resolution depth map, reflectivity and 3D image
(Fig. 3c-3e). The illumination time of each pattern was 2.66 ms,
corresponding to 20 laser pulses. The total time for acquisition,
ab c
Objects
reflection
Digitizer
sampling
Image
computation
de
Image cube
510
555
610
Greyscale
255
0
Greyscale
255
0
Greyscale
255
0
Depth (cm)
Depth (cm)
Depth (cm)
450
450
450
750
750
750
Processing
Laser pulse
V
Time (ns)
010
Broadened signal
V
Time (ns)
30 50
V
Time-discretized signal
Time (ns)
30 50
Intensity distribution of depth for
different transverse pixels
Reflectivity
estimation
Depth
estimation
Depth map Reflectivity
fg
.... .... .... ....
Distance (m)
6.30
5.10
z
y
x
Figure 2 | Overview of the reconstruction algorithm. The incident laser pulses (a) back-scattered from a 3D scene are temporally broadened (b) and
discretely sampled using a high-speed digitizer (c). An image cube (d) is obtained using a reconstruction algorithm, having an intensity distribution along
the longitudinal axis (e), from which the depth map (f) and the reflectivity (g) of the scene can be estimated.
NATURE COMMUNICATIONS | DOI: 10.1038/ncomms12010 ARTICLE
NATURE COMMUNICATIONS | 7:12010 | DOI: 10.1038/ncomms12010 | www.nature.com/naturecommunications 3
V
Time (ns)32 52
b
c
Distance (m)
6.00
5.50
ea
0.36 m
0.25 m
Scene Temporal signal
Depth map Reflectivity
3D image
x
y
z
d
Figure 3 | 3D image of a scene. (a) Illustration of the scene containing multiple objects in close proximity. (b) Reflected intensity measured for
uniform illumination, indicating temporally indistinguishable objects. (c) The estimated depth map of the scene. (d) The reconstructed scene reflectivity.
(e) A textured 3D image of the scene.
Distance (m)
5.70
5.50
0
9
Depth map
Reflectivity
3D image
Error (mm)
Side view
Front view
Error map
x
y
z
50 mm
b
ca
e
fd
Figure 4 | Quantitative analysis of 3D reconstruction. The depth estimation (a) reflectivity (b) and 3D reconstruction (c) of a white polystyrene
mannequin head at a range of B5.5 m. Superposed depth reconstruction and photograph of the mannequin head, viewed from the front (d) and side
(e). For a chosen region of interest an error map (f) showing the absolute differences between our depth result and that obtained using a
stereophotogrammetric camera system.
0.5 m
Netting
Head
Line of sight
Scene Conventional photo
Time-gated reflectivity
3D image
Distance (m)
Depth map
3.67
3.50
x
y
z
c
bea
d
Figure 5 | 3D imaging through obscuring material. (a) Illustration of scene containing a mannequin head with black netting material obscuring the
line-of-sight. (b) A photograph of the scene taken from the perspective of the 3D imaging system. The scene depth (c) and reflectivity (d) reconstructed
by time-gating the measured intensity data. (e) 3D reconstruction of the mannequin head.
ARTICLE NATURE COMMUNICATIONS | DOI: 10.1038/ncomms12010
4NATURE COMMUNICATIONS | 7:12010 | DOI: 10.1038/ncomms12010 | www.nature.com/naturecommunications
data transfer from the digitizer buffer to the computer and image
processing was B130 s. The 3D reconstruction shown in Fig. 3e
exhibits distinguishable features, such as the profile of the head
and the ball.
To quantitatively determine the accuracy of our 3D imaging
system, the scene was modified to contain only a polystyrene
mannequin head (180 270 250 mm), for which we had
reference 3D data obtained via a high-accuracy stereophotogram-
metric camera system17,26. The mannequin head was located at a
distance of 5.5 m from the imaging system. To further
demonstrate the system capability for retrieving reflectivity in
addition to the depth, two grey stripes were placed on the head.
Performing the same acquisition and imaging processing used in
Fig. 3, we obtained the results shown in Fig. 4a–c. Figure 4d,e
show the front and side view comparisons between our 3D
reconstruction (green) and photographs of the head (white),
respectively. After lateral and angular registration and subsequent
depth scaling, an error map representing the absolute differences
for a chosen region of interest was obtained (shown in Fig. 4f).
From this comparison we find our single-pixel 3D imaging system
has a root mean squared error of 2.62 mm. More detailed analysis
is provided in Supplementary Fig. 1 and Supplementary Note 2.
Time-gated imaging. One advantage of time-resolved imaging is
the ability to distinguish objects at different depths, by artificially
time-gating the measured intensity. In certain cases this enables
obscuring objects to be isolated from objects of interest. Similar to
previous demonstrations20, we constructed a 3D scene containing
a polystyrene mannequin head (located at a distance of B3.5 m)
and black-coloured netting used to obstruct the line of sight
(located at a distance of B3 m), as illustrated in Fig. 5a. An
image of the scene taken using a conventional camera is shown in
Fig. 5b, where the head is obscured by the black netting.
Performing the same acquisition and imaging processing used in
Figs 3 and 4, along with an artificial gating on the photodiode
data to ensure no reflected signals from the black netting are
included in the 3D reconstruction, we obtained the results shown
in Fig. 5c–e. As before, we note the characteristic features of the
mannequin head can be resolved.
Real-time compressed video. In addition to obtaining
high-quality 3D images of static scenes, many applications
demand video frame-rates for motion tracking in dynamic scenes.
A key merit of single-pixel imaging is the ability to take advantage
of the sparsity of the scene and use compressive sensing to reduce
the acquisition time. Most compressive-sensing schemes are
performed by minimizing a certain measure of the sparsity,
such as L
1
-norm, to find the sparsest image as the optimal
reconstruction of the scene. However, for resolutions greater than
32 32 pixels, the time taken by the construction algorithm often
prohibits real-time application22,27.
In this work we use an alternative scheme, known as
evolutionary compressive sensing6,7, which aims to reconstruct
the image with significant less time than conventional compressive
sensing by performing a linear iteration (Supplementary Figs 2 and
3 and Supplementary Note 3). In short, the evolutionary
compressive sensing scheme chooses a subset of the Hadamard
basis to display, by selecting the patterns with the most significant
intensities measured in the previous frame, in addition to a fraction
of randomly selected patterns that were not displayed. In this
experiment, a scene consisting of a static polystyrene mannequin
head and a polystyrene white ball (140mm diameter) swinging
along the line of sight with a period of B3 s (Fig. 6a). The scene
was located at a distance of B4 m from the imaging system. Two
laser pulses were used per illumination pattern. With the approach
described above we obtained continuous real-time 3D video with a
frame-rate of 5Hz using 600 patterns (including their inverse)
from the available 6464 Hadamard set, equivalent to a
t= 11.4 s t= 11.6 s t= 11.8 s
t= 12 s t= 12.2 s t=12.4 s t=12.6 s
Distance (m)
4.3
3.7
Reflectivity
255
0
0.6 m
Scene
a
Distance (m)
4.3
3.7
Reflectivity
255
0
bc d
efgh
Figure 6 | Real-time 3D video. (a) Illustration of the scene containing a static mannequin head and a swinging ball. (bh) Sample of consecutive depth and
reflectivity frames reconstructed at B5 Hz frame-rate in real-time for a transverse resolution of 64 64 pixels. The experiments of Figs 3–6 were
replicated more than 10 times.
NATURE COMMUNICATIONS | DOI: 10.1038/ncomms12010 ARTICLE
NATURE COMMUNICATIONS | 7:12010 | DOI: 10.1038/ncomms12010 | www.nature.com/naturecommunications 5
compression ratio of C7% (Supplementary Movie 1). The
experimental parameters for this result were chosen to balance
the inherent trade-off between frame-rate and image quality
(Supplementary Note 3). Figure 6b–h show a sample of
consecutive frames from the 3D video. The result shows an
identifiable 3D image of the mannequin head and ball, in addition
to the real-time motion of the ball. Importantly, however, 3D
reconstruction can be performed using fewer patterns to achieve
higher frame-rates if required, for instance using 256 patterns
provides 12 Hz video (Supplementary Movie 2).
Discussion
We have demonstrated that our single-pixel 3D imaging system is
capable of reconstructing a scene with millimetric ranging
accuracy using modest hardware. In addition, we obtained real-
time video rates by taking advantage of a modified compressive
sensing scheme that does not rely on lengthy post-processing.
The performance of the system in this work was mainly limited
by the repetition rate of the laser used, 7.4 kHz. Using a laser with
a repetition rate greater than or equal to the DMD modulation
rate, could enable faster 3D video rates by a factor of three and/or
increase reconstruction accuracy by increased averaging.
Furthermore, the broad operational spectrum (400–2,500 nm)
of the DMD could enable the system to be extended to the
non-visible wavelength, such as the infrared, using modified
source and detection optics. The use of DMDs in the infrared has
already been demonstrated in microscopy6and real-time video
cameras7. The potential application of 3D imaging in the infrared
could provide enhanced visibility at long-range, due to reduced
atmospheric scattering28.
Methods
Hardware specifications.The following components were used in the
experimental set-up (Fig. 1): a pulsed laser (Teem Photonics SNG-03E-100,
532 nm); a DMD (Texas Instruments Discovery 4100 DMD); a projection lens
(Nikon ED, f¼180 mm); a collection lens (customized fresnel condenser lens,
f¼20 mm); a Si biased photodiode (Thorlabs DET10A); and a high-speed USB
digitizer (PicoScope 6407, 2.5 GS s 1for two-channel acquisition).
Operating configurations.There are several important points worth mentioning.
(a) The modulation rate of the DMD can reach up to 22.7kHz, however, in this
experiment the DMD is operated in slave-mode, meaning the modulation rate is
determined by the repetition rate of the laser at 7.4 kHz. (b) The active area of the
photodiode is 0.8 mm2, used in conjuction with a 20-mm focal length fresnel lens
system, giving a 2.6°field of view, which matches that of the projection system. The
depth estimation includes Gaussian smoothing, intensity calibration, cubic spline
interpolation and depth determination. More experiment methods and details are
provided in Supplementary Methods.
Data availability.The data used to generate all of the figures in this study can be
found at http://dx.doi.org/10.5525/gla.researchdata.317.
References
1. Schwarz, B. Lidar: mapping the world in 3D. Nat. Photon. 4, 429–430 (2010).
2. McCarthy, A. et al. Long-range time-of-flight scanning sensor based on
high-speed time-correlated single-photon counting. Appl. Opt. 48, 6241–6251
(2009).
3. Shapiro, J. H. Computational ghost imaging. Phys. Rev. A 78, 061802 (2008).
4. Bromberg, Y., Katz, O. & Silberberg, Y. Ghost imaging with a single detector.
Phys. Rev. A 79, 053840 (2009).
5. McCarthy, A. et al. Kilometer-range, high resolution depth imaging via 1560 nm
wavelength single-photon detection. Opt. Express 21, 8904–8915 (2013).
6. Radwell, N. et al. Single-pixel infrared and visible microscope. Optica 1,
285–289 (2014).
7. Edgar, M. P. et al. Simultaneous real-time visible and infrared video with
single-pixel detectors. Sci. Rep. 5, 10669 (2015).
8. Sun, M.-J., Edgar, M. P., Phillips, D. B., Gibson, G. M. & Padgett, M. J.
Improving the signal-to-noise ratio of single-pixel imaging using digital
microscanning. Opt. Express 24, 260309 (2016).
9. Niclass, C. & Charbon, E. in Solid-State Circuits Conference. Digest of Technical
Papers 364–604 (San Francisco, CA, USA, 2005).
10. Richardson, J. et al. in Custom Integrated Circuits Conference, 77–80 (San Jose,
CA, USA, 2009).
11. Entwistle, M. et al. in Proc. SPIE, (ed. Itzler, M.A.) Vol. 8375 (SPIE, 2012).
12. Baraniuk, R. G. Compressive sensing [lecture notes]. IEEE Signal Process. Mag.
24, 118–121 (2007).
13. Duarte, M. F. et al. Single-pixel imaging via compressive sampling. IEEE Signal
Process. Mag. 25, 83–91 (2008).
14. Young, M. et al. Real-time high-speed volumetric imaging using compressive
sampling optical coherence tomography. Biomed. Opt. Express 2, 2690–2697
(2011).
15. Hatef, M., Sina, J., Matan, G. & Donoho, D. L. Deterministic matrices matching
the compressed sensing phase transitions of gaussian random matrices. Proc.
Natl Acad. Sci. USA 110, 1181–1186 (2013).
16. Herman, A. M. A., Tidman, J., Hewitt, D., Weston, T. & Mcmackin, L. in Proc.
SPIE (ed. Fauzia Ahmad) Vol. 8717 (SPIE, 2013).
17. Sun, B. et al. 3D computational imaging with single-pixel detectors. Science
340, 844–847 (2013).
18. Kirmani, A. et al. First-photon imaging. Science 343, 58–61 (2014).
19. Kirmani, A., Colac¸o, A., Wong, F. N. & Goyal, V. K. Exploiting sparsity in
time-of-flight range acquisition using a single time-resolved sensor. Opt.
Express 19, 21485–21507 (2011).
20. Howland, G. A., Dixon, P. B. & Howell, J. C. Photon-counting compressive
sensing laser radar for 3D imaging. Appl. Opt. 50, 5917–5920 (2011).
21. Colaco, A., Kirmani, A., Howland, G. A., Howell, J. C. & Goyal, V. K. in
Proceedings of the 2012 IEEE Conference on Computer Vision Pattern
Recognition (CVPR) 96–102 (Washington, DC, USA, 2012).
22. Howland, G. A., Lum, D. J., Ware, M. R. & Howell, J. C. Photon counting
compressive depth mapping. Opt. Express 21, 23822–23837 (2013).
23. Pratt, W. K., Kane, J. & Andrews, H. C. Hadamard transform image coding.
Proc. IEEE 57, 58–68 (1969).
24. Sun, B. et al. in Computational Optical Sensing and Imaging (OSA, CTu1C–4,
2013).
25. Sun, M.-J., Li, M.-F. & Wu, L.-A. Nonlocal imaging of a reflective
object using positive and negative correlations. Appl. Opt. 54, 7494–7499
(2015).
26. Khambay, B. et al. Validation and reproducibility of a high-resolution
three-dimensional facial imaging system. Br. J. Oral Maxillofac. Surg. 46, 27
(2008).
27. Abmann, M. & Bayer, M. Compressive adaptive computational ghost imaging.
Sci. Rep. 3, 1545 (2013).
28. Bucholtz, A. Rayleigh-scattering calculations for the terrestrial atmosphere.
Appl. Opt. 34, 2765–2773 (1995).
Acknowledgements
We thank Prof. Adrian Bowman and Dr Liberty Vittert for providing the
stereophotogrammetric 3D data of the polystyrene mannequin head. M.S. acknowledges
the support from National Natural Foundation of China (Grant No. 61307021) and
China Scholarship Council (Grant No. 201306025016). M.J.P. acknowledges financial
support from UK Quantum Technology Hub in Quantum Enhanced Imaging (Grant No.
EP/M01326X/1), the Wolfson foundation and the Royal Society.
Author contributions
M.-J.S., R.L. and M.J.P. conceived the concept of the experiment; M.-J.S., M.P.E. and
G.M.G. designed and performed the experiments; M.-J.S., M.P.E. and M.J.P. designed the
reconstruction algorithm; N.R. developed the evolutionary compressed sensing
algorithm; M.-J.S., M.P.E., B.S. and M.J.P. analysed the results; M.-J.S., M.P.E. and M.J.P.
wrote the manuscript and other authors provided editorial input.
Additional Information
Supplementary Information accompanies this paper at http://www.nature.com/
naturecommunications
Competing financial interests: The authors declare no competing financial interests.
Reprints and permission information is available online at http://npg.nature.com/
reprintsandpermissions/
How to cite this article: Sun, M.-J. et al. Single-pixel three-dimensional imaging with
time-based depth resolution. Nat. Commun. 7:12010 doi: 10.1038/ncomms12010 (2016).
This work is licensed under a Creative Commons Attribution 4.0
International License. The images or other third party material in this
article are included in the article’s Creative Commons license, unless indicated otherwise
in the credit line; if the material is not included under the Creative Commons license,
users will need to obtain permission from the license holder to reproduce the material.
To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/
ARTICLE NATURE COMMUNICATIONS | DOI: 10.1038/ncomms12010
6NATURE COMMUNICATIONS | 7:12010 | DOI: 10.1038/ncomms12010 | www.nature.com/naturecommunications

Supplementary resources (3)

Data
July 2016
Ming-Jie Sun · Matthew P. Edgar · Graham M. Gibson · Baoqing Sun · Miles John Padgett
Data
July 2016
Ming-Jie Sun · Matthew P. Edgar · Graham M. Gibson · Baoqing Sun · Miles John Padgett
... SPI modulates the image of a scene using spatially resolved patterns to obtain a set of total light intensities, the correlation of which with the patterns yields the reconstructed images. The appeal of SPI lies in the efficiency of single-pixel detectors and post-processing algorithms, leading to its successful application in fields such as terahertz imaging [2] , fluorescence imaging [3] , and 3D imaging [4] . Spectral imaging, another application of SPI, involves using a spectrometer as the single-pixel detector. ...
... Single pixel imaging, where 2D images can be captured using a spatial light modulation element and single pixel detector, can be advantageous in a range of application scenarios [1]. These include challenging wavelength ranges where focal plane array technologies are not available [2,3], terahertz frequencies [4][5][6], where precise timing and depth resolution are sought [7][8][9][10][11][12][13], or for multispectral [14][15][16][17], hyperspectral [18,19], and microscopy imaging [18,20,21], or imaging through scattering media [22,23]. Single pixel imaging has been developed over the last couple of decades since first reported by Duarte et al. [24]. ...
Article
Full-text available
Achieving high frame-rate operation in single pixel imaging schemes normally demands significant compromises in the flexibility of the imaging system, requiring either complex optical setups or a hardware-limited pattern mask set. Here, we demonstrate a single pixel imaging capability with pattern frame-rates approaching 400 kfps with a recently developed microLED light projector and an otherwise simple optical setup. The microLED array has individually addressable pixels and can operate significantly faster than digital micromirror devices, allowing flexibility with regards to the pattern masks employed for imaging even at the fastest frame-rates. Using a full set of Hadamard or Noiselet patterns, we demonstrate 128 × 128 pixel images being generated at 7.3 fps. We generate a pattern set specifically for the light projector using deep learning tools and use these patterns to demonstrate single pixel imaging at almost 800 fps.
... Standard Compressed Sensing (CS) formulations and solvers were utilized in [26] to achieve resolutions of up to 256 × 256 pixels within a practical acquisition time of 3 s. The single-pixel three-dimensional imaging system constructed in [27] achieved a 128 × 128 resolution in 3D imaging. In that study, the authors used a comprehensive Hadamard set consisting of 16,384 patterns and their inverses as structured illumination. ...
Article
Full-text available
The real-time tracking of moving objects has extensive applications in various domains. Existing tracking methods typically utilize video image processing, but their performance is limited due to the high information throughput and computational requirements associated with processing continuous images. Additionally, imaging in certain spectral bands can be costly. This paper proposes a non-imaging real-time three-dimensional tracking technique for distant moving targets using single-pixel LiDAR. This novel approach involves compressing scene information from three-dimensional to one-dimensional space using spatial encoding modulation and then obtaining this information through single-pixel detection. A LiDAR system is constructed based on this method, where the peak position of the detected full-path one-dimensional echo signal is used to obtain the target distance, while the peak intensity is used to obtain the azimuth and pitch information of the moving target. The entire process requires minimal data collection and a low computational load, making it feasible for the real-time three-dimensional tracking of single or multiple moving targets. Outdoor experiments confirmed the efficacy of the proposed technology, achieving a distance accuracy of 0.45 m and an azimuth and pitch angle accuracy of approximately 0.03° in localizing and tracking a flying target at a distance of 3 km.
Conference Paper
We demonstrate photon-counting single-pixel 3D imaging using a multimode-fiber-coupled fractal SNSPD and showcase 32 × 32-pixel imaging with reflectance and depth contrasts at the wavelength of 1560 nm.
Article
Full-text available
In this study, a tracking and pointing control system with a dual-FSM (fast steering mirror) two-dimensional flexible turntable composite axis is proposed. It is applied to the target-tracking accuracy control in a GI LiDAR (ghost imaging LiDAR) system. Ghost imaging is a multi-measurement imaging method; the dual-FSM GI LiDAR tracking and pointing imaging control system proposed in this study mainly solves the problems of the high-resolution remote sensing imaging of high-speed moving targets and various nonlinear disturbances when this technology is transformed into practical applications. Addressing the detrimental effects of nonlinear disturbances originating from internal flexible mechanisms and assorted external environmental factors on motion control’s velocity, stability, and tracking accuracy, a nonlinear active disturbance rejection control (NLADRC) method based on artificial neural networks is advanced. Additionally, to overcome the limitations imposed by receiving aperture constraints in GI LiDAR systems, a novel optical path design for the dual-FSM GI LiDAR tracking and imaging system is put forth. The implementation of the described methodologies culminated in the development of a dual-FSM GI LiDAR tracking and imaging system, which, upon thorough experimental validation, demonstrated significant improvements. Notably, it achieved an improvement in the coarse tracking accuracy from 193.29 μrad (3σ) to 87.21 μrad (3σ) and enhanced the tracking accuracy from 10.1 μrad (σ) to 1.5 μrad (σ) under specified operational parameters. Furthermore, the method notably diminished the overshoot during the target capture process from 28.85% to 12.8%, concurrently facilitating clear recognition of the target contour. This research contributes significantly to the advancement of GI LiDAR technology for practical application, showcasing the potential of the proposed control and design strategies in enhancing system performance in the face of complex disturbances.
Article
Full-text available
Single-pixel cameras provide a means to perform imaging at wavelengths where pixelated detector arrays are expensive or limited. The image is reconstructed from measurements of the correlation between the scene and a series of masks. Although there has been much research in the field in recent years, the fact that the signal-to-noise ratio (SNR) scales poorly with increasing resolution has been one of the main limitations prohibiting the uptake of such systems. Microscanning is a technique that provides a final higher resolution image by combining multiple images of a lower resolution. Each of these low resolution images is subject to a sub-pixel sized lateral displacement. In this work we apply a digital microscanning approach to an infrared single-pixel camera. Our approach requires no additional hardware, but is achieved simply by using a modified set of masks. Compared to the conventional Hadamard based single-pixel imaging scheme, our proposed framework improves the SNR of reconstructed images by ∼ 50 % for the same acquisition time. In addition, this strategy also provides access to a stream of low-resolution ‘preview’ images throughout each high-resolution acquisition.
Article
Full-text available
The Hanbury Brown and Twiss (HBT) effect is a classical intensity correlation effect, but it is also widely used in the quantum optics regime, and has led to many important breakthroughs in both basic and applied physics, among which ghost imaging (GI) has aroused particular interest. In this article, the positive and negative intensity correlations in HBT correlation are analyzed, based on which we describe experiments on thermal light nonlocal imaging of a reflective object using the positive and negative correlations of correspondence imaging. An improvement of 16.3% in the signal-to-noise ratio of the reconstructed image has been achieved, indicating that this method may have promising potential in future GI applications where noise is a serious problem and smaller sampling numbers are necessary.
Article
Full-text available
Conventional cameras rely upon a pixelated sensor to provide spatial resolution. An alternative approach replaces the sensor with a pixelated transmission mask encoded with a series of binary patterns. Combining knowledge of the series of patterns and the associated filtered intensities, measured by single-pixel detectors, allows an image to be deduced through data inversion. In this work we extend the concept of a 'single-pixel camera' to provide continuous real-time video at 10 Hz , simultaneously in the visible and short-wave infrared, using an efficient computer algorithm. We demonstrate our camera for imaging through smoke, through a tinted screen, whilst performing compressive sampling and recovering high-resolution detail by arbitrarily controlling the pixel-binning of the masks. We anticipate real-time single-pixel video cameras to have considerable importance where pixelated sensors are limited, allowing for low-cost, non-visible imaging systems in applications such as night-vision, gas sensing and medical diagnostics.
Article
Full-text available
Microscopy is an essential tool in a huge range of research areas. Until now, microscopy has been largely restricted to imaging in the visible region of the electromagnetic spectrum. Here we present a microscope system that uses single-pixel imaging techniques to produce images simultaneously in the visible and shortwave infrared. We apply our microscope to the inspection of various objects, including a silicon CMOS sensor, highlighting the complementarity of the visible and shortwave infrared wavebands. The system is capable of producing images with resolutions between 32 × 32 and 128 × 128 pixels at corresponding frame rates between 10 and 0.6 Hz. We introduce a compressive technique that does not require postprocessing, resulting in a predicted frame rate increase by a factor 8 from a compressive ratio of 12.5% with only 28% relative error.
Conference Paper
Full-text available
Active range acquisition systems such as light detection and ranging (LIDAR) and time-of-flight (TOF) cameras achieve high depth resolution but suffer from poor spatial resolution. In this paper we introduce a new range acquisition architecture that does not rely on scene raster scanning as in LIDAR or on a two-dimensional array of sensors as used in TOF cameras. Instead, we achieve spatial resolution through patterned sensing of the scene using a digital micromirror device (DMD) array. Our depth map reconstruction uses parametric signal modeling to recover the set of distinct depth ranges present in the scene. Then, using a convex program that exploits the sparsity of the Laplacian of the depth map, we recover the spatial content at the estimated depth ranges. In our experiments we acquired 64×64-pixel depth maps of fronto-parallel scenes at ranges up to 2.1 M using a pulsed laser, a DMD array and a single photon-counting detector. We also demonstrated imaging in the presence of unknown partially-transmissive occluders. The prototype and results provide promising directions for non-scanning, low-complexity range acquisition devices for various computer vision applications.
Article
Full-text available
We demonstrate a compressed sensing, photon counting lidar system based on the single-pixel camera. Our technique recovers both depth and intensity maps from a single under-sampled set of incoherent, linear projections of a scene of interest at ultra-low light levels around 0.5 picowatts. Only two-dimensional reconstructions are required to image a three-dimensional scene. We demonstrate intensity imaging and depth mapping at 256 × 256 pixel transverse resolution with acquisition times as short as 3 seconds. We also show novelty filtering, reconstructing only the difference between two instances of a scene. Finally, we acquire 32 × 32 pixel real-time video for three-dimensional object tracking at 14 frames-per-second.
Article
Full-text available
Cheap Pix Three-dimensional (3D) images can be captured by, for example, holographic imaging or stereoimaging techniques. To avoid using expensive optical components that are limited to specialized bands of wavelengths, Sun et al. (p. 844 ; see the Perspective by Faccio and Leach ) projected pulses of randomly textured light onto an object. They were able to reconstruct an image of the 3D object by detecting the reflected light with several photodetectors without any need for lenses. The patterned light beams can thus in principle be substituted for light sources of any wavelength.
Article
Full-text available
This paper highlights a significant advance in time-of-flight depth imaging: by using a scanning transceiver which incorporated a free-running, low noise superconducting nanowire single-photon detector, we were able to obtain centimeter resolution depth images of low-signature objects in daylight at stand-off distances of the order of one kilometer at the relatively eye-safe wavelength of 1560 nm. The detector used had an efficiency of 18% at 1 kHz dark count rate, and the overall system jitter was ~100 ps. The depth images were acquired by illuminating the scene with an optical output power level of less than 250 µW average, and using per-pixel dwell times in the millisecond regime.
Conference Paper
A computational ghost-imaging arrangement that uses only a single-pixel detector is described. It affords a new 3D sectioning capability and matches the resolution of pseudothermal ghost imaging.
Article
Imagers that use their own illumination can capture three-dimensional (3D) structure and reflectivity information. With photon-counting detectors, images can be acquired at extremely low photon fluxes. To suppress the Poisson noise inherent in low-flux operation, such imagers typically require hundreds of detected photons per pixel for accurate range and reflectivity determination. We introduce a low-flux imaging technique, called first-photon imaging, which is a computational imager that exploits spatial correlations found in real-world scenes and the physics of low-flux measurements. Our technique recovers 3D structure and reflectivity from the first detected photon at each pixel. We demonstrate simultaneous acquisition of sub–pulse duration range and 4-bit reflectivity information in the presence of high background noise. First-photon imaging may be of considerable value to both microscopy and remote sensing.