Figure 4 - uploaded by Qin Cai
Content may be subject to copyright.
Computing 3D mesh from a depthmap. Every 2×2 pixel square on the left is transformed into two triangles as in the right figure. After 2D triangulation, corresponding 3D points for each pixel location is computed by using the depth map and the 3D triangulated mesh is computed.
Source publication
Virtual view synthesis from an array of cameras has been an essential element of three-dimensional video broadcasting/conferencing. In this paper, we propose a scheme based on a hybrid camera array consisting of four regular video cameras and one time-of-flight depth camera. During rendering, we use the depth image from the depth camera as initiali...
Contexts in source publication
Context 1
... depth estimate is first transformed into a 3D mesh in order to render disre- garding the depth camera resolution. This is a straightforward process. As shown in Fig. 4, the 2D grid of the depth map is first triangulated by defining 2 triangles for every 2 × 2 square pixels and then the 3D position of each pixel is computed using the depth estimate. When necessary, mesh simplification can be conducted to reduce the number of vertices and facets. The final result is a 3D mesh repre- sentation that is ...
Context 2
... Fig. 14, we show two failure cases to demonstrate the limitation of our algorithm. One problem we need to address in the future work is the heavy de- pendence on the depth map provided by the depth camera. For example, in cases where the depth camera fails to return a depth estimate due to the lack of infrared light reflection, our current ...
Similar publications
This paper addresses the problem of wireless transmission of a captured scene from multiple cameras, which do not communicate
among each other, to a joint decoder. Correlation among different camera views calls for distributed source coding for efficient
multiview image compression. The fact that cameras are placed within a short range of each othe...
The objective of this paper is to investigate the counterjet dynamics generated during the bubble rebound stage near a rigid boundary via both experimental and numerical methods. In the experiments, the temporal evolution of the bubble shapes and the formation of the counterjet are recorded by the high-speed camera. The results are presented for a...
Citations
... Qualitative evaluation In Fig. 9 we show some sample input and output from our method. We also show the results from other alternative solutions, including single-view image warping (best input view used here) and a multiview depth and texture fusion scheme similar to [75]. Visually inspected, our method produces realistic rendering results significantly better than the compared solutions. ...
The VirtualCube system is a 3D video conference system that attempts to overcome some limitations of conventional technologies. The key ingredient is VirtualCube, an abstract representation of a real-world cubicle instrumented with RGBD cameras for capturing the 3D geometry and texture of a user. We design VirtualCube so that the task of data capturing is standardized and significantly simplified, and everything can be built using off-the-shelf hardware. We use VirtualCubes as the basic building blocks of a virtual conferencing environment, and we provide each VirtualCube user with a surrounding display showing life-size videos of remote participants. To achieve real-time rendering of remote participants, we develop the V-Cube View algorithm, which uses multi-view stereo for more accurate depth estimation and Lumi-Net rendering for better rendering quality. The VirtualCube system correctly preserves the mutual eye gaze between participants, allowing them to establish eye contact and be aware of who is visually paying attention to them. The system also allows a participant to have side discussions with remote participants as if they were in the same room. Finally, the system sheds lights on how to support the shared space of work items (e.g., documents and applications) and track the visual attention of participants to work items.
... In [16][17][18], a ToF sensor and the classical stereovision method are combined to complement or speed up the results. Other hybrid systems [19,20] generate an improved free-viewpoint video in terms of accuracy and computational time by using both a depth camera and regular cameras: the depth camera to obtain an initial estimate of the view-dependent geometry and the regular cameras to refine the geometry. On the other hand, hull-or template-based concepts, which are effective in generating a single object, have been proposed in various forms. ...
Hybrid concepts are often used to improve existing methods in many fields. We developed a hybrid optical system that consists of multiple color cameras and one depth camera to make up the concavity problem of the visual hull construction. The heterogeneous data from the color cameras and the depth camera is fused in an effective way. The experimental results show that the proposed hybrid system can reconstruct concave objects successfully by combining the visual hull and the depth data.
... The concept of a hybrid camera was developed by Sawhney et al. [37], and a hybrid imaging system with a low-resolution camera and a high-resolution camera was used to generate highresolution stereoscopic images. After that, the hybrid systems comprising different kinds of cameras can be used to deblur the blurred images [38][39][40][41], produce the high-resolution hyper-spectral images [42], generate high-resolution multi-spectral videos [43], etc. Tola et al. [44] combined a time-offlight depth camera and four regular cameras, which record videos, into a hybrid camera array to significantly improve the quality of rendering virtual views. Lu et al. [45] enhanced the resolution of 3D images from a microscope, through integrating a Shack-Hartmann sensor [46] and a high-resolution CCD camera into a light field hybrid imaging system. ...
Recently, light field cameras have drawn much attraction for their innovative performance in photographic and scientific applications. However, narrow baselines and constrained spatial resolution of current light field cameras impose restrictions on their usability. Therefore, we design a hybrid imaging system containing a light field camera and a high-resolution digital single lens reflex camera, and these two kinds of cameras share the same optical path with a beam splitter so as to achieve the reconstruction of high-resolution light fields. The high-resolution 4D light fields are reconstructed with a phase-based perspective variation strategy. First, we apply complex steerable pyramid decomposition on the high-resolution image from the digital single lens reflex camera. Then, we perform phase-based perspective-shift processing with the disparity value, which is extracted from the upsampled light field depth map, to create high-resolution synthetic light field images. High-resolution digital refocused images and high-resolution depth maps can be generated in this way. Furthermore, controlling the magnitude of the perspective shift enables us to change the depth of field rendering in the digital refocused images. We show several experimental results to demonstrate the effectiveness of our approach.
... Cao et al. [9] used a co-located RBG video camera and LR multi-spectral camera to produce HR multispectral video. Another example of hybrid imaging system is the virtual view synthesis system proposed by Tola et al. [36], where four regular video cameras and a time-of-flight sensor is used. They show that by adding the time-of-flight camera they could render better quality virtual views than just using camera array with similar sparsity. ...
Current light field (LF) cameras provide low spatial resolution and limited depth-of-field (DOF) control when compared to traditional digital SLR (DSLR) cameras. We show that a hybrid imaging system consisting of a standard LF camera and a high-resolution standard camera enables (a) achieve high-resolution digital refocusing, (b) better DOF control than LF cameras, and (c) render graceful high-resolution viewpoint variations, all of which were previously unachievable. We propose a simple patch-based algorithm to super-resolve the low-resolution views of the light field using the high-resolution patches captured using a high-resolution SLR camera. The algorithm does not require the LF camera and the DSLR to be co-located or for any calibration information regarding the two imaging systems. We build an example prototype using a Lytro camera (380×380 pixel spatial resolution) and a 18 megapixel (MP) Canon DSLR camera to generate a light field with 11 MP resolution (9× super-resolution) and about 1 over 9 th of the DOF of the Lytro camera. We show several experimental results on challenging scenes containing occlusions, specularities and complex non-lambertian materials, demonstrating the effectiveness of our approach.
... In [16][17][18], a ToF sensor and the classical stereovision method are combined to complement or speed up the results. Other hybrid systems [19,20] generate an improved free-viewpoint video in terms of accuracy and computational time by using both a depth camera and regular cameras: the depth camera to obtain an initial estimate of the view-dependent geometry and the regular cameras to refine the geometry. On the other hand, hull-or template-based concepts, which are effective in generating a single object, have been proposed in various forms. ...
Hybrid concepts are often used to improve existing methods in many fields. We developed a hybrid optical system that consists of multiple color cameras and one depth camera to make up the concavity problem of the visual hull construction. The heterogeneous data from the color cameras and the depth camera is fused in an effective way. The experimental results show that the proposed hybrid system can reconstruct concave objects successfully by combining the visual hull and the depth data.
... Tola et. al [8] capture 3D video using a ToF sensor and registered cameras, by generating a 3D mesh based on the depth values from the depth sensor. The images from the camera are then used as textures for the 3D mesh. ...
This paper describes the computation of depth maps for a high-quality reference camera augmented by a set of satellite sensors. The satellite sensors include support cameras, a TOF (time-of-flight) sensor, and a thermal camera, all rigidly attached to the reference camera. There is extensive previous work on computing depth maps with stereo alone, and high-quality results have been achieved. However it has proved difficult to achieve good results for cases such as texture less areas, or similar fore- and background colors. We show that with our proposed sensor fusion we can achieve high quality results. The paper makes two contributions. The first is a method for combining TOF data with multi-camera data that includes reasoning about occlusions, to produce an improved depth estimate near depth discontinuities. The second contribution is to show the benefit of thermal sensing as a segmentation prior. Thermal cameras were formerly high-cost devices but are now available at the same cost as machine vision cameras. This work demonstrates their advantages, particularly for scenes including humans.
The VirtualCube system is a 3D video conference system that attempts to overcome some limitations of conventional technologies. The key ingredient is VirtualCube, an abstract representation of a real-world cubicle instrumented with RGBD cameras for capturing the user's 3D geometry and texture. We design VirtualCube so that the task of data capturing is standardized and significantly simplified, and everything can be built using off-the-shelf hardware. We use VirtualCubes as the basic building blocks of a virtual conferencing environment, and we provide each VirtualCube user with a surrounding display showing life-size videos of remote participants. To achieve real-time rendering of remote participants, we develop the V-Cube View algorithm, which uses multi-view stereo for more accurate depth estimation and Lumi-Net rendering for better rendering quality. The VirtualCube system correctly preserves the mutual eye gaze between participants, allowing them to establish eye contact and be aware of who is visually paying attention to them. The system also allows a participant to have side discussions with remote participants as if they were in the same room. Finally, the system sheds lights on how to support the shared space of work items (e.g., documents and applications) and track participants' visual attention to work items.
Light fields suffer from a fundamental resolution tradeoff between the angular and the spatial domain. In this paper, we present a novel cross-scale light field super-resolution approach (up to
$\text{8}\times$
resolution gap) to super-resolve low-resolution (LR) light field images that are arranged around a high-resolution (HR) reference image. To bridge the enormous resolution gap between the cross-scale inputs, we introduce an intermediate view denoted as single image super-resolution (SISR) image, i.e., super-resolving LR input via single image based super-resolution scheme, which owns identical resolution as HR image yet lacks high-frequency details that SISR scheme cannot recover under such significant resolution gap. By treating the intermediate SISR image as the low-frequency part of our desired HR image, the remaining issue of recovering high-frequency components can be effectively solved by the proposed high-frequency compensation super-resolution (HCSR) method. Essentially, HCSR works by transferring as much as possible the high-frequency details from the HR reference view to the LR light field image views. Moreover, to solve the nontrivial warping problem that induced by the significant resolution gaps between the cross-scale inputs, we compute multiple disparity maps from the reference image to all the LR light field images, followed by a blending strategy to fuse for a refined disparity map; finally, a high-quality super-resolved light field can be obtained. The superiority of our proposed HCSR method is validated on extensive datasets including synthetic, real-world and challenging microscope scenes.
This paper addresses the problem of generating a super-resolution (SR) image from a low-resolution (LR) image assisted by nearby high-resolution (HR) image. The scaling factor of the super-resolved image is up to 8 times and even more, which is much larger than the ordinary super-resolution scaling factor. Combined patch match and learning based method for image super-resolution using a cross-resolution input. The method is used to super-resolve the images captured by a hybrid light field system consisting of a standard LF camera and a HR DSLR camera. We take the central high-resolution image as a reference, to deal with the around low-resolution images. Unlike other relative algorithm, the proposed method is exploited for the existence of large parallax between the captured images. The main process of our method is a combination of patch based (i.e., example based) algorithm and learning based (e.g., convolutional neural network) method, and does not require any calibration information. Experimental results show that our proposed method performs better than existing method on challenging scenes containing complex texture, specularity and large parallax. Both accuracy and visual improvements in our results are noticeable.
