Sample light probes. Left and mid- dle: Real light probes captured from St. Peter’s Cathedral and Uffizi Gallery, courtesy of Paul Debevec (from debevec.org ). Right: Syn- thetic light probe created by the artist Crinity. 

Contexts in source publication

Context 1

... Light Fields (SLFs) [14, 21] are image-based representations of lighting for the capture and display of complex, view-dependent illumination of real-world objects. SLFs are constructed by capturing a set of images from different positions around an object. These images are projected onto the surface of a known geometric model and compressed [3]. This parameterization results in a compact representation that can be rendered at interactive rates. A SLF is a 4D functon that represents the exitant radi- ance under fixed illumination conditions. However, it can only represent the lighting of the environment where the model was captured. This is problematic for synthetic environments such as games or virtual environments in which a specific illumination environment is desired. These virtual illumination environments could come from light probes captured in real locations, or from synthetic lighting environments created by artists. Examples of lighting environments are shown in Figure 2. To achieve correct object appearance the lighting environments must be physically duplicated in the lab at the time of capture, which can be difficult. One approach is to use lights or projectors that are physically situated to mimic the virtual lighting positions and colors. This approach is constrained to the resolution of the physical lights, and is time-consuming to construct. Another approach is to collect the full 6D Bidirectional Texture Function, which enables the object to be rendered under arbitrary lighting conditions. This requires a significant increase in the amount of data that is acquired, most of which is unnecessary if the lighting environment is already known. In this paper we describe a third approach: an efficient method for capturing a surface light field using the virtual illumination from an environment map. We use a simple setup consisting of a projector, a camera, tracking fiducials, and a pan-tilt unit to recreate the desired lighting environment. To decrease noise and improve the quality of the capture under low- and high-dynamic range environment maps, we use an extended version of the multiplexed illumination algorithm [19]. This results in a high-dynamic range SLF which accurately represents the interaction of the virtual illumination with the real object. Two examples of objects embedded into a virtual light environment are shown in Figure 1. The remainder of the paper is organized as follows. The next section describes related work in the areas of computer vision and computer graphics. Section 3 gives an overview of our virtual illumination system. Both the geometric and photometric calibration of the system are discussed in Section 4. A method for handling multiple cameras is described in Section 5. Section 6 introduces multiplexed illumination and the extension to high-dynamic range. This is followed by results for objects under low- and high-dynamic range illumination. In this section we briefly review related work in the areas of high-dynamic range illumination, the sampling of mate- rial properties from objects, and instruments similar to ours. We then give an overview of multiplexed illumination. Since the range of illumination present in the world is much larger than the range that can be reproduced by displays or captured by cameras, we need to use high-dynamic range (HDR) imaging techniques. Research in this area was pioneered by Debevec [9] in a paper that described how to linearize the response of cameras and combine multiple exposures into a single HDR image. Debevec also describes a technique for illuminating synthetic objects under HDR illumination [9]. A recent book serves as an excellent reference to the body of work surrounding HDR imaging [18]. Examples of HDR light probes are shown in Figure 2. There has been somewhat less work on the lighting of real objects under virtual, user-specified illumination. One way to do this is to capture the BRDF of the materials. A survey of this work was presented in [15]. Many of these ap- proaches attempt to reconstruct a 6-dimensional (or higher) function, which requires complicated equipment and con- siderable time to sample. In this paper we capture the 4D surface light field [21], which naturally takes fewer samples to estimate. Instruments designed to capture BRDFs and reflectance fields of objects can collect surface light fields also. Our system could also be used to capture BRDF and reflectance fields. The light stage presented in [8] used several cameras and a lamp mounted to a gantry that could move the lamp over the hemisphere around a person. The main advantage to the system we describe is cost; we estimate that a total of $4000 is enough to buy the required equipment, although in fact most labs already have this gear. Furthermore, our camera-based calibration does not require precise positioning, so we are able to use inexpensive tripods to hold the camera and the pan-tilt motor. The system most similar to ours is that of Masselus, et al. [12]. They used two plasma panels, six cameras, and four Halogen lamps in their instrument. They fixed a camera to a turntable with an object at the center, and rotated both under a projector. Thus they were able to capture the response to a light field for a single view and then relight the object. The main limitation is that the object to camera relationship was fixed. They also lit the object directly from the projector, thus obtaining a very different set of illumination rays than those obtained by our system. Capturing images under dim lighting is difficult due to the presence of camera CCD noise. This noise can significantly degrade the quality of the image due to the low signal-to-noise ratio. Schechner et al. [19] introduced a technique to significantly reduce the noise in the captured images with multiple low intensity light sources. Using n light sources, we can increase the signal-to-noise ratio by up √ n to with the same number of images. We briefly describe 2 the Multiplexed Illumination algorithm in order to provide background on our HDR approach. Consider the problem of acquiring images of an object that are lit from a set of light sources. A reasonable approach would be to acquire one image of the object for each light source. For n light sources, this means that each image receives only 1 /n -th of the total available light. In addition, when each of the light sources are dim, the noise from the CCD cameras can corrupt the images. Consequently, for many of the light probes that we use, the signal-to-noise ratio (SNR) of the captured images is very low. Multiplexed illumination is a technique to improve the SNR of the images. Each image is captured using multiple lights, and a post-process is performed to demultiplex the contribution of each individual light source. The light are additive quantities and are linearly related by superposition where a ζ k ( x, y ) is the light observed at pixel ( x, y ) under the set of lights ζ k and i l ( x, y ) is the energy contributed by light source l at pixel ( x, y ) . The multiplexing matrix W for the light sources l = 1 , . . . , m describes which light sources illuminate the scene. An element W i,j is one if the light j is illuminated in image i , and zero if the light was “off”. The sets ζ k consists of all of the lights in row k which are “on”. In order to recover the images i l ( x, y ) as lit under a single light source l , we demultiplex the observed images a ( x, y ) by inverting the matrix W : It’s important to note that multiplexed illumination does not require taking any more images than single light illumination. The only added computation is the post-process demultiplexing step. Our proposed method to capture a SLF under virtual lighting requires modest equipment and infrastructure. A diagram of our setup is shown in Figure 3. Light is projected onto a screen and reflected onto the object, which mimics the light from the light probe falling onto the object. In this way the projector and screen act as large area programmable light source. This is in contrast to a system which directly illuminates the object with the projector, as such a system would only represent a small portion of the lighting environment. This is because the projector essen- tially becomes a point light source, as shown in Figure 3. In addition, this configuration would require a rig to move the projector around the object to approximate the desired spherical lighting environment [12]. A photograph of our laboratory setup in shown in Figure 4. While the screen covers a large area of the lighting environment, it does not cover the entire environment. In order to recreate a given virtual lighting environment, the light source must illuminate the object from every ...

Context 2

... of Masselus, et al. [12]. They used two plasma panels, six cameras, and four Halogen lamps in their instrument. They fixed a camera to a turntable with an object at the center, and rotated both under a projector. Thus they were able to capture the response to a light field for a single view and then relight the object. The main limitation is that the object to camera relationship was fixed. They also lit the object directly from the projector, thus obtaining a very different set of illumination rays than those obtained by our system. Capturing images under dim lighting is difficult due to the presence of camera CCD noise. This noise can significantly degrade the quality of the image due to the low signal-to-noise ratio. Schechner et al. [19] introduced a technique to significantly reduce the noise in the captured images with multiple low intensity light sources. Using n light sources, we can increase the signal-to-noise ratio by up √ n to with the same number of images. We briefly describe 2 the Multiplexed Illumination algorithm in order to provide background on our HDR approach. Consider the problem of acquiring images of an object that are lit from a set of light sources. A reasonable approach would be to acquire one image of the object for each light source. For n light sources, this means that each image receives only 1 /n -th of the total available light. In addition, when each of the light sources are dim, the noise from the CCD cameras can corrupt the images. Consequently, for many of the light probes that we use, the signal-to-noise ratio (SNR) of the captured images is very low. Multiplexed illumination is a technique to improve the SNR of the images. Each image is captured using multiple lights, and a post-process is performed to demultiplex the contribution of each individual light source. The light are additive quantities and are linearly related by superposition where a ζ k ( x, y ) is the light observed at pixel ( x, y ) under the set of lights ζ k and i l ( x, y ) is the energy contributed by light source l at pixel ( x, y ) . The multiplexing matrix W for the light sources l = 1 , . . . , m describes which light sources illuminate the scene. An element W i,j is one if the light j is illuminated in image i , and zero if the light was “off”. The sets ζ k consists of all of the lights in row k which are “on”. In order to recover the images i l ( x, y ) as lit under a single light source l , we demultiplex the observed images a ( x, y ) by inverting the matrix W : It’s important to note that multiplexed illumination does not require taking any more images than single light illumination. The only added computation is the post-process demultiplexing step. Our proposed method to capture a SLF under virtual lighting requires modest equipment and infrastructure. A diagram of our setup is shown in Figure 3. Light is projected onto a screen and reflected onto the object, which mimics the light from the light probe falling onto the object. In this way the projector and screen act as large area programmable light source. This is in contrast to a system which directly illuminates the object with the projector, as such a system would only represent a small portion of the lighting environment. This is because the projector essen- tially becomes a point light source, as shown in Figure 3. In addition, this configuration would require a rig to move the projector around the object to approximate the desired spherical lighting environment [12]. A photograph of our laboratory setup in shown in Figure 4. While the screen covers a large area of the lighting environment, it does not cover the entire environment. In order to recreate a given virtual lighting environment, the light source must illuminate the object from every possible direction. This would require moving the projector, screen and the camera around the object as we acquire images. However, instead of moving the camera as in similar sys- tems [5], we move the object itself using a programmable pan/tilt unit. Moving the object simplifies the setup since the screen and projector can be fixed in place and calibrated once. A static setup of camera and screen also simplifies the problem of the camera (and user) occluding the light. As the object rotates and tilts, the corresponding portion of the lighting environment changes. From the point-of- view of the object, it is as if a “window” is moving around it, allowing light from the environment to hit the object. In order to maintain a fixed orientation in the fixed lighting environment, we need to track the object’s orientation relative to the camera. This information is also needed to correctly composit the images together. We use fiducials to track the orientation, which can be seen in Figure 4. The lighting environment can be arbitrarily complex, and can be either fully synthetic or captured from a real scene as a light probe [9]. Examples of light probes that we used in our experiments are shown in Figure 2. Since these light probes often have dramatic contrast between the dark- est and brightest areas, we developed a novel approach to create a high dynamic lighting environment using a low dynamic range camera and a low dynamic range projector. Our technique is an extension of Multiplexed Illumination [19], which was developed in the computer vision com- munity to reduce the noise in acquired images. We describe this technique in Section 6, after we discuss the calibration and registration of the system. To successfully illuminate a real object with a virtual lighting environment, the mapping between the physical setup and the virtual lighting environment must be determined. This requires calibrating two transformations; first, the static transformation between light rays that are emitted from the projector and reflected off the screen onto the lighting stage. The second transformation is a dynamic correspondence between the camera and the object on the pan-tilt unit. In this section, we describe the first correspondence, which depends mainly on the projector characteristics and the screen geometry. The result of this calibration is a mapping between pixels on the screen and points on the surface of the object. This mapping is used to determine how to display the lighting environment on the screen to achieve correct illumination in the desired lighting environment, as shown in Figure 5. The calibration of the system registers the virtual lighting environment to the real physical object. This involves four components: the camera, the tracking fiducials, the physical object, and the projected image on the screen. The camera is fixed in relation to the screen, and the fiducials are fixed in relation to the object. One of the challenges of this calibration method is that we would like to be able to use arbitrary screen geometries. For example, consider projecting images into the corner of a room. Since the screen covers a larger solid angle above the object, we can reduce the number of cameras required to capture the full hemisphere of incident light. These considerations led us to look for a general calibration method that does not make assumptions about the geometry of the projected image. The general screen geometry poses a significant chal- lenge for the calibration process. Further problems arise from the deviation of the projector from an ideal projection due to aperture and lens distortions. These factors im- ply that physically measuring the system is difficult and often inaccurate. To account for all these effects we need to choose a fully automatic calibration technique, as po- tentially every ray illuminating the object needs to be calibrated separately. Our method addresses these consideration by using a reflective sphere to register the pixels from the projector with a set of tracking fiducials from ARToolkit [11]. This procedure takes advantage of the property that light probes are independent of translation. Thus the calibration procedure needs to only compute the rays emanating from the object, and does not need to compute the translation information. This simplifies the calibration procedure to one of deter- mining the relation between a pixel on the screen and a ray in object space. We place a mirrored sphere with a measured radius in approximately the same location as the object. This mirrored sphere reflects the illuminated points on the screen back to the camera. Using the known projector pixel and the reflection of this point into the camera, the ray associated with each projector pixel can be computed. This is shown in Figure 6. The process works as follows. A small block of pixels is projected onto the screen, which reflects off the mirrored sphere and back to the camera. Using the position of the pixels in the image, the ray through the camera plane into the scene can be computed from the pose of the tracking board. This ray is then traced into the scene and intersected with the sphere. At the intersection point, the normal is computed and a reflected ray is generated. This reflected ray is the ray in world space that corresponds to the projector pixel. This automatically establishes a correspondence between projector pixels and rays in the scene. For rendering, these rays are rotated according to the delta rotation of the tracking board, and used to index into the environment map. The resulting images for the projector are shown in Figure 6. Alternatively a more advanced structured light pattern can be used to determine all correspondences from very few images. One of the sources of error in our setup is the color mapping of the projector, which is composed of aperture, lens, and CCD mapping of the colors. It is not correct to assume that the light coming from the projector is linearly corre- lated with the values that are sent, even with the gamma set to 1.0 and the controls adjusted. This is important because both the HDR exposures and the Multiplexed Illumination assume ...

Context 3

... Light Fields (SLFs) [14, 21] are image-based representations of lighting for the capture and display of complex, view-dependent illumination of real-world objects. SLFs are constructed by capturing a set of images from different positions around an object. These images are projected onto the surface of a known geometric model and compressed [3]. This parameterization results in a compact representation that can be rendered at interactive rates. A SLF is a 4D functon that represents the exitant radi- ance under fixed illumination conditions. However, it can only represent the lighting of the environment where the model was captured. This is problematic for synthetic environments such as games or virtual environments in which a specific illumination environment is desired. These virtual illumination environments could come from light probes captured in real locations, or from synthetic lighting environments created by artists. Examples of lighting environments are shown in Figure 2. To achieve correct object appearance the lighting environments must be physically duplicated in the lab at the time of capture, which can be difficult. One approach is to use lights or projectors that are physically situated to mimic the virtual lighting positions and colors. This approach is constrained to the resolution of the physical lights, and is time-consuming to construct. Another approach is to collect the full 6D Bidirectional Texture Function, which enables the object to be rendered under arbitrary lighting conditions. This requires a significant increase in the amount of data that is acquired, most of which is unnecessary if the lighting environment is already known. In this paper we describe a third approach: an efficient method for capturing a surface light field using the virtual illumination from an environment map. We use a simple setup consisting of a projector, a camera, tracking fiducials, and a pan-tilt unit to recreate the desired lighting environment. To decrease noise and improve the quality of the capture under low- and high-dynamic range environment maps, we use an extended version of the multiplexed illumination algorithm [19]. This results in a high-dynamic range SLF which accurately represents the interaction of the virtual illumination with the real object. Two examples of objects embedded into a virtual light environment are shown in Figure 1. The remainder of the paper is organized as follows. The next section describes related work in the areas of computer vision and computer graphics. Section 3 gives an overview of our virtual illumination system. Both the geometric and photometric calibration of the system are discussed in Section 4. A method for handling multiple cameras is described in Section 5. Section 6 introduces multiplexed illumination and the extension to high-dynamic range. This is followed by results for objects under low- and high-dynamic range illumination. In this section we briefly review related work in the areas of high-dynamic range illumination, the sampling of mate- rial properties from objects, and instruments similar to ours. We then give an overview of multiplexed illumination. Since the range of illumination present in the world is much larger than the range that can be reproduced by displays or captured by cameras, we need to use high-dynamic range (HDR) imaging techniques. Research in this area was pioneered by Debevec [9] in a paper that described how to linearize the response of cameras and combine multiple exposures into a single HDR image. Debevec also describes a technique for illuminating synthetic objects under HDR illumination [9]. A recent book serves as an excellent reference to the body of work surrounding HDR imaging [18]. Examples of HDR light probes are shown in Figure 2. There has been somewhat less work on the lighting of real objects under virtual, user-specified illumination. One way to do this is to capture the BRDF of the materials. A survey of this work was presented in [15]. Many of these ap- proaches attempt to reconstruct a 6-dimensional (or higher) function, which requires complicated equipment and con- siderable time to sample. In this paper we capture the 4D surface light field [21], which naturally takes fewer samples to estimate. Instruments designed to capture BRDFs and reflectance fields of objects can collect surface light fields also. Our system could also be used to capture BRDF and reflectance fields. The light stage presented in [8] used several cameras and a lamp mounted to a gantry that could move the lamp over the hemisphere around a person. The main advantage to the system we describe is cost; we estimate that a total of $4000 is enough to buy the required equipment, although in fact most labs already have this gear. Furthermore, our camera-based calibration does not require precise positioning, so we are able to use inexpensive tripods to hold the camera and the pan-tilt motor. The system most similar to ours is that of Masselus, et al. [12]. They used two plasma panels, six cameras, and four Halogen lamps in their instrument. They fixed a camera to a turntable with an object at the center, and rotated both under a projector. Thus they were able to capture the response to a light field for a single view and then relight the object. The main limitation is that the object to camera relationship was fixed. They also lit the object directly from the projector, thus obtaining a very different set of illumination rays than those obtained by our system. Capturing images under dim lighting is difficult due to the presence of camera CCD noise. This noise can significantly degrade the quality of the image due to the low signal-to-noise ratio. Schechner et al. [19] introduced a technique to significantly reduce the noise in the captured images with multiple low intensity light sources. Using n light sources, we can increase the signal-to-noise ratio by up √ n to with the same number of images. We briefly ...