Figure 1 - uploaded by Tinsae Gebrechristos Dulecha
Content may be subject to copyright.
Objects. Original images of the objects chosen for relighting test. First Row: two metallic coins (Coin1,Coin2). Second Row: an ancient gold lamina (Lamina), and an imprint of a shell (Shell).
Source publication
Reflectance Transformation Imaging (RTI) is widely used to produce relightable models from multi-light image collections. These models are used for a variety of tasks in the Cultural Heritage field. In this work, we carry out an objective and subjective evaluation of RTI data visualization. We start from the acquisition of a series of objects with...
Contexts in source publication
Context 1
... order to evaluate the quality of the chosen relighting techniques, we consider RTI acquisitions of four objects made of materials that are relevant for the Cultural Heritage domain: two coins made of one and two different metal alloys, an ancient gold lamina, and the imprint of a shell (see Figure 1). In general, we have cho- sen those objects because they all exhibit different micro/meso- geometry, and different optical material behaviors (both specular and diffuse response). ...
Context 2
... the objective evaluation we have witnessed how PTM is the technique that behaves poorly across datasets compared with the other two; HSH improves a little, by trying to render better non- diffuse material properties, while RBF on average wins over both other techniques. Now, we are interested in understanding if this Figure 10. While scores in Test 1 (ground truth comparison) are similar between experts and non-experts, Test 2 produces non consistent results between the two groups. ...
Context 3
... in the Test 2, the CH people tend to increase the preference on a smoothed PTM signal, which, although it fails to render high-frequency material behavior, it con- veys a cleaner (and readable) surface visualization. An example of this statistical behavior is given in Figure 10 and Figure 11. There we show only one image of the dataset Lamina and we present the original photo, and the PTM, HSH, and RBF rendering. ...
Context 4
... in the Test 2, the CH people tend to increase the preference on a smoothed PTM signal, which, although it fails to render high-frequency material behavior, it con- veys a cleaner (and readable) surface visualization. An example of this statistical behavior is given in Figure 10 and Figure 11. There we show only one image of the dataset Lamina and we present the original photo, and the PTM, HSH, and RBF rendering. ...
Context 5
... may be caused by large-scale luminance shifts between original and relighted images, compensated by the percep- tive adjustments of the SSIM metric. While the RBF interpolation seems to provide the best similarity with respect to original images both in objective and subjective tests (and 3rd order HSH is clearly superior to second order PTM), CH experts been asked to compare relighted images without reference do not show clear preferences (Figure 7 and 10), showing that non-photorealistic depictions might be accepted if judged more readable. These results lead to the iden- tification of two competing research directions. ...
Citations
... Nevertheless, different methods are proposed for visualisation and data analysis. An extensive state of the art in RTI was presented by Pintus et al. (2019) [15] . ...
Imaging techniques, along with their subsequent processing and analysis, are of utmost importance in the visual documentation of conservation processes of cultural heritage (CH) objects. Amongst them, Reflectance Transformation Imaging (RTI) is being used as a tool for the enhancement of surface topography, such as decorative details. This paper proposes a new approach based on advanced RTI data processing and analysis to document the condition of metal artefacts or monitor their conservation treatments. First, the methodology for mapping geometric and statistical information from the stack of RTI image data and their relation to the surface topography and the appearance attributes resulting in feature maps is described. Additionally, the possibility of quantifying intra- or inter-surface changes based on saliency and distance measurements by applying the Mahalanobis distance (MD) on feature maps is demonstrated. This methodology is then used for documenting the condition and monitoring the cleaning treatment of a late Roman coin.
... RTI was originally developed at HP labs by Malzbender et al. [8,9] under a name, Polynomials Texture Mappings (PTM), that refers to the implemented modeling method. Other modeling methods have been developed, such as the HSH (Hemispherical Harmonics) model [10][11][12][13], the DMD (Discrete Modal Decomposition) method [14][15][16][17], and the approach based on RBF (Radial Basis Function) [18,19]. Particularly important development has occurred in the field of the digitization of historical and cultural heritage objects [20][21][22]. ...
Reflectance Transformation Imaging (RTI) is a non-contact technique which consists in acquiring a set of multi-light images by varying the direction of the illumination source on a scene or a surface. This technique provides access to a wide variety of local surface attributes which describe the angular reflectance of surfaces as well as their local microgeometry (stereo photometric approach). In the context of the inspection of the visual quality of surfaces, an essential issue is to be able to estimate the local visual saliency of the inspected surfaces from the often-voluminous acquired RTI data in order to quantitatively evaluate the local appearance properties of a surface. In this work, a multi-scale and multi-level methodology is proposed and the approach is extended to allow for the global comparison of different surface roughnesses in terms of their visual properties. The methodology is applied on different industrial surfaces, and the results show that the visual saliency maps thus obtained allow an objective quantitative evaluation of the local and global visual properties on the inspected surfaces.
... The core objective of this paper is to evaluate the use of these relightings in an industrial context of inspection and assessment of the visual quality of surfaces, as a tool for computeraided visual inspection, (i.e., to assist the experts and operators of sensory control). Studies in this sense have already been conducted, especially in the field of heritage [25,28], to quantify the performance of the existing RTI reconstruction models in particular [29,30]. This type of analysis evaluates the performance from a quantitative, numerical point of view but does not take into account the aspects related to human perception. ...
... The most commonly used models are global models, such as the historical PTM approach [31,32], the HSH method [45], or the more recent Discrete Modal Decomposition (DMD) [44,47], which we chose to evaluate in this paper. The proposed methodology could, however, be applied in other reconstruction models, such as the Radial Basis functions (RBFs) recently proposed by [28], which are based on local interpolations. For the choice of these parameters having to be correlated with the type of surface to be inspected, three different industrial surface samples were retained. ...
... The main result extracted from the global data presented in Figure 8b for the S 2 sample (equivalent performance of the HSH and DMD approaches) was also confirmed by these consistency data (no significant difference). These perceptual results are consistent with those obtained in previous studies that evaluated the performance of PTM, HSH, and DMD reconstruction models from a numerical quantitative point of view [28,44,47]. Concerning the other RTI factors, such as the scale and the density of the acquisition points, the existing RTI devices often did not allow varying these parameters, and to our knowledge, although their effect is known by the users of the technique [25,41,42], their influence has not been evaluated in previous works. ...
This work investigates the use of Reflectance Transformation Imaging (RTI) rendering for visual inspection. This imaging technique is being used more and more often for the inspection of the visual quality of manufactured surfaces. It allows reconstructing a dynamic virtual rendering of a surface from the acquisition of a sequence of images where only the illumination direction varies. We investigate, through psychometric experimentation, the influence of different essential parameters in the RTI approach, including modeling methods, the number of lighting positions and the measurement scale. In addition, to include the dynamic aspect of perception mechanisms in the methodology, the psychometric experiments are based on a design of experiments approach and conducted on reconstructed visual rendering videos. The proposed methodology is applied to different industrial surfaces. The results show that the RTI approach can be a relevant tool for computer-aided visual inspection. The proposed methodology makes it possible to objectively quantify the influence of RTI acquisition and processing factors on the perception of visual properties, and the results obtained show that their impact in terms of visual perception can be significant.
... To build a continuous model, it's common to use data fitting by approximation [16] or interpolation [17]. Therefore, It is essential to have an efficient approximation model allowing for the purposes of relighting. ...
Reflectance Transformation Imaging (RTI) is a technique for estimating surface local angular reflectance from a set of stereo-photometric images captured with variable lighting directions. The digitization of this information fully fits into the industry 4.0 approach and makes it possible to characterize the visual properties of a surface. The proposed method, namely HD-RTI, is based on the coupling of RTI and HDR imaging techniques. This coupling is carried out adaptively according to the response at each angle of illumination. The proposed method is applied to five industrial samples which have high local variations of reflectivity because of their heterogeneity of geometric texture and/or material. Results show that coupling HDR and RTI improves the relighting quality compared to RTI, and makes the proposed approach particularly relevant for glossy and heterogeneous surfaces. Moreover, HD-RTI enhances significantly the characterization of the local angular reflectance, which leads to more discriminating visual saliency maps, and more generally to an increase in robustness for visual quality assessment tasks.
... It is common to use RTI for texture visualization purposes of paintings [44] and capturing of low-relief surfaces [45]. For example, Pintus et al. [46] performed RTI of cultural heritage data visualization assessment both subjectively and objectively. Recently, Kitanovski et al. [47] assessed the quality of relighting from images acquired through their proposed multispectral RTI system. ...
Quality assessment is an important aspect in a variety of application areas. In this work, the objective quality assessment of 2.5D prints was performed. The work is done on camera captures under both diffuse (single-shot) and directional (multiple-shot) illumination. Current state-of-the-art 2D full-reference image quality metrics were used to predict the quality of 2.5D prints. The results showed that the selected metrics can detect differences between the prints as well as between a print and its 2D reference image. Moreover, the metrics better detected differences in the multiple-shot set-up captures than in the single-shot set-up ones. Although the results are based on a limited number of images, they show existing metrics’ ability to work with 2.5D prints under limited conditions.
... where α i is estimated per pixel by solving a least-squares problem. A recent objective and subjective evaluation have shown, in particular, how RBF interpolation is capable of better visualizing the behavior of complex materials for in-between views with respect to classic fitting approaches [105] (Fig. 2.1). However, the need to access large amounts of data makes the method very computation and memory-intensive. ...
... RealRTI is a dataset made of real MLIC acquisitions, made with devices and protocols typically used in the MLIC acquisition (see section 1.1). This dataset is composed of 12 multi-light image collections (cropped and resized to allow a fast processing/evaluation) acquired with light domes(using a setup of Ciortan et al. [24] and Pintus et al. [105]) or handheld RTI protocols(using a setup of Giachetti et al. [48]) on surfaces with different shape and material complexity. The items imaged are: (1) a wooden painted door (handheld acquisition, 60 light directions), (2) a fresco (dome acquisition, 47 lights), (3,4) two painted icons (handheld 63 and 72 lights), (5,6) two paintings on canvas (handheld, 49 lights and dome, 48 lights), impressions on the plaster of a leaf (7) and a shell (8) (light dome, 48 lights), (9,10) two coins (both with a light dome, 48 lights), (11,12) two metallic statues (dome, 48 lights and handheld, 54 lights). ...
... Relighting based on low-frequency fitting (Sec. 2.3) is by far the most commonly employed visualization, but significant inaccuracies have been shown to exist [105]. Early attempts at using resampling and compression methods to support direct interpolation are promising [116], but this research is in the early stage and only supports aggressive lossy compression. ...
Multi-Light Image Collections (MLICs) are stacks of photos of a scene acquired with a
fixed viewpoint and a varying surface illumination that provides large amounts of visual
and geometric information. Over the last decades, a wide variety of methods have been
devised to extract information from MLICs and have shown its use in different application
domains to support daily activities. In this thesis, we present methods that leverage a
MLICs for surface analysis and visualization.
First, we provide background information: acquisition setup, light calibration and ap-
plication areas where MLICs have been successfully used for the research of daily analysis
work. Following, we discuss the use of MLIC for surface visualization and analysis and
available tools used to support the analysis. Here, we discuss methods that strive to sup-
port the direct exploration of the captured MLIC, methods that generate relightable models
from MLIC, non-photorealistic visualization methods that rely on MLIC, methods that es-
timate normal map from MLIC and we point out visualization tools used to do MLIC
analysis.
In chapter 3 we propose novel benchmark datasets (RealRTI, SynthRTI and SynthPS)
that can be used to evaluate algorithms that rely on MLIC and discusses available bench-
mark for validation of photometric algorithms that can be also used to validate other
MLIC-based algorithms. In chapter 4, we evaluate the performance of different photo-
metric stereo algorithms using SynthPS for cultural heritage applications. RealRTI and
SynthRTI have been used to evaluate the performance of (Neural)RTI method.
Then, in chapter 5, we present a neural network-based RTI method, aka NeuralRTI,
a framework for pixel-based encoding and relighting of RTI data. In this method using a
simple autoencoder architecture, we show that it is possible to obtain a highly compressed
representation that better preserves the original information and provides increased quality
of virtual images relighted from novel directions, particularly in the case of challenging
glossy materials.
Finally, in chapter 6, we present a method for the detection of crack on the surface
of paintings from multi-light image acquisitions and that can be used as well on single
images and conclude our presentation.
... A good RTI encoding should be compact and allow interactive relighting from an arbitrary direction, rendering the correct diffuse and specular behaviors of the imaged materials and limiting interpolation artifacts. These requirements are not always satisfied with the methods currently employed in practical applications [13]. ...
Reflectance transformation imaging (RTI) is a computational photography technique widely used in the cultural heritage and material science domains to characterize relieved surfaces. It basically consists of capturing multiple images from a fixed viewpoint with varying lights. Handling the potentially huge amount of information stored in an RTI acquisition that consists typically of 50–100 RGB values per pixel, allowing data exchange, interactive visualization, and material analysis, is not easy. The solution used in practical applications consists of creating “relightable images” by approximating the pixel information with a function of the light direction, encoded with a small number of parameters. This encoding allows the estimation of images relighted from novel, arbitrary lights, with a quality that, however, is not always satisfactory. In this paper, we present NeuralRTI, a framework for pixel-based encoding and relighting of RTI data. Using a simple autoencoder architecture, we show that it is possible to obtain a highly compressed representation that better preserves the original information and provides increased quality of virtual images relighted from novel directions, especially in the case of challenging glossy materials. We also address the problem of validating the relight quality on different surfaces, proposing a specific benchmark, SynthRTI, including image collections synthetically created with physical-based rendering and featuring objects with different materials and geometric complexity. On this dataset and as well on a collection of real acquisitions performed on heterogeneous surfaces, we demonstrate the advantages of the proposed relightable image encoding.
... A good RTI encoding should be compact and allow interactive relighting from an arbitrary direction, rendering the correct diffuse and specular behaviors of the imaged materials and limiting interpolation artifacts. These requirements are not always satisfied with the methods currently employed in practical applications [13]. ...
Reflectance Transformation Imaging (RTI) is a computational photography technique widely used in the cultural heritage and material science domains to characterize relieved surfaces. It basically consists of capturing multiple images from a fixed viewpoint with varying lights.
Handling the potentially huge amount of information stored in an RTI acquisition, that consists typically of 50-100 RGB values per pixel, allowing data exchange, interactive visualization, and material analysis is not easy.
The solution used in practical applications consists of creating "relightable images" by approximating the pixel information with a function of the light direction, encoded with a small number of parameters. This encoding allows the estimation of images relighted from novel, arbitrary lights, with a quality that, however, is not always satisfactory.
In this paper, we present NeuralRTI, a framework for pixel-based encoding and relighting of RTI data. Using a simple autoencoder architecture, we show that it is possible to obtain a highly compressed representation that better preserves the original information and provides increased quality of virtual images relighted from novel directions, especially in the case of challenging glossy materials.
We also address the problem of validating the relight quality on different surfaces, proposing a specific benchmark, SynthRTI, including image collections synthetically created with physically-based rendering and featuring objects with different materials and geometric complexity.
On this dataset and as well on a collection of real acquisitions performed on heterogeneous surfaces, we demonstrate the advantages of the proposed relightable image encoding.
... In this example, RBF RTI provides the most realistic virtual relighting. Pintus et al. 38 provides an objective and subjective evaluation on the relighting of these RTI methods. The bottom row provides an overview of the recovered surface gradient with the different methods. ...
Multi-light, single-camera imaging techniques like Reflectance Transformation Imaging (RTI, including PTM, HSH, and PCA-RBF) and the Portable Light Dome (PLD) have been used by cultural heritage scholars and collection curators extensively because of the extra interactive visual information that can be revealed on artefacts when compared to standard digital photography. Besides a virtual relighting of the scanned object, these techniques offer filters to accentuate different aspects of the studied surface. The main focus of RTI, developed at HP, CHI and among others elaborated by ISTI CNR, is aimed at photo-realistic virtual relighting. PLD, developed at KU Leuven, on the other hand, is aimed at extracting surface properties such as surface color (albedo), surface gradient (normals), 3D surface features (height profiles) and reflectance distribution (reflectance maps). PLD and RTI both produce interactive pixel based file formats, which are dissimilar, resulting in incompatible datasets. The pixel+ project (Art and History Museum, KU Leuven, and KBR, Belspo BRAIN-be funded) aims to merge both technologies into one web-based consultation platform, allowing existing PLD and RTI datasets to be viewed in one web environment with their respective viewing filters as well as to illuminate the virtual model. Moreover, as both methods are alike in terms of required input and processed output, pixel+ focuses on other types of integration, resulting in new viewing modes for processed data as well as a novel reprocessing pipeline for existing source images. In addition, for sustainable and flexible web consultation a new open format, based on glTF, is suggested and a first elaboration is presented.
... From this set of images, each pixel is associated a set of discrete values (measured gray-levels, considered to be proportional to the luminance [10]). To model the surface visual appearance continuously and to allow relighting the surface for any virtual direction of light, this set of luminance values can be approximated or interpolated locally [4], [11], [12]. The main approximation methods used to model this information are the Polynomial Texture Mappings approach (PTM), based on 2nd order polynomial functions [13], the Hemispherical Harmonics (HSH) approach [14] and the Discrete Modal Decomposition (DMD) [8], [15]- [17]. ...
... The first metric we used for the objective evaluation is the Peak Signal to Noise Ratio (PSNR) since it has already proven its relevance in RTI data. The PSNR has been used to evaluate the efficiency of the fitting models for reflectance reconstruction [8], [12], [15], to evaluate RTI compressed images quality visualised on a mobile device [27], or for web relighting tools [28] In this study, we propose to use this metric in order to to investigate its correlation with subjective assessments of how surfaces are perceived, and therefore evaluate the the quality of the reconstruction models in a perceptual sense [15]. PSNR results are presented in Table II. ...
In this paper, we propose to evaluate the quality of the reconstruction and relighting from images acquired by a Reflectance Transformation Imaging (RTI) device. Three relighting models, namely the PTM, HSH, and DMD, are evaluated using PSNR and SSIM. A visual assessment of how the reconstructed surfaces are perceived is also carried out through a sensory experiment. This study allows estimating the relevance of these models to reproduce the appearance of the manufactured surfaces. It also shows that DMD reproduces the most accurate reconstruction/relighting to an acquired measurement and that a higher sampling density doesn’t mean necessarily a higher
perceptual quality.
