Color images taken by a color camera versus NIR images taken by the present NIR imaging system. While unfavorable lighting is obvious in the color face images, it is almost unseen in the NIR face images. 

Context in source publication

Context 1

... spectral measurements of facial skin sampled at some facial points are used for face recognition; they are shown to differ significantly from person to person. Further investigations of using NIR images for face localization and recognition are found in [10], [44], [45]. All of those works use the “bright pupil” effect, namely, specular reflection of active NIR lights on pupils to detect eyes in NIR images. In addition, Zhao and Grigat’s system [44] uses DCT coefficients as features and an SVM as the classifier. Zou et al.’s work [45] derives their matching methods based on an LDA transform and shows that the NIR illuminated faces are better separable than faces under varying ambient illumination. The method of using the “bright pupil” effect to detect eyes has a serious drawback that limits its applications. It assumes that the “bright pupils” are present in the eyes and can be detected using an algorithm. However, the assumption can be invalidated, for example, when there are specular reflections of NIR lights on the eyeglasses, with “narrow eyes” where eyelids may occlude “bright pupils,” when an eye is closed, or when the eyes are looking aside (see examples in Fig. 7). These happen for most of the face images with eyeglasses and also for a significant percentage of face images without eyeglasses. Therefore, we have considered eye detection a main problem to be solved in face recognition using active NIR images. The goal of making this special-purpose hardware is to overcome the problem arising from uncontrolled environmental lights so as to produce face images of a good illumination condition for face recognition. By “a good illumination condition,” we mean that the lighting is from the frontal direction and the image has suitable pixel intensities, i.e., having good contrast and not saturated. We propose two strategies to control the light direction: 1) Mount active lights on the camera to provide frontal lighting and 2) minimize environmental lighting. We set two requirements on the active lighting: 1) The lights should be strong enough to produce a clear frontal-lighted face image without causing disturbance to human eyes and 2) the minimization of the environmental lighting should have minimum reduction of the intended active lighting. Radiation spectrum ranges are shown in Fig. 1. While far (thermal) infrared imaging reflects heat radiation, NIR imaging is more like normal visible light imaging, though NIR is invisible to the naked eyes. Ultraviolet radiation is harmful to the human body and cannot be used for face recognition applications. Our solution for requirement 1, is to choose the active lights in the near infrared (NIR) spectrum between 780-1,100 nm and mount them on the camera. We use NIR light- emitting diodes (LEDs) as active radiation sources, which are strong enough for indoor use and are power-effective. A convenient wavelength is 850 nm. Such NIR lights are almost invisible to the human eye, yet most CCD and CMOS sensors have sufficient response at this spectrum point. When mounted on the camera, the LEDs are approximately coaxial to the camera direction and, thus, provide the best possible straight frontal lighting, better than mounting anywhere else; moreover, when the LEDs and camera are together, control of the lights can be easier using a circuit in the box. The geometric layout of the LEDs on the camera panel may be carefully designed such that the illumination on the face is as homogeneous as possible. The strength of the total LED lighting should be such that it results in the NIR face images with good S/N ratio when the camera-face distance is between 50-100 cm, a convenient range for the user. A guideline is that it should be as strong as possible, at least stronger than expected environmental illumination, yet not cause sensor saturation. A concern is the safety of human eyes. When the sensor working in the normal mode is not saturated, the safety is guaranteed. Our solution for requirement 2, above is to use a long pass optical filter to cut off visible light while allowing NIR light to pass. We choose a filter such that ray passing rates are 0, 50, 88, and 99 percent at the wavelength points of 720, 800, 850, and 880 nm, respectively. The filter cuts off visible environmental lights (< 700 nm) while allowing most of the 850nm NIR light to pass. Fig. 2 illustrates a design of the hardware device and its relationship with the face. The device consists of 18 NIR LEDs, an NIR camera, a color camera, and the box. The NIR LEDs and camera are for NIR face image acquisition. The color camera capture color face images may be used for fusion with the NIR images or for other purposes. The hardware and the face are relatively positioned in such a way that the lighting is frontal and NIR rays provide nearly homogenous illumination on face. The imaging hardware works at a rate of 30 frames per second with the USB 2.0 protocol for 640 Â 480 images and costs less than 20 US dollars. Fig. 3 shows example images of a face illuminated by NIR LED lights from the front, a lamp aside and environmental lights. We can see the following: 1) The lighting conditions are likely to cause problems for face recognition with the color images. 2) The NIR images, with the visible light composition cut off by the filter, are mostly frontal-lighted by the NIR lights, with minimum influence from the side lighting, and provide a good basis for face recognition. In visible light images, an intrapersonal change due to different lighting directions could be larger than an extrapersonal change under similar lighting conditions. This can be illustrated by an analysis on correlation coefficients and matching scores, shown in Fig. 4. There are seven pairs of face images; each pair is taken of the same person’s face but illuminated by a visible light lamp from left and right, respectively. The correlation table shows intrapersonal and extrapersonal correlation coefficients. There, the diagonal entries (in bold font) are for the intrapersonal pairs (e.g., entry ð i; i Þ is the correlation between the two images in column i ); the upper triangles are for the extrapersonal right-right pairs (e.g., entry ð i; j Þ is the correlation between the two images in column i and j in the first row); the lower triangle entries are for the extrapersonal left-left pairs (e.g., entry ð i; j Þ is the correlation between the two images in column i and j in the second row). We see that the correlation coefficients between two images of faces illuminated from left and right are all negative numbers regardless of the face identity; those between two images under similar lighting directions can be either positive or negative. The mean and variance are -0.4738 and 0.1015 for the intrapersonal pairs and 0.0327 and 0.5932 for the extrapersonal pairs. Therefore, it is not a surprise that a PCA matching engine has no chance of making correct matches in this case. The score table is produced by an advanced matching engine trained using AdaBoost with LBP features on a visible light face image training set (the AdaBoost-trained matching engine produces better results than trained using LDA). The mean and variance of the scores are 0.4808 and 0.0238 for intrapersonal pairs and 0.5694 and 0.0359 for extrapersonal pairs. We see that the scores for intrapersonal pairs under different lighting directions are generally lower than those for extrapersonal pairs under similar lighting directions. This means that reliable recognition cannot be achieved with visible light images, even using the advanced matching engine. The impact of environmental lighting is much reduced by the present NIR imaging system, as shown by the correlations and scores in Fig. 5. There, the corresponding NIR face images are taken in the same visible light conditions as in Fig. 4 and the explanations of the two tables are similar to those for Fig. 4. The correlation coefficients between NIR images are all positive, regardless of the visible light conditions and person identity. They have mean and variance of 0.8785 and 0.1502 for intrapersonal pairs and 0.6806 and 0.0830 for extrapersonal pairs. However, the intrapersonal correlation coefficients may not necessarily be higher than the extrapersonal ones, meaning possible recognition errors, even with the NIR images. Therefore, a better matching engine than correlation or PCA is still needed for highly accurate face recognition, even with NIR face images. The score table shows the matching score produced by an LBP+AdaBoost classifier trained on active NIR images. The mean and variance of the scores are 0.6750 and 0.0296 for intrapersonal pairs and 0.3113 and 0.0358 for extrapersonal pairs. By examining all of the entries, we can see that the intrapersonal scores are consistently much higher than the extrapersonal ones. The above case studies suggest that the proposed active NIR imaging system with an advanced LBP+AdaBoost recognition engine can yield the highest recognition performance of all of the schemes. In this section, we first provide an analysis from the Lambertian imaging model to show that such images contain the most relevant, intrinsic information about a face, subject only to a multiplying constant or a monotonic transform due to lighting intensity changes. We then present an LBP-based representation to amend the degree of freedom of the monotonic transform to achieve an illumination invariant representation of faces for indoor face recognition applications. According to the Lambertian model, an image I ð x; y Þ under a point light source is formed according to the following: where ð x; y Þ is the albedo of the facial surface material at point ð x; y Þ , n 1⁄4 ð n x ; n y ; n z Þ is the surface normal (a unit row vector) in the 3D space, and s 1⁄4 ð s x ; s y ; s z Þ is the lighting direction (a column vector, with magnitude). Here, albedo ð x; y Þ reflects the photometric properties of facial skin and hairs; n ð ...