Shadow Remover Image Shadow Removal Based on Illumination Recovering Optimization（word版本） - 图文

Shadow Remover: Image Shadow Removal Based

on Illumination Recovering Optimization

Ling Zhang, Qing Zhang, and Chunxia Xiao, Member, IEEE,

Abstract—In this paper, we present a novel shadow removal system for single natural images as well as color aerial images using an illumination recovering optimization method. We first adaptively decompose the input image into overlapped patches according to the shadow distribution. Then, by building the correspondence between the shadow patch and the lit patch based on texture similarity, we construct an optimized illumination recovering operator, which effectively removes the shadows and recovers the texture detail under the shadow patches. Based on coherent optimization processing among the neighboring patches, we finally produce high-quality shadow-free results with consistent illumination. Our shadow removal system is simple and effective, and can process shadow images with rich texture types and nonuniform shadows. The illumination of shadow-free results is consistent with that of surrounding environment. We further present several shadow editing applications to illustrate the versatility of the proposed method.

Index Terms —Shadow detection, shadow removal, shadow matting, patch matching, aerial images.

I. INTRODUCTION

HADOWS are natural phenomena, which occur when the

light is blocked. Although shadows provide important visual cues for object shape perception, illumination position, objects occlusion, etc., shadow-free images can help to improve the performance of the tasks such as object recognition, object tracking and information enhancement [1], [2]. For example, for high spatial resolution remote sense image, shadow removal is very critical for target identification and information recovering. Shadow removal and editing can also improve the visual realism and physical realism in image processing [3], [4]. Shadow removal is now an popular research direction in computer vision and image processing communities. Many shadow removal approaches have been proposed in the last decade (see [2], [5], and [6] for a survey).

As image shadow is formulated in different lighting conditions, surface materials and scene shape, producing

S

Manuscript received November 3, 2014; revised May 3, 2015 and July 5, 2015; accepted July 28, 2015. Date of publication August 5, 2015; date of current version August 31, 2015. This work was supported in part by the National Basic Research Program of China under Grant 2012CB725303, in part by the National Natural Science Foundation of China under Grant 61472288, New Century Excellent Talents under Grant NCET-13-0441, and in part by the Key Grant Project of Hubei Province under Grant 2013AAA02. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Javier Mateos. (Corresponding author: Chunxia Xiao.)

The authors are with the School of Computer, Wuhan University, Wuhan 430072, China (e-mail: lingzhang@whu.edu.cn; zhangqing.whu.cs@ gmail.com; cxxiao@whu.edu.cn).

Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TIP.2015.2465159

high-quality shadow-free results is a challenging work for both

natural images and aerial remote sensing images. The first problem to be addressed is shadow detection. For image with complex shadows, accurately shadow detecting is a difficult problem. For example, sometimes it is even difficult for human to distinguish little dark objects from the scattered shadow points. The second problem is, when the illumination conditions, object materials, and scene shapes are complex, the shadows in the image are usually nonuniform, which makes it difficult to obtain consistent shadow removal results. Finally, as the illumination usually changes dramatically in the boundary regions, effectively recovering the illumination on the shadow boundaries is also a challenging task.

In this paper, we present a novel shadow removal approach using an illumination recovering optimization method. We first detect the shadows in the input image, and compute the shadow alpha for the shadows. Then we adaptively decompose the input image into overlapped patches according to the shadow distribution. Denser patches are put on the shadow boundaries and the regions with dramatically changed illumination. Finally, by building the correspondence between the shadow patch and the lit patch based on illumination independent texture similarity, we develop an optimized illumination recovering operator which can effectively remove the shadows and recover the texture detail under the shadow patches. By using coherent illumination optimization processing among the neighboring patches, we produce high-quality shadow-free results. The texture details under the shadow regions are effectively recovered, and the recovered illumination is consistent with that of surrounding environment. Fig. 1 shows the overview of the proposed shadow removal system.

The proposed method is simple and effective. With the adaptively image patch decomposition and illumination inde-pendent patch matching, our method can process nonuniform shadows and shadow regions with rich texture types. While most existing shadow removal methods do not work well in processing images with complex shadows and various texture materials. Furthermore, with the adaptively image decompo-sition, our shadow removal system also works well on the shadow boundaries. In addition, we present several applications for the proposed method, such as illumination and color transfer, shadow edge softening. In our applications, shadow sharpness, position, and intensity can be freely adjusted, which illustrate the versatility of the proposed method.

The main contributions and advantages of our work are as follows:

? Introduce a novel local illumination recovering optimization,

which produces illumination consistent

1057-7149 ? 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.

See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

⑻

(d) (e)

Fig. 1. The overview of our shadow removal system. (a) Input image, (b) nearest patch search for the shadow patches, (c) shadow removal result based on the patch pairs, (d) shadow-free result with coherence optimization, (e) final shadow-free result with boundary processing.

(C)

（b)

and gradient change in penumbra area. Although impressive

results on images with nonuniform shadows and results have been presented, the gradient domain based shadow

removal methods may fail to produce high-quality shadow-free multi-texture types.

? Develop an adaptive image decomposition strategy for results for the nonuniform shadows since they only modify the patch matching used in shadow removal, which generates gradients in shadow edges or penumbra regions. To receive compelling results on both shadow interior and shadow satisfied results, the gradient domain based method needs to

detect accurate shadow edges [1], or find best function fit for the boundaries.

? Present several image editing applications using the shadow edge intensity [3], [11]. Since shadow edge detection is proposed shadow removal operator, which illustrates the also a challenging work, this limits the practicality of these versatility of the proposed approach. methods.

Methods Based on Illumination Transfer: Inspired by color The remainder of this paper is organized as follows. In

Section II, we introduce the related work. In section III, we transfer technique [20], several shadow removal methods have perform the shadow detection and shadow alpha matting, which been proposed. The basic idea of color transfer is applying is a preprocessing step. Section IV introduces the local statistical analysis to transfer one image’s color characteristics to illumination recovering operator. Section V presents the shadow another. Shor and Lischinski [10] built linear mapping models removal approach. In Section VI, we present several applications from the shadow and nonshadow areas based on illumination of our method. Section VII shows the experimental results, and transfer. However, the shadow and nonshadow samples used for Section VIII concludes the paper. estimating the illuminated recovering parameters should have

similar texture materials, which means this method can only

II. RELATED WORK

handle shadow regions with uniform texture. Xiao et al. [21]

Shadow removal involves two basic stages: shadow detection proposed an adaptive multi-scale illumination transfer technique and shadow removal. So far, many shadow detection methods considering the material reflectance variation, and improved to have been proposed, including automatic shadow detection recover the texture details of the shadow regions. The global methods [1], [5], [7], [8] and user-assisted shadow detection color transfer methods [10], [21] applied a global transformation methods [9]-[11]. Many video shadow detection methods also on the shadow regions to match color statistics of the nonshadow have been proposed [12], [13]. A variety of shadow removal regions. As they do not take texture variety into account, they methods have been developed in computer vision, including the work well when the shadow and nonshadow sample regions shadow removal from single images [1], [2], [5], [6], [14], [15], exhibit similar texture. and shadow removal from multiple images and video sequences More recently, Xiao et al. [5] further improved [21] by [16], [17]. A comprehensive survey is beyond the scope of this performing illumination transfer on the matched subregion pairs paper, we only focus on shadow removal methods from a single between the shadow regions and the nonshadow regions. This image and review the most related work to our paper. method can process complex images with different kinds of Methods Based on Gradient Domain: Finlayson et al. shadow texture regions and illumination conditions. But it fails proposed a series of shadow removal methods [1], [18], [19] to ensure smooth transition between subregions, and may based on gradient domain manipulation. The basic idea in these produce illumination inconsistent results. Li et al. [6] applied methods is to reconstruct the shadow-free image based on the color transfer results as the predicted shadow-free image used in gradient information in shadow regions by nullifying the the fidelity item. They also introduced an adaptive NL (nonlocal) gradients on shadow boundaries. Mohan et al. [3] proposed a regularized shadow removal method for aerial images by shadow removal method by fitting a gradient domain shadow regularizing the shadow scale and the updated shadow-free edge model. This method also simulates a variety of lighting image. The adaptive behavior regionally smooths the shadow conditions, such as ambient lighting, for shadow editing. Liu and scale, which can preserve the edges and the textures in the Gleicher [11] removed shadow by assigning gradients for the shadow regions. The NL Laplacian priorshadow and lit area according to illumination

can reduce noises in the shadow regions, but introduces texture detail blurring artifacts.

Methods Based on Shadow Matting: Chuang et al. [17] proposed a shadow matting method which considered the input image as a linear combination of a shadow-free image and a shadow matte image. Although this method can be used to remove shadow, it focuses on shadow extracting and compositing, and does not pay much effort for shadow removal. Instead of considering shadow extraction as the conventional matting equation, Wu et al. [9] and Wu and Tang [22] supposed shadow effect as a light attenuation problem. Wu and Tang [22] proposed a Bayesian framework for shadow extraction and produced shadowless image. Later, Wu et al. [9] first estimated an approximate shadowless image using color transfer techniques [20], and then defined an optimization function incorporating the approximate shadowless image for shadow extracting. Both [9] and [22] applied user-supplied hints to identify shadow and nonshadow regions. Although [9], [22] tried to preserve the texture appearance under the extracted shadow, to process complex nonuniform shadows, both methods still may not work well in recovering the image detail in the shadow areas. Other Shadow Removal Methods: Guo et al. [8] presented a regions-based approach for shadow detection and removal. By segmenting the input shadow image using mean shift algorithm, Guo et al. explored the pairwise relationship between regions, and provided information about illumination condition of regions for shadow detection. They estimated the ratio between direct light and environment light, and recovered the illumination of shadow regions by relighting each pixel. This approach can produce high-quality shadow-free results for simple shadow images. However, for image with complex shadows and nonuniform shadows, this method cannot recover satisfactory texture detail since it does not take the reflectance variation into account. More recently, Xiao et al. [23] transformed the shadow removal problem into nonlocal feature matching between unshadowed samples and shadow pixels, and recovered the illumination from a single RGB-D image by applying an energy minimization method. They implemented the feature matching by measuring similarities between nonlocal pixels using normals, chromaticity and spatial locations. However, the performance of this algorithm relies on accurate depth map provided by the depth sensor such as MS Kinect. Arbel and Hel-Or [2] considered each image channel as an intensity surface, and approximated the surface shape of shadow regions using a smooth thin-plate with the constraint on the shadow edges and the texture anchor points. This method can remove some nonuniform shadows. However, smooth thin-plate approximation is difficult to accurately estimate the shadow scale factors on the textured and highly structured images. Although texture anchor points have been used to alleviate this problem, there is still much room for improvement. Assuming a single flat texture shadow surface, Baba et al. [24] presented a shadow removal method based on color and variance adjustment of shadow pixels in RGB space. The method in [25] is based on the estimation of shadow scale factors assuming uniform shadow intensities

(a) (b) (c)

Fig. 2. Shadow matte result. (a) Input image, (b) brushes for sample regions, (c) shadow matte map, where the white regions represent shadow regions, the black regions are lit regions and the rest is the shadow boundary, and the bottom left is close-up for the red box.

and hard shadows. Both methods [24], [25] used inpainting techniques for completion of missing information in shadow boundary regions.

III. SHADOW DETECTION AND ALPHA MATTING

To perform shadow removal, we first have to detect the shadows. The current shadow detection methods can be divided into two categories: automatic shadow detection [26], [27] and interactive shadow detection [6], [10], [11]. As automatic shadow detection is an extremely difficult task, we incorporate user interaction and shadow alpha matting for shadow detection. Similar to interactive image matting, we first specify some shadow samples and lit samples, and construct a trimap for the input image. With the tripmap, we can extract the shadow alpha matte using the specified samples as the constraints. More specially, we set a = 0 for the specified shadow samples, and a = 1 for the specified lit samples. Then we apply the closed-form matting method [28] and minimize the following energy equation for computing the shadow alpha:

a = argmin a T L a + X(a T — b^ ) DS (a — bS) (1)

where L is the Laplacian matrix, DS is a diagonal matrix whose diagonal elements are one for constrained pixels and zero for all other pixels, and the vector bS contains specified alpha values for the constrained nonshadow pixels and zero for all other pixels. Minimizing above energy equation, we can get the shadow alpha a.

With the computed shadow alpha matte a, we can identify the shadow regions of the input image. Specially, given threshold 而 and 办，the pixels with a < 而 can be considered as the pixels in the shadow regions (umbra regions). The pixels with a > S2 can be considered as pixels in the nonshadow regions. The pixels with ^1 < a < S2 can be considered as the pixels in the penumbra regions, for hard shadow, these pixels can be considered as the shadow boundaries. In the shadow boundaries, the a value usually changes dramatically. In our experiments, we set 而=0.2 and S2 = 0.9. Fig. 2 shows a shadow matte result. Note that several methods [6], [10] also used closed-form matting method [28], [29] for shadow detection.

IV. LOCAL ILLUMINATION RECOVERING OPERATOR

In image formation equation [30], an image is the pixel-wise product of illumination and reflectance: Ix = RxLx, where

lx is the observed RGB color at pixel x. Lx and Rx are the illumination and the reflectance (albedo) at pixel x, respectively. We assume shadows in the scene are cast due to the single primary source of illumination is blocked. If the pixel x is in the lit regions, the illumination can be described as a sum of the direct illumination Ld and the indirect (ambient) illumination La: Lx = Ld + La. In the umbra regions, since the object occludes the primary light source, it will cast a shadow at pixel x, and the cast illumination is only the ambient illumination: Lx = La. In the penumbra regions or shadow boundaries, as the occluder blocks some of the direct illumination, the practical illumination including the ambient illumination and some of the direct illumination, which can be described as: Lx = ?xLd + La, where ax is the shadow matting alpha value of pixel x (introduced in Section III), which can be regarded as the attenuation factor of the direction illumination. Thus, the shadow color at pixel x and its shadow-free value l[ree can be expressed as: \(a Ld + La) Rx

lx z

(Ld + La) Rx If (2) Shor and Lischinski [10] claimed that there is a linear relationship between the observed value of pixel in a shadow region and its shadow-free value. The shadow-free value at pixel x can be represented as Irx = kIx-\\-by where k = and b = ju(L) — kju(S). ju(S) is the average value of the shadow sample regions, and ju(L) is the average value of the lit sample regions which have similar textures to the shadow sample regions. a(L) and a(S) are the standard deviation corresponding to the regions. Note that, this shadow removal model is only used in images with single texture, while can not work well for images with complex textures. The use of local patch in our method is just to meet this demand. We employ the results of Shor et al. to estimate the initial value in our shadow removal system. Specifically, we replace the shadow sample regions and the lit sample regions with the shadow patch and its matched lit patch, and the shadow-free value at pixel x in a shadow patch can be estimated as:

- fi(S)) + fi(L)

^(S)

Then Equation (2) can be reformulated as:

Ld

we can efficiently remove the shadows in the shadow patch using information of the matched patch with similar texture. To obtain a smooth shadow alpha a on the shadow boundaries, we estimate the alpha a for pixel on shadow boundaries using the average a value of the neighboring pixels.

V. SHADOW REMOVAL

l'x =

La

Ixa

From Equation (6) we know that, with the computed shadow matting alpha and the local reflectance constant assumption, for one patch in the shadow regions, if we find a lit patch in the nearby lit regions with similar material or texture, the shadow of this patch can be removed using our illumination recovering operator. To remove the whole shadows of the input image, we can transform this problem into the following solver: for each patch in the shadow regions, we find a corresponding patch with similar texture in the lit regions, and then by using the illumination recovering method on each corresponding patch pair, the shadows in the image can be removed.

However, to receive satisfactory results, the following several issues need to be addressed. The first issue is that if we remove the shadow of each patch separately, the results may be inconsistent and visually unnatural, thus, we should ensure that the shadow-free results have natural transitions between neighboring patches. The second important issue is that as the dramatically change of illumination may lead to texture detail loss on shadow boundaries, these boundaries should be elaborately processed to recover the texture information. Finally, for each patch in the shadow regions, to make the shadow removal processing efficient, the corresponding matched patch should be found efficiently in the lit regions.

To address above issues, we incorporate the illumination recovering operator into coherent optimization process. We first decompose the input image into adaptive overlapped patches. For each patches in the shadow regions, we find a corresponding patch in the lit regions for shadow removal. The shadow-free value for the pixel in the overlapped region is computed as the weighted average of all the shadow-removed values at its position from patches containing this pixel. By using this (3) technique, we can receive shadow removal result with satisfied

transition between patches. Finally, we exploit the image synthesis technology to recover the illumination and texture information on the shadow boundaries.

A. image Decomposition

We decompose the input image into adaptive patches according to the illumination distribution. Formally, let I denote the input image, S and L are the shadow regions and lit regions of I, respectively. We first decompose the input image I into uniformly overlapped patches with patch size of w x w (w > 10). Instead of moving patch one pixel at a time, we shift patch five pixels at a time. Thus, we set patches that sufficiently overlap with each other which ensures that a given pixels can be affected by multiple patches, as shown in Fig. 3. This makes the shadow removal result exhibit natural and smooth transition between adjacent patches, and also alleviates the artifacts when a certain patch is matched inaccurately.

a

x!x — lx

x^x —Ix

j\\ , we have Ld = tLa

(4)

Let t Based on the above analysis, the shadow-free result of pixel x in shadow regions (umbra

(5) regions or penumbra regions) can be estimated as:

(t + l)LaRx = (6)

axt + 1

Our illumination recovering operator (Equation (6)) is based

on the assumption that, in a local patch,

if the illumination and reflectance variation

are small. Using this operator,