Variational and Deep Learning Approaches for Intrinsic Light Field Decomposition

Abstract

Intrinsic image decomposition aims to separate an illumination invariant reflectance image from an input color image, which is still one of the fundamental problems in computer vision. This decomposition is widely used in photo and material editing, image segmentation and shape estimation tasks. According to the dichromatic reflection model, the light reflected from a scene point has two independent components: light reflected from the surface body and light at the interface. Body reflection is known as the diffuse component and it is independent of viewing direction, while interface reflection is known as the specular component and it is view-dependent. Most intrinsic image algorithms are designed for Lambertian scenes, with only diffuse reflection. However, their performance decreases if a scene contains specularity. In the real world, there are few scenes with only Lambertian objects. Instead, they have specular surfaces, which makes the decomposition problem harder due to the complicated nature of specular reflection. This thesis focuses on intrinsic light field decomposition, where we formulate and solve the problem with respect to three variables: albedo, shading, and specularity. Thus, we can deal with non-Lambertian scenes. We use a 4D light field, which is a collection of images sampled on a regular grid, instead of a single image. Rich information inherited from the light field allows us to distinguish between diffuse and specular reflection, and also allows us to robustly recover the intrinsic components. We tackle the problem with variational and deep learning approaches, compare their performance, and discuss the strengths and weaknesses of both techniques. In the variational method, we introduce priors for the intrinsic components and we solve an energy minimization problem with convex optimization. Because geometrical information plays an important role in the appearance and behavior of intrinsic components, we develop a disparity estimation method, where we not only optimize the disparity labels but also enforce piecewise smoothness of a normal map. Our deep learning approach is based on the assumption that if mathematical models allow us to compute a disparity and intrinsic components from a light field, then these models can be approximated with a deep convolutional neural network. Moreover, because disparity estimation and intrinsic light fields are closely related, a single network can be sufficient to perform all tasks together and they can benefit from each other. Thus, we establish a multi-task learning strategy for light fields, which is not only limited to the particular collection of tasks but (in theory) can also be used for various computer vision applications. We demonstrate the advantage of our approach on four state-of-the-art computer vision problems: disparity estimation, reflection separation, intrinsic images, and super-resolution. Extensive evaluations based on multiple, publicly-available, synthetic and real-world datasets prove our methodology and show the advantage of using light fields over other data structures. Our proposed algorithms outperform state-of-the-art methods for intrinsic images and disparity estimation, and achieve a competing quality for super-resolution and reflection separation.