Variational and Deep Learning Approaches for Intrinsic Light Field Decomposition

Loading...
Thumbnail Image
Date
2020
Editors
Contact
Journal ISSN
Electronic ISSN
ISBN
Bibliographical data
Publisher
Series
DOI (citable link)
ArXiv-ID
International patent number
Link to the license
EU project number
Project
Open Access publication
Restricted until
Title in another language
Research Projects
Organizational Units
Journal Issue
Publication type
Dissertation
Publication status
Published
Published in
Abstract
Intrinsic image decomposition aims to separate an illumination invariant reflectance image from an input color image, which is still one of the fundamental problems in computer vision. This decomposition is widely used in photo and material editing, image segmentation and shape estimation tasks. According to the dichromatic reflection model, the light reflected from a scene point has two independent components: light reflected from the surface body and light at the interface. Body reflection is known as the diffuse component and it is independent of viewing direction, while interface reflection is known as the specular component and it is view-dependent. Most intrinsic image algorithms are designed for Lambertian scenes, with only diffuse reflection. However, their performance decreases if a scene contains specularity. In the real world, there are few scenes with only Lambertian objects. Instead, they have specular surfaces, which makes the decomposition problem harder due to the complicated nature of specular reflection. This thesis focuses on intrinsic light field decomposition, where we formulate and solve the problem with respect to three variables: albedo, shading, and specularity. Thus, we can deal with non-Lambertian scenes. We use a 4D light field, which is a collection of images sampled on a regular grid, instead of a single image. Rich information inherited from the light field allows us to distinguish between diffuse and specular reflection, and also allows us to robustly recover the intrinsic components. We tackle the problem with variational and deep learning approaches, compare their performance, and discuss the strengths and weaknesses of both techniques. In the variational method, we introduce priors for the intrinsic components and we solve an energy minimization problem with convex optimization. Because geometrical information plays an important role in the appearance and behavior of intrinsic components, we develop a disparity estimation method, where we not only optimize the disparity labels but also enforce piecewise smoothness of a normal map. Our deep learning approach is based on the assumption that if mathematical models allow us to compute a disparity and intrinsic components from a light field, then these models can be approximated with a deep convolutional neural network. Moreover, because disparity estimation and intrinsic light fields are closely related, a single network can be sufficient to perform all tasks together and they can benefit from each other. Thus, we establish a multi-task learning strategy for light fields, which is not only limited to the particular collection of tasks but (in theory) can also be used for various computer vision applications. We demonstrate the advantage of our approach on four state-of-the-art computer vision problems: disparity estimation, reflection separation, intrinsic images, and super-resolution. Extensive evaluations based on multiple, publicly-available, synthetic and real-world datasets prove our methodology and show the advantage of using light fields over other data structures. Our proposed algorithms outperform state-of-the-art methods for intrinsic images and disparity estimation, and achieve a competing quality for super-resolution and reflection separation.
Summary in another language
Subject (DDC)
004 Computer Science
Keywords
Conference
Review
undefined / . - undefined, undefined. - (undefined; undefined)
Cite This
ISO 690ALPEROVICH, Anna, 2020. Variational and Deep Learning Approaches for Intrinsic Light Field Decomposition [Dissertation]. Konstanz: University of Konstanz
BibTex
@phdthesis{Alperovich2020Varia-49781,
  year={2020},
  title={Variational and Deep Learning Approaches for Intrinsic Light Field Decomposition},
  author={Alperovich, Anna},
  address={Konstanz},
  school={Universität Konstanz}
}
RDF
<rdf:RDF
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:bibo="http://purl.org/ontology/bibo/"
    xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#"
    xmlns:foaf="http://xmlns.com/foaf/0.1/"
    xmlns:void="http://rdfs.org/ns/void#"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > 
  <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/49781">
    <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2020-06-03T07:10:36Z</dc:date>
    <dcterms:rights rdf:resource="https://rightsstatements.org/page/InC/1.0/"/>
    <dc:contributor>Alperovich, Anna</dc:contributor>
    <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/>
    <dcterms:abstract xml:lang="eng">Intrinsic image decomposition aims to separate an illumination invariant reflectance image from an input color image, which is still one of the fundamental problems in computer vision. This decomposition is widely used in photo and material editing, image segmentation and shape estimation tasks. According to the dichromatic reflection model, the light reflected from a scene point has two independent components: light reflected from the surface body and light at the interface. Body reflection is known as the diffuse component and it is independent of viewing direction, while interface reflection is known as the specular component and it is view-dependent. Most intrinsic image algorithms are designed for Lambertian scenes, with only diffuse reflection. However, their performance decreases if a scene contains specularity. In the real world, there are few scenes with only Lambertian objects. Instead, they have specular surfaces, which makes the decomposition problem harder due to the complicated nature of specular reflection. This thesis focuses on intrinsic light field decomposition, where we formulate and solve the problem with respect to three variables: albedo, shading, and specularity. Thus, we can deal with non-Lambertian scenes. We use a 4D light field, which is a collection of images sampled on a regular grid, instead of a single image. Rich information inherited from the light field allows us to distinguish between diffuse and specular reflection, and also allows us to robustly recover the intrinsic components. We tackle the problem with variational and deep learning approaches, compare their performance, and discuss the strengths and weaknesses of both techniques. In the variational method, we introduce priors for the intrinsic components and we solve an energy minimization problem with convex optimization. Because geometrical information plays an important role in the appearance and behavior of intrinsic components, we develop a disparity estimation method, where we not only optimize the disparity labels but also enforce piecewise smoothness of a normal map. Our deep learning approach is based on the assumption that if mathematical models allow us to compute a disparity and intrinsic components from a light field, then these models can be approximated with a deep convolutional neural network. Moreover, because disparity estimation and intrinsic light fields are closely related, a single network can be sufficient to perform all tasks together and they can benefit from each other. Thus, we establish a multi-task learning strategy for light fields, which is not only limited to the particular collection of tasks but (in theory) can also be used for various computer vision applications. We demonstrate the advantage of our approach on four state-of-the-art computer vision problems: disparity estimation, reflection separation, intrinsic images, and super-resolution. Extensive evaluations based on multiple, publicly-available, synthetic and real-world datasets prove our methodology and show the advantage of using light fields over other data structures. Our proposed algorithms outperform state-of-the-art methods for intrinsic images and disparity estimation, and achieve a competing quality for super-resolution and reflection separation.</dcterms:abstract>
    <dc:language>eng</dc:language>
    <dcterms:title>Variational and Deep Learning Approaches for Intrinsic Light Field Decomposition</dcterms:title>
    <dcterms:hasPart rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/49781/3/Alperovich_2-ee236e5qzmf94.pdf"/>
    <dc:creator>Alperovich, Anna</dc:creator>
    <foaf:homepage rdf:resource="http://localhost:8080/"/>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <dc:rights>terms-of-use</dc:rights>
    <dcterms:issued>2020</dcterms:issued>
    <dspace:hasBitstream rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/49781/3/Alperovich_2-ee236e5qzmf94.pdf"/>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2020-06-03T07:10:36Z</dcterms:available>
    <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/49781"/>
  </rdf:Description>
</rdf:RDF>
Internal note
xmlui.Submission.submit.DescribeStep.inputForms.label.kops_note_fromSubmitter
Contact
URL of original publication
Test date of URL
Examination date of dissertation
February 19, 2020
University note
Konstanz, Univ., Doctoral dissertation, 2020
Method of financing
Comment on publication
Alliance license
Corresponding Authors der Uni Konstanz vorhanden
International Co-Authors
Bibliography of Konstanz
Refereed