Critical analysis on the reproducibility of visual quality assessment using deep features
Critical analysis on the reproducibility of visual quality assessment using deep features
Loading...
Date
2022
Editors
Journal ISSN
Electronic ISSN
ISBN
Bibliographical data
Publisher
Series
URI (citable link)
DOI (citable link)
International patent number
Link to the license
EU project number
Project
Open Access publication
Collections
Title in another language
Publication type
Journal article
Publication status
Published
Published in
PLoS ONE ; 17 (2022), 8. - e0269715. - Public Library of Science (PLoS). - eISSN 1932-6203
Abstract
Data used to train supervised machine learning models are commonly split into independent training, validation, and test sets. This paper illustrates that complex data leakage cases have occurred in the no-reference image and video quality assessment literature. Recently, papers in several journals reported performance results well above the best in the field. However, our analysis shows that information from the test set was inappropriately used in the training process in different ways and that the claimed performance results cannot be achieved. When correcting for the data leakage, the performances of the approaches drop even below the state-of-the-art by a large margin. Additionally, we investigate end-to-end variations to the discussed approaches, which do not improve upon the original.
Summary in another language
Subject (DDC)
004 Computer Science
Keywords
Conference
Review
undefined / . - undefined, undefined. - (undefined; undefined)
Cite This
ISO 690
GÖTZ-HAHN, Franz, Vlad HOSU, Dietmar SAUPE, 2022. Critical analysis on the reproducibility of visual quality assessment using deep features. In: PLoS ONE. Public Library of Science (PLoS). 17(8), e0269715. eISSN 1932-6203. Available under: doi: 10.1371/journal.pone.0269715BibTex
@article{GotzHahn2022Criti-59105, year={2022}, doi={10.1371/journal.pone.0269715}, title={Critical analysis on the reproducibility of visual quality assessment using deep features}, number={8}, volume={17}, journal={PLoS ONE}, author={Götz-Hahn, Franz and Hosu, Vlad and Saupe, Dietmar}, note={Article Number: e0269715} }
RDF
<rdf:RDF xmlns:dcterms="http://purl.org/dc/terms/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:bibo="http://purl.org/ontology/bibo/" xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:void="http://rdfs.org/ns/void#" xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/59105"> <dc:rights>Attribution 4.0 International</dc:rights> <dc:language>eng</dc:language> <dcterms:rights rdf:resource="http://creativecommons.org/licenses/by/4.0/"/> <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2022-11-10T13:22:25Z</dcterms:available> <dcterms:abstract xml:lang="eng">Data used to train supervised machine learning models are commonly split into independent training, validation, and test sets. This paper illustrates that complex data leakage cases have occurred in the no-reference image and video quality assessment literature. Recently, papers in several journals reported performance results well above the best in the field. However, our analysis shows that information from the test set was inappropriately used in the training process in different ways and that the claimed performance results cannot be achieved. When correcting for the data leakage, the performances of the approaches drop even below the state-of-the-art by a large margin. Additionally, we investigate end-to-end variations to the discussed approaches, which do not improve upon the original.</dcterms:abstract> <dc:creator>Hosu, Vlad</dc:creator> <dc:creator>Saupe, Dietmar</dc:creator> <dcterms:title>Critical analysis on the reproducibility of visual quality assessment using deep features</dcterms:title> <dc:creator>Götz-Hahn, Franz</dc:creator> <dc:contributor>Saupe, Dietmar</dc:contributor> <dc:contributor>Hosu, Vlad</dc:contributor> <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/> <foaf:homepage rdf:resource="http://localhost:8080/"/> <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/> <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/59105"/> <dcterms:hasPart rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/59105/1/Goetz-Hahn_2-1slsft0xvpsz07.pdf"/> <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/> <dc:contributor>Götz-Hahn, Franz</dc:contributor> <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2022-11-10T13:22:25Z</dc:date> <dspace:hasBitstream rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/59105/1/Goetz-Hahn_2-1slsft0xvpsz07.pdf"/> <dcterms:issued>2022</dcterms:issued> </rdf:Description> </rdf:RDF>
Internal note
xmlui.Submission.submit.DescribeStep.inputForms.label.kops_note_fromSubmitter
Examination date of dissertation
Method of financing
Comment on publication
Alliance license
Corresponding Authors der Uni Konstanz vorhanden
International Co-Authors
Bibliography of Konstanz
Yes
Refereed
Yes