Impact of feedback on crowdsourced visual quality assessment with paired comparisons
Lade...
Dateien
Datum
2024
Autor:innen
Herausgeber:innen
ISSN der Zeitschrift
Electronic ISSN
ISBN
Bibliografische Daten
Verlag
Schriftenreihe
Auflagebezeichnung
URI (zitierfähiger Link)
DOI (zitierfähiger Link)
Internationale Patentnummer
Link zur Lizenz
Angaben zur Forschungsförderung
Deutsche Forschungsgemeinschaft (DFG): 496858717
Deutsche Forschungsgemeinschaft (DFG): 251654672
Deutsche Forschungsgemeinschaft (DFG): 251654672
Projekt
JND - basierte perzeptuelle Videoqualitätsanalyse und -modellierung
Open Access-Veröffentlichung
Open Access Green
Sammlungen
Core Facility der Universität Konstanz
Titel in einer weiteren Sprache
Publikationstyp
Beitrag zu einem Konferenzband
Publikationsstatus
Published
Erschienen in
16th International Conference on Quality of Multimedia Experience (QoMEX). Piscataway, NJ: IEEE, 2024, S. 125-131. ISBN 979-8-3503-6158-2. Verfügbar unter: doi: 10.1109/QoMEX61742.2024.10598256
Zusammenfassung
This paper presents a comprehensive investigation into the effects of immediate feedback on crowdworkers’ performance in subjective image quality assessment tasks using paired comparisons. The study is motivated by the need for reliable and efficient crowdsourcing tasks for image quality assessment. A large-scale experiment involving 200 participants was conducted, where participants completed 120 paired comparisons with and without feedback. The feedback informed the workers of the correctness of their responses to comparisons. Almost all of the participants (97%) preferred receiving feedback. The results indicate that feedback reduced response time, improved user experience, and did not cause a bias in the estimation of the just noticeable difference (JND). On the other hand, feedback did not significantly affect accuracy, correlation with the ground truth, or create a learning effect. This study contributes to the field by being one of the first to examine the impact of feedback on crowdworker performance in subjective image quality assessment tasks. The dataset which includes the images and ratings can be accessed at https://database.mmsp-kn.de/feedback-study-dataset.html.
Zusammenfassung in einer weiteren Sprache
Fachgebiet (DDC)
004 Informatik
Schlagwörter
Konferenz
2024 16th International Conference on Quality of Multimedia Experience (QoMEX), 18. Juni 2024 - 20. Juni 2024, Karlshamn, Sweden
Rezension
undefined / . - undefined, undefined
Zitieren
ISO 690
JENADELEH, Mohsen, Alexander HESS, Simon HVIID DEL PIN, Edwin GAMBOA, Matthias HIRTH, Dietmar SAUPE, 2024. Impact of feedback on crowdsourced visual quality assessment with paired comparisons. 2024 16th International Conference on Quality of Multimedia Experience (QoMEX). Karlshamn, Sweden, 18. Juni 2024 - 20. Juni 2024. In: 16th International Conference on Quality of Multimedia Experience (QoMEX). Piscataway, NJ: IEEE, 2024, S. 125-131. ISBN 979-8-3503-6158-2. Verfügbar unter: doi: 10.1109/QoMEX61742.2024.10598256BibTex
@inproceedings{Jenadeleh2024Impac-70264,
year={2024},
doi={10.1109/QoMEX61742.2024.10598256},
title={Impact of feedback on crowdsourced visual quality assessment with paired comparisons},
isbn={979-8-3503-6158-2},
publisher={IEEE},
address={Piscataway, NJ},
booktitle={16th International Conference on Quality of Multimedia Experience (QoMEX)},
pages={125--131},
author={Jenadeleh, Mohsen and Heß, Alexander and Hviid Del Pin, Simon and Gamboa, Edwin and Hirth, Matthias and Saupe, Dietmar},
note={This research is funded by the DFG (German Research Foundation) – Project
ID 496858717, titled “JND-based Perceptual Video Quality Analysis and
Modeling”. D.S. is funded by DFG Project ID 251654672 – TRR 161. S.H.D.P.
is supported by the project “Quality and Content: understanding the influence
of content on subjective and objective image quality assessment” (grant number
324663, approved on 1 October 2021) from the Research Council of Norway.}
}RDF
<rdf:RDF
xmlns:dcterms="http://purl.org/dc/terms/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:bibo="http://purl.org/ontology/bibo/"
xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#"
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:void="http://rdfs.org/ns/void#"
xmlns:xsd="http://www.w3.org/2001/XMLSchema#" >
<rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/70264">
<dcterms:hasPart rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/70264/1/Jenadeleh_2-cavpuj6fpri01.pdf"/>
<dc:contributor>Gamboa, Edwin</dc:contributor>
<dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2024-06-27T10:41:13Z</dc:date>
<dc:contributor>Hirth, Matthias</dc:contributor>
<bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/70264"/>
<dcterms:abstract>This paper presents a comprehensive investigation into the effects of immediate feedback on crowdworkers’ performance in subjective image quality assessment tasks using paired comparisons. The study is motivated by the need for reliable and efficient crowdsourcing tasks for image quality assessment. A large-scale experiment involving 200 participants was conducted, where participants completed 120 paired comparisons with and without feedback. The feedback informed the workers of the correctness of their responses to comparisons. Almost all of the participants (97%) preferred receiving feedback. The results indicate that feedback reduced response time, improved user experience, and did not cause a bias in the estimation of the just noticeable difference (JND). On the other hand, feedback did not significantly affect accuracy, correlation with the ground truth, or create a learning effect. This study contributes to the field by being one of the first to examine the impact of feedback on crowdworker performance in subjective image quality assessment tasks. The dataset which includes the images and ratings can be accessed at https://database.mmsp-kn.de/feedback-study-dataset.html.</dcterms:abstract>
<dc:creator>Heß, Alexander</dc:creator>
<dcterms:title>Impact of feedback on crowdsourced visual quality assessment with paired comparisons</dcterms:title>
<dc:creator>Hirth, Matthias</dc:creator>
<dc:creator>Hviid Del Pin, Simon</dc:creator>
<dc:contributor>Heß, Alexander</dc:contributor>
<dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
<dc:rights>terms-of-use</dc:rights>
<foaf:homepage rdf:resource="http://localhost:8080/"/>
<dc:contributor>Jenadeleh, Mohsen</dc:contributor>
<dcterms:issued>2024</dcterms:issued>
<dc:language>eng</dc:language>
<dspace:hasBitstream rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/70264/1/Jenadeleh_2-cavpuj6fpri01.pdf"/>
<dc:contributor>Saupe, Dietmar</dc:contributor>
<void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/>
<dc:creator>Gamboa, Edwin</dc:creator>
<dc:creator>Saupe, Dietmar</dc:creator>
<dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2024-06-27T10:41:13Z</dcterms:available>
<dcterms:rights rdf:resource="https://rightsstatements.org/page/InC/1.0/"/>
<dc:creator>Jenadeleh, Mohsen</dc:creator>
<dc:contributor>Hviid Del Pin, Simon</dc:contributor>
<dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
</rdf:Description>
</rdf:RDF>Interner Vermerk
xmlui.Submission.submit.DescribeStep.inputForms.label.kops_note_fromSubmitter
Prüfungsdatum der Dissertation
Finanzierungsart
Kommentar zur Publikation
This research is funded by the DFG (German Research Foundation) – Project
ID 496858717, titled “JND-based Perceptual Video Quality Analysis and
Modeling”. D.S. is funded by DFG Project ID 251654672 – TRR 161. S.H.D.P.
is supported by the project “Quality and Content: understanding the influence
of content on subjective and objective image quality assessment” (grant number
324663, approved on 1 October 2021) from the Research Council of Norway.
Allianzlizenz
Corresponding Authors der Uni Konstanz vorhanden
Internationale Co-Autor:innen
Universitätsbibliographie
Ja
Begutachtet
Link zu Forschungsdaten
Beschreibung der Forschungsdaten
Feedback IQA dataset
