KOPS - The Institutional Repository of the University of Konstanz

Can the online crowd match real expert judgments? : How task complexity and coder location affect the validity of crowd‐coded data

Aufgrund von Vorbereitungen auf eine neue Version von KOPS, können kommenden Montag und Dienstag keine Publikationen eingereicht werden. (Due to preparations for a new version of KOPS, no publications can be submitted next Monday and Tuesday.)

Can the online crowd match real expert judgments? : How task complexity and coder location affect the validity of crowd‐coded data

Cite This

Files in this item

Checksum: MD5:485b94a73d462f08343c829c72991a59

HORN, Alexander, 2019. Can the online crowd match real expert judgments? : How task complexity and coder location affect the validity of crowd‐coded data. In: European Journal of Political Research. Wiley-Blackwell - SSH. 58(1), pp. 236-247. ISSN 0304-4130. eISSN 1475-6765. Available under: doi: 10.1111/1475-6765.12278

@article{Horn2019-02onlin-50741, title={Can the online crowd match real expert judgments? : How task complexity and coder location affect the validity of crowd‐coded data}, year={2019}, doi={10.1111/1475-6765.12278}, number={1}, volume={58}, issn={0304-4130}, journal={European Journal of Political Research}, pages={236--247}, author={Horn, Alexander} }

<rdf:RDF xmlns:dcterms="http://purl.org/dc/terms/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:bibo="http://purl.org/ontology/bibo/" xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:void="http://rdfs.org/ns/void#" xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > <rdf:Description rdf:about="https://kops.uni-konstanz.de/rdf/resource/123456789/50741"> <dc:contributor>Horn, Alexander</dc:contributor> <dc:language>eng</dc:language> <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2020-09-09T12:12:59Z</dcterms:available> <dcterms:rights rdf:resource="https://rightsstatements.org/page/InC/1.0/"/> <dc:creator>Horn, Alexander</dc:creator> <dcterms:issued>2019-02</dcterms:issued> <dspace:hasBitstream rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/50741/1/Horn_2-r9zfzv6w3q7q2.pdf"/> <foaf:homepage rdf:resource="http://localhost:8080/jspui"/> <dcterms:title>Can the online crowd match real expert judgments? : How task complexity and coder location affect the validity of crowd‐coded data</dcterms:title> <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/50741"/> <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/> <dcterms:abstract xml:lang="eng">Crowd‐coding is a novel technique that allows for fast, affordable and reproducible online categorisation of large numbers of statements. It combines judgements by multiple, paid, non‐expert coders to avoid miscoding(s). It has been argued that crowd‐coding could replace expert judgements, using the coding of political texts as an example in which both strategies produce similar results. Since crowd‐coding yields the potential to extend the replication standard to data production and to ‘scale’ coding schemes based on a modest number of carefully devised test questions and answers, it is important that its possibilities and limitations are better understood. While previous results for low complexity coding tasks are encouraging, this study assesses whether and under what conditions simple and complex coding tasks can be outsourced to the crowd without sacrificing content validity in return for scalability. The simple task is to decide whether a party statement counts as positive reference to a concept – in this case: equality. The complex task is to distinguish between five concepts of equality. To account for the crowd‐coder's contextual knowledge, the IP restrictions are varied. The basis for comparisons is 1,404 party statements, coded by experts and the crowd (resulting in 30,000 online judgements). Comparisons of the expert‐crowd match at the level of statements and party manifestos show that the results are substantively similar even for the complex task, suggesting that complex category schemes can be scaled via crowd‐coding. The match is only slightly higher when IP restrictions are used as an approximation of coder expertise.</dcterms:abstract> <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/rdf/resource/123456789/42"/> <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/rdf/resource/123456789/42"/> <dc:rights>terms-of-use</dc:rights> <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2020-09-09T12:12:59Z</dc:date> <dcterms:hasPart rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/50741/1/Horn_2-r9zfzv6w3q7q2.pdf"/> </rdf:Description> </rdf:RDF>

Downloads since Sep 9, 2020 (Information about access statistics)

Horn_2-r9zfzv6w3q7q2.pdf 6

This item appears in the following Collection(s)

Search KOPS


Browse

My Account