Towards reproducibility in recommender-systems research

Cite This

Files in this item

Checksum: MD5:6aed2e5ba4e10c7b6651c181e778a464

BEEL, Joeran, Corinna BREITINGER, Stefan LANGER, Andreas LOMMATZSCH, Bela GIPP, 2016. Towards reproducibility in recommender-systems research. In: User Modeling and User-Adapted Interaction : umuai. 26(1), pp. 69-101. ISSN 0924-1868. eISSN 1573-1391. Available under: doi: 10.1007/s11257-016-9174-x

@article{Beel2016-03-12Towar-33528, title={Towards reproducibility in recommender-systems research}, year={2016}, doi={10.1007/s11257-016-9174-x}, number={1}, volume={26}, issn={0924-1868}, journal={User Modeling and User-Adapted Interaction : umuai}, pages={69--101}, author={Beel, Joeran and Breitinger, Corinna and Langer, Stefan and Lommatzsch, Andreas and Gipp, Bela} }

<rdf:RDF xmlns:dcterms="" xmlns:dc="" xmlns:rdf="" xmlns:bibo="" xmlns:dspace="" xmlns:foaf="" xmlns:void="" xmlns:xsd="" > <rdf:Description rdf:about=""> <dcterms:issued>2016-03-12</dcterms:issued> <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/> <dc:language>eng</dc:language> <foaf:homepage rdf:resource="http://localhost:8080/jspui"/> <dcterms:available rdf:datatype="">2016-04-01T08:21:57Z</dcterms:available> <dc:contributor>Gipp, Bela</dc:contributor> <dc:contributor>Langer, Stefan</dc:contributor> <dc:contributor>Beel, Joeran</dc:contributor> <dc:contributor>Lommatzsch, Andreas</dc:contributor> <dc:creator>Gipp, Bela</dc:creator> <dcterms:hasPart rdf:resource=""/> <dspace:hasBitstream rdf:resource=""/> <dc:rights>terms-of-use</dc:rights> <bibo:uri rdf:resource=""/> <dspace:isPartOfCollection rdf:resource=""/> <dc:creator>Breitinger, Corinna</dc:creator> <dc:creator>Lommatzsch, Andreas</dc:creator> <dc:date rdf:datatype="">2016-04-01T08:21:57Z</dc:date> <dc:creator>Beel, Joeran</dc:creator> <dcterms:abstract xml:lang="eng">Numerous recommendation approaches are in use today. However, comparing their effectiveness is a challenging task because evaluation results are rarely reproducible. In this article, we examine the challenge of reproducibility in recommender-system research. We conduct experiments using Plista’s news recommender system, and Docear’s research-paper recommender system. The experiments show that there are large discrepancies in the effectiveness of identical recommendation approaches in only slightly different scenarios, as well as large discrepancies for slightly different approaches in identical scenarios. For example, in one news-recommendation scenario, the performance of a content-based filtering approach was twice as high as the second-best approach, while in another scenario the same content-based filtering approach was the worst performing approach. We found several determinants that may contribute to the large discrepancies observed in recommendation effectiveness. Determinants we examined include user characteristics (gender and age), datasets, weighting schemes, the time at which recommendations were shown, and user-model size. Some of the determinants have interdependencies. For instance, the optimal size of an algorithms’ user model depended on users’ age. Since minor variations in approaches and scenarios can lead to significant changes in a recommendation approach’s performance, ensuring reproducibility of experimental results is difficult. We discuss these findings and conclude that to ensure reproducibility, the recommender-system community needs to (1) survey other research fields and learn from them, (2) find a common understanding of reproducibility, (3) identify and understand the determinants that affect reproducibility, (4) conduct more comprehensive experiments, (5) modernize publication practices, (6) foster the development and use of recommendation frameworks, and (7) establish best-practice guidelines for recommender-systems research.</dcterms:abstract> <dcterms:title>Towards reproducibility in recommender-systems research</dcterms:title> <dc:contributor>Breitinger, Corinna</dc:contributor> <dcterms:rights rdf:resource=""/> <dc:creator>Langer, Stefan</dc:creator> <dcterms:isPartOf rdf:resource=""/> </rdf:Description> </rdf:RDF>

Downloads since Apr 1, 2016 (Information about access statistics)

Beel_0-324818.pdf 1025

This item appears in the following Collection(s)

Search KOPS


My Account