The performance of international organizations : a new measure and dataset based on computational text analysis of evaluation reports

Lade...
Vorschaubild
Dateien
Eckhard_2-mz3jdaqm1vf66.pdf
Eckhard_2-mz3jdaqm1vf66.pdfGröße: 1.28 MBDownloads: 6
Datum
2023
Herausgeber:innen
Kontakt
ISSN der Zeitschrift
Electronic ISSN
ISBN
Bibliografische Daten
Verlag
Schriftenreihe
Auflagebezeichnung
ArXiv-ID
Internationale Patentnummer
Link zur Lizenz
Angaben zur Forschungsförderung
Projekt
Open Access-Veröffentlichung
Open Access Hybrid
Core Facility der Universität Konstanz
Gesperrt bis
Titel in einer weiteren Sprache
Forschungsvorhaben
Organisationseinheiten
Zeitschriftenheft
Publikationstyp
Zeitschriftenartikel
Publikationsstatus
Published
Erschienen in
The Review of International Organizations. Springer. 2023, 18(4), pp. 753-776. ISSN 1559-7431. eISSN 1559-744X. Available under: doi: 10.1007/s11558-023-09489-1
Zusammenfassung

International organizations (IOs) of the United Nations (UN) system publish around 750 evaluation reports per year, offering insights on their performance across project, program, institutional, and thematic activities. So far, it was not feasible to extract quantitative performance measures from these text-based reports. Using deep learning, this article presents a novel text-based performance metric: We classify individual sentences as containing a negative, positive, or neutral assessment of the evaluated IO activity and then compute the share of positive sentences per report. Content validation yields that the measure adequately reflects the underlying concept of performance; convergent validation finds high correlation with human-provided performance scores by the World Bank; and construct validation shows that our measure has theoretically expected results. Based on this, we present a novel dataset with performance measures for 1,082 evaluated activities implemented by nine UN system IOs and discuss avenues for further research.

Zusammenfassung in einer weiteren Sprache
Fachgebiet (DDC)
320 Politik
Schlagwörter
Economics and Econometrics
Konferenz
Rezension
undefined / . - undefined, undefined
Zitieren
ISO 690ECKHARD, Steffen, Vytautas JANKAUSKAS, Elena LEUSCHNER, Ian BURTON, Tilman KERL, Rita SEVASTJANOVA, 2023. The performance of international organizations : a new measure and dataset based on computational text analysis of evaluation reports. In: The Review of International Organizations. Springer. 2023, 18(4), pp. 753-776. ISSN 1559-7431. eISSN 1559-744X. Available under: doi: 10.1007/s11558-023-09489-1
BibTex
@article{Eckhard2023-05-06perfo-66865,
  year={2023},
  doi={10.1007/s11558-023-09489-1},
  title={The performance of international organizations : a new measure and dataset based on computational text analysis of evaluation reports},
  number={4},
  volume={18},
  issn={1559-7431},
  journal={The Review of International Organizations},
  pages={753--776},
  author={Eckhard, Steffen and Jankauskas, Vytautas and Leuschner, Elena and Burton, Ian and Kerl, Tilman and Sevastjanova, Rita},
  note={DFG under Grant Agreement No EC 506/2–1}
}
RDF
<rdf:RDF
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:bibo="http://purl.org/ontology/bibo/"
    xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#"
    xmlns:foaf="http://xmlns.com/foaf/0.1/"
    xmlns:void="http://rdfs.org/ns/void#"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > 
  <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/66865">
    <dspace:hasBitstream rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/66865/1/Eckhard_2-mz3jdaqm1vf66.pdf"/>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <dc:creator>Kerl, Tilman</dc:creator>
    <dcterms:abstract>International organizations (IOs) of the United Nations (UN) system publish around 750 evaluation reports per year, offering insights on their performance across project, program, institutional, and thematic activities. So far, it was not feasible to extract quantitative performance measures from these text-based reports. Using deep learning, this article presents a novel text-based performance metric: We classify individual sentences as containing a negative, positive, or neutral assessment of the evaluated IO activity and then compute the share of positive sentences per report. Content validation yields that the measure adequately reflects the underlying concept of performance; convergent validation finds high correlation with human-provided performance scores by the World Bank; and construct validation shows that our measure has theoretically expected results. Based on this, we present a novel dataset with performance measures for 1,082 evaluated activities implemented by nine UN system IOs and discuss avenues for further research.</dcterms:abstract>
    <dc:contributor>Kerl, Tilman</dc:contributor>
    <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2023-05-11T07:39:17Z</dc:date>
    <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/66865"/>
    <foaf:homepage rdf:resource="http://localhost:8080/"/>
    <dc:creator>Burton, Ian</dc:creator>
    <dc:contributor>Jankauskas, Vytautas</dc:contributor>
    <dcterms:issued>2023-05-06</dcterms:issued>
    <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/>
    <dc:contributor>Eckhard, Steffen</dc:contributor>
    <dc:creator>Eckhard, Steffen</dc:creator>
    <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2023-05-11T07:39:17Z</dcterms:available>
    <dcterms:rights rdf:resource="http://creativecommons.org/licenses/by/4.0/"/>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <dc:creator>Sevastjanova, Rita</dc:creator>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/42"/>
    <dcterms:hasPart rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/66865/1/Eckhard_2-mz3jdaqm1vf66.pdf"/>
    <dc:creator>Leuschner, Elena</dc:creator>
    <dc:contributor>Sevastjanova, Rita</dc:contributor>
    <dc:contributor>Burton, Ian</dc:contributor>
    <dc:language>eng</dc:language>
    <dc:rights>Attribution 4.0 International</dc:rights>
    <dcterms:title>The performance of international organizations : a new measure and dataset based on computational text analysis of evaluation reports</dcterms:title>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/42"/>
    <dc:contributor>Leuschner, Elena</dc:contributor>
    <dc:creator>Jankauskas, Vytautas</dc:creator>
  </rdf:Description>
</rdf:RDF>
Interner Vermerk
xmlui.Submission.submit.DescribeStep.inputForms.label.kops_note_fromSubmitter
Kontakt
URL der Originalveröffentl.
Prüfdatum der URL
Prüfungsdatum der Dissertation
Finanzierungsart
Kommentar zur Publikation
DFG under Grant Agreement No EC 506/2–1
Allianzlizenz
Corresponding Authors der Uni Konstanz vorhanden
Internationale Co-Autor:innen
Universitätsbibliographie
Ja
Begutachtet
Ja
Link zu Forschungsdaten
Beschreibung der Forschungsdaten
All data generated and analyzed during the current study, including the language model developed as part of this study
Diese Publikation teilen