International Organization Evaluation Report Dataset (IOEval) and replication data for ‘The Performance of International Organizations : A New Measure and Dataset Based on Computational Text Analysis of Evaluation Reports, Review of International Organizations'

Abstract

Eckhard, Steffen; Jankauskas, Vytautas; Leuschner, Elena; Burton, Ian; Kerl, Tilman; Sevastjanova, Rita, 2023, "International Organization Evaluation Report Dataset (IOEval) and replication data for ‘The Performance of International Organizations: A New Measure and Dataset Based on Computational Text Analysis of Evaluation Reports’", https://doi.org/10.7910/DVN/0SI2VX, Harvard Dataverse, V1, UNF:6:fBGGclS7HUPoO8PEGwGFZg== [fileUNF]

This dataset contains: • the sentence-level text of 1,082 evaluation reports published by nine international organizations of the United Nations (UN) system between 2012 to 2021); • a fine-tuned BERT language model that allows classifying individual sentences in evaluation reports as containing a positive, negative or neutral assessment of the evaluated activity; • and replication files for our publication DOI: 10.1007/s11558-023-09489-1.

When using the data, please cite: “Eckhard, Steffen; Jankauskas, Vytautas; Leuschner, Elena; Burton, Ian; Kerl, Tilman; Sevastjanova, Rita (2023). The Performance of International Organizations: A New Measure and Dataset Based on Computational Text Analysis of Evaluation Reports. Review of International Organizations, DOI: 10.1007/s11558-023-09489-1.”

Summary of the IOEval Dataset:

The IOEval dataset contains the sentence-level text of 1,082 evaluation reports published by nine international organizations of the United Nations (UN) system between 2012 to 2021. Raw text was cleaned by applying standard procedures of natural language processing (e.g., removal of special characters and numbers) and split into sentences.

The text is taken from evaluation reports by International Labor Organization (ILO), the UN Development Program (UNDP), the UN International Children's Emergency Fund (UNICEF), the Food and Agricultural Organization (FAO), the UN Educational, Scientific and Cultural Organization (UNESCO), the World Health Organization (WHO), the International Organization for Migration (IOM), the UN High Commissioner for Refugees (UNHCR) and the UN Entity for Gender Equality and the Empowerment of Women (UN WOMEN).

At a sentence level, the dataset specifies to which text section a sentence belongs (executive summary, main text, appendix).

The IOEval dataset also includes metadata variables at the level of reports: report title, publication date, evaluation type (project, program, institutional or thematic), evaluation level (country (specifying its name), regional, global), and commissioning unit (centralized or decentralized).

Summary of language model:

The fine-tuned BERT language model (Devlin et al., 2019) allows classifying individual sentences in evaluation reports as containing a positive, negative or neutral assessment of the evaluated activity. It was fine-tuned and evaluated on around 10,000 hand-coded sentences from evaluation reports, reaching a recall of 89 percent.