Kolesnyk A.S., Khairova N.F.
Justification for the use of Cohen's kappa statistic in experimental studies of NLP and Text Mining

Modern metrics for evaluating agreement coefficients between the experimental results and expert opinion are compared, and the possibility of using these metrics in experimental research in automatic text processing by machine learning methods is assessed. The choice of Cohen’s kappa coefficient as a measure of expert opinion agreement in the NLP and Text Mining problems is justified. An example of using Cohen’s kappa coefficient for evaluating the level of agreement between the opinion of an expert and the results of ML classification and the measure of agreement of expert opinions in the alignment of sentences of the Kazakh-Russian parallel corpus is given. Based on this analysis, it is proved that Cohen’s kappa coefficient is one of the best statistical methods for determining the level of agreement in experimental studies due to its ease of use, computing simplicity, and high accuracy of the results. © 2022, Springer Science+Business Media, LLC, part of Springer Nature.

Keywords: agreement statistic, Cohen’s kappa statistic, NLP, parallel corpus, text classification with machine learning, Text Mining, Data mining, Machine learning, Natural language processing systems, Text processing, Agreement statistic, Cohen’s kappa statistic, Expert opinion, Kappa coefficient, Kappa statistic, Machine-learning, Parallel corpora, Text classification, Text classification with machine learning, Text-mining, Classification (of information)

Cite:
Kolesnyk A.S., Khairova N.F. (2022). Justification for the use of Cohen's kappa statistic in experimental studies of NLP and Text Mining . Cybernetics and Systems Analysis, 58 (2), 143–153. doi: https://doi.org/10.1007/s10559-022-00460-3 http://jnas.nbuv.gov.ua/article/UJRN-0001313069 [In Ukrainian].