Accepted short paper at WI-IAT 2020
Interpretable and Fair Comparison of Link Prediction or Entity Alignment Methods
06.11.2020
Authors
Max Berrendorf, Evgeniy Faerman, Laurent Vermue, Volker Tresp
The 2020 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT'20)
14–17 December 2020, Virtual
Abstract
Link Prediction and Entity Alignment methods are usually evaluated in a rank-based setting. We show that there are multiple competing implementations on how to deal with the case of equal scores, only one of which is preferrable. Moreover, some of these rank-based metrics get automatically better for smaller evaluation set sizes, and we show this case occurs practially. We propose a new adjusted version of the mean rank metric normalizing for chance, which is comparable for differently sized evaluation sets.