A research paper (10.1109/ACCESS.2020.3046293) and web tool (https://www.lucami.org/en/WGT/) on Weak Ground Truth Determination of Continuous Human-Rated Data. The article presents a novel weak ground truth (WGT) determination and rater bias removal procedure on continuous human-rated data. The notion of WGT is essential in cases where there is no direct empirical evidence for an observed construct and human annotations provide the most reliable means for determining the ground truth. The core idea behind the proposed procedure is to transform the ratings to reduce rater bias, maximize interrater agreement, and improve WGT.
The tool available at https://www.lucami.org/en/WGT/ allows researchers to determine their weak ground truth and remove a part of the bias introduced by raters.