The puzzles have had their rating generated following largely the same methodology as the training puzzles. However, some aspects of the annotation process were impossible to replicate due to the nature of the dataset. See https://knowledgepit.ml/post/747/ for details.