Federated learning (FL) trains a machine learning model on mobile devices in a distributed manner using each device's private data and computing resources. A critical issues is to evaluate individual users' contributions so that (1) users' effort in model training can be compensated with proper incentives and (2) malicious and low-quality users can be detected and removed. The state-of-the-art solutions require a representative test dataset for the evaluation purpose, but such a dataset is often unavailable and hard to synthesize. In this paper, we propose a method called Pairwise Correlated Agreement (PCA) based on the idea of peer prediction to evaluate user contribution in FL without a test dataset. PCA achieves this using the statistical correlation of the model parameters uploaded by users. We then apply PCA to designing (1) a new federated learning algorithm called Fed-PCA, and (2) a new incentive mechanism that guarantees truthfulness. We evaluate the performance of PCA and Fed-PCA using the MNIST dataset and a large industrial product recommendation dataset. The results demonstrate that our Fed-PCA outperforms the canonical FedAvg algorithm and other baseline methods in accuracy, and at the same time, PCA effectively incentivizes users to behave truthfully.

中文翻译：

---

联邦学习中用户贡献的无数据评估

联邦学习 (FL) 使用每个设备的私有数据和计算资源以分布式方式在移动设备上训练机器学习模型。一个关键问题是评估个人用户的贡献，以便（1）可以通过适当的激励来补偿用户在模型训练中的努力，以及（2）可以检测和删除恶意和低质量的用户。最先进的解决方案需要具有代表性的测试数据集用于评估目的，但这样的数据集通常不可用且难以合成。在本文中，我们基于对等预测的思想提出了一种称为成对相关协议（PCA）的方法，以在没有测试数据集的情况下评估 FL 中的用户贡献。PCA 使用用户上传的模型参数的统计相关性来实现这一点。然后我们应用 PCA 来设计 (1) 一种称为 Fed-PCA 的新联邦学习算法，以及 (2) 一种保证真实性的新激励机制。我们使用 MNIST 数据集和大型工业产品推荐数据集评估 PCA 和 Fed-PCA 的性能。结果表明，我们的 Fed-PCA 在准确性上优于规范的 FedAvg 算法和其他基线方法，同时，PCA 有效地激励用户诚实行事。