We consider the problem of evaluating representations of data for use in solving a downstream task. We propose to measure the quality of a representation by the complexity of learning a predictor on top of the representation that achieves low loss on a task of interest, and introduce two methods, surplus description length (SDL) and $\varepsilon$ sample complexity ($\varepsilon$SC). In contrast to prior methods, which measure the amount of information about the optimal predictor that is present in a specific amount of data, our methods measure the amount of information needed from the data to recover an approximation of the optimal predictor up to a specified tolerance. We present a framework to compare these methods based on plotting the validation loss versus training set size (the "loss-data" curve). Existing measures, such as mutual information and minimum description length probes, correspond to slices and integrals along the data-axis of the loss-data curve, while ours correspond to slices and integrals along the loss-axis. We provide experiments on real data to compare the behavior of each of these methods over datasets of varying size along with a high performance open source library for representation evaluation at https://github.com/willwhitney/reprieve.

中文翻译：

---

通过学习低损失预测器的复杂性评估表示

我们考虑评估用于解决下游任务的数据表示的问题。我们建议通过在感兴趣的任务上实现低损失的表示之上学习预测器的复杂性来衡量表示的质量，并引入了两种方法，剩余描述长度（SDL）和 $\varepsilon$ 样本复杂度（ $\varepsilon$SC)。与测量特定数据量中存在的最佳预测器的信息量的先前方法相比，我们的方法测量数据所需的信息量，以将最佳预测器的近似值恢复到指定的容差. 我们提出了一个框架，基于绘制验证损失与训练集大小（“损失数据”曲线）来比较这些方法。现有措施，例如互信息和最小描述长度探针，对应于沿损失数据曲线数据轴的切片和积分，而我们的则对应于沿损失轴的切片和积分。我们提供了对真实数据的实验，以比较每种方法在不同大小的数据集上的行为以及用于表征评估的高性能开源库，网址为 https://github.com/willwhitney/reprieve。