We explore the floating-point arithmetic implemented in the NVIDIA tensor cores, which are hardware accelerators for mixed-precision matrix multiplication available on the Volta, Turing, and Ampere microarchitectures. Using Volta V100, Turing T4, and Ampere A100 graphics cards, we determine what precision is used for the intermediate results, whether subnormal numbers are supported, what rounding mode is used, in which order the operations underlying the matrix multiplication are performed, and whether partial sums are normalized. These aspects are not documented by NVIDIA, and we gain insight by running carefully designed numerical experiments on these hardware units. Knowing the answers to these questions is important if one wishes to: (1) accurately simulate NVIDIA tensor cores on conventional hardware; (2) understand the differences between results produced by code that utilizes tensor cores and code that uses only IEEE 754-compliant arithmetic operations; and (3) build custom hardware whose behavior matches that of NVIDIA tensor cores. As part of this work we provide a test suite that can be easily adapted to test newer versions of the NVIDIA tensor cores as well as similar accelerators from other vendors, as they become available. Moreover, we identify a non-monotonicity issue affecting floating point multi-operand adders if the intermediate results are not normalized after each step.

中文翻译：

---

NVIDIA张量核心的数值行为

我们探索了在NVIDIA张量内核中实现的浮点算法，这是在Volta，Turing和Ampere微体系结构上可用的用于混合精度矩阵乘法的硬件加速器。使用Volta V100，Turing T4和Ampere A100图形卡，我们确定中间结果使用的精度是多少，是否支持非正规数，使用哪种舍入模式，以何种顺序执行矩阵乘法的基础以及是否部分和被标准化。NVIDIA未记录这些方面，我们通过在这些硬件单元上进行精心设计的数值实验来获得深入的了解。如果愿意，了解这些问题的答案很重要：（1）在常规硬件上准确模拟NVIDIA张量内核；（2）了解使用张量核的代码与仅使用符合IEEE 754的算术运算的代码产生的结果之间的区别；（3）构建行为与NVIDIA张量内核相匹配的自定义硬件。作为这项工作的一部分，我们提供了一个测试套件，可以轻松地对其进行测试，以测试NVIDIA张量内核的较新版本以及其他供应商提供的类似加速器。此外，如果中间结果在每个步骤之后都未标准化，我们将确定一个影响浮点多操作数加法器的非单调性问题。作为这项工作的一部分，我们提供了一个测试套件，可以轻松地对其进行测试，以测试NVIDIA张量内核的较新版本以及其他供应商提供的类似加速器。此外，如果中间结果在每个步骤之后都未标准化，我们将确定一个影响浮点多操作数加法器的非单调性问题。作为这项工作的一部分，我们提供了一个测试套件，可以轻松地对其进行测试，以测试NVIDIA张量内核的较新版本以及其他供应商提供的类似加速器。此外，如果中间结果在每个步骤之后都未标准化，我们将确定一个影响浮点多操作数加法器的非单调性问题。