当前位置: X-MOL 学术IMA J. Numer. Anal. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Accurately computing the log-sum-exp and softmax functions
IMA Journal of Numerical Analysis ( IF 2.1 ) Pub Date : 2020-08-19 , DOI: 10.1093/imanum/draa038
Pierre Blanchard 1 , Desmond J Higham 2 , Nicholas J Higham 1
Affiliation  

Evaluating the log-sum-exp function or the softmax function is a key step in many modern data science algorithms, notably in inference and classification. Because of the exponentials that these functions contain, the evaluation is prone to overflow and underflow, especially in low-precision arithmetic. Software implementations commonly use alternative formulas that avoid overflow and reduce the chance of harmful underflow, employing a shift or another rewriting. Although mathematically equivalent, these variants behave differently in floating-point arithmetic and shifting can introduce subtractive cancellation. We give rounding error analyses of different evaluation algorithms and interpret the error bounds using condition numbers for the functions. We conclude, based on the analysis and numerical experiments, that the shifted formulas are of similar accuracy to the unshifted ones, so can safely be used, but that a division-free variant of softmax can suffer from loss of accuracy.

中文翻译:

准确计算log-sum-exp和softmax函数

在许多现代数据科学算法中,尤其是在推理和分类中,评估log-sum-exp函数或softmax函数是关键的一步。由于这些函数包含的指数,因此评估容易上溢和下溢,尤其是在低精度算术中。软件实现通常使用替代公式,这些公式可以避免上溢,并通过轮班或其他重写来减少有害下溢的机会。尽管在数学上是等效的,但是这些变体在浮点算术中的行为有所不同,并且移位会引入减法抵消。我们给出了不同评估算法的舍入误差分析,并使用函数的条件编号来解释误差范围。我们基于分析和数值实验得出结论,
更新日期:2020-08-20
down
wechat
bug