FP32 hypotf_u05
and FP64 hypot_u05
are inaccurate near underflow
#469
Labels
hypotf_u05
and FP64 hypot_u05
are inaccurate near underflow
#469
The current
Sleef_hypotf_u05
andSleef_hypot_u05
functions are less accurate than the advertised 0.5001 ulp for input pairs whose absolute maximum approaches the underflow threshold.Based on random testing, the maximum ulps error follows roughly:
that is, the maximum error climbs to 1.0 ulp exponentially as the maximum of x and y approaches the subnormals.
E.g. we can find inputs that produce errors greater than the 0.5001 ulp advertised accuracy when the maximum absolute value of the inputs is less than and
0x1p-112
for fp32:and less than
0x1p-1008
for fp64:The text was updated successfully, but these errors were encountered: