List Question
20 TechQA 2024-03-19T20:14:19.877000Float calculation problem. 12.7 - 20 + 7.3 not equal zero
42 views
Asked by Ryal Naufal
How much accuracy is lost due to representing time as a IEEE 754 binary64 number?
38 views
Asked by KamilCuk
ceil(double(int)*double) or ceil(float(int*double))?
114 views
Asked by Le_Square
Fast implementation of complementary error function with 5-digit accuracy
338 views
Asked by njuffa
R: floating point errors in simple math operations
45 views
Asked by Johann
Truncating decimal digits like Excel in python
51 views
Asked by Joseph Rakhimov
Calculating power of (1-1/x)^y for very large x,y
82 views
Asked by user3563894
Model Accuracy Using Transformer
35 views
Asked by Syedumeerr
Why assignment changes the precision of calculation of MATLAB?
74 views
Asked by Moon Traveler
Rounding errors when computing intersection between two lines in python
105 views
Asked by Edoardo Serra
Workarounds for High Floating-Point Error in NumPy Arrays Raised to Powers
179 views
Asked by javery
imshow plotting very large integers, but "dtype object cannot be converted to float"
149 views
Asked by D.R
Precision in SVD calculation through eigenvalues/eigenvectors
46 views
Asked by ATu
Conversion accuracy issues of E57 to LAS in Python using pye57 and laspy
265 views
Asked by Arjan
Without overflow and underflow, for a float number x, is x/4 always equals to x/2/2 (also for x*4 == x*2*2)?
96 views
Asked by wcminipgasker2023
How to properly round to the nearest integer in double-double arithmetic
241 views
Asked by kleite
CPU float32 precision differences between projects in pytorch
31 views
Asked by xaviersjs
Different floating point precision in Matrix multiplication in Python and C++
102 views
Asked by skm
Why explicit Integer conversion doesn't apply as expected
135 views
Asked by MarlonB
python rounding error when sampling variable Y as a function of X with histogram?
81 views
Asked by ClimateUnboxed