Float precision problem in metal shading language

678 views Asked by At

I pass a float param with value 0.00291545 to fragment shader. But fragment function got n/a.

swift code

fragment shader

In Xcode debugger's "Bound Resources" view, it shows that the value is 0.003

Bound Resources

In Xcode shader debugger view, I made some tests. It shows that 0.0004 will be cast to 0 and 0.0005 will be casted to 0.001. The float value behaves like half value.

test code in shader debugger

So, my question is:

  1. Why 0.00291545 is first casted to 0.003, then casted to 0 ?

  2. Why float value has the same precision with half ?

  3. How could I use float value that has normal precision ?

1

There are 1 answers

0
blabla On

I found what the problem was!

Xcode debugger misled me.

0.00291545 was not casted to 0.003 and was not casted to n/a. The debugger' display is not accurate, but the value is accurate actually.

Thanks for Ken Thomases