I am using below line to check for numbers
when(df[COL_VALUE_DEC].rlike("^[0-9.]+$"), 1)
and datatype of column COL_VALUE_DEC is DECIMAL(38,10)
When the value is 0 in this column, pyspark is returning it as 0E-10 and thus it is not matching the regex.
I tried below trick with format_number but then the actual numbers are not getting matched.
when(format_number(df[COL_VALUE_DEC],2).rlike("^[0-9.]+$"), 1)
How do I solve this problem?
0E-10
0.00
10.13
10
All above should be treated as 1 in my condition.