Convert OffsetDateTimeUdt to date or string

Question

I am working on Pyspark. I have a column end_date, I want to make some work on it. But I can't due to his type: OffsetDateTimeUdt.

It's possible to convert this type in date or string ?

Exemple of value:2021-08-15T00:00:00.000Z

If you have any idea please let me know :)

Thanks in advance

Answer 1

Convert into timestamp by ignoring the milliseconds like below

x = spark.createDataFrame([("2021-08-15T00:00:00.000Z",)], ['date_str'])
x.select(x.date_str, from_unixtime(unix_timestamp(x.date_str,"yyyy-MM-dd'T'HH:mm:ss'.000Z'"))).show(truncate=False)

output:

+------------------------+------------------------------------------------------------------------------------------+
|date_str                |from_unixtime(unix_timestamp(date_str, yyyy-MM-dd'T'HH:mm:ss'.000Z'), yyyy-MM-dd HH:mm:ss)|
+------------------------+------------------------------------------------------------------------------------------+
|2021-08-15T00:00:00.000Z|2021-08-15 00:00:00                                                                       |
+------------------------+------------------------------------------------------------------------------------------+