Oracle CDC source connector Date based column is not taking sysdate value in kafka topic

194 views Asked by At

I was working on a Connector development project, where I have to use Oracle CDC as a source connector, and Target connector was JDBC sink connector. where I was able to do data transfer from source to target end successfully, But when I tried to change any DATE based column at source end, then those changes were not reflecting at target.(mentioning here DATE based column has 'sysdate' as default value), but only the date value changes reflected at target end.

what can I do so that my kafka topic can pick both date and time value for a date based column and same should reflect at target DB end.

sharing the source table structure below>>

enter image description here

*both target and source have same table structure.

Solution I have tried as per Confluent kafka cdc documentation. mentioning below

solution1: oracle.date.mapping="timestamp"

thought of, if by converting date based columns to timestamp epoch value, might solve this issue,but it has no use in this case.

solution2:

"transforms": "TimestampConverter", "transforms.TimestampConverter.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value", "transforms.TimestampConverter.format": "yyyy-MM-dd", "transforms.TimestampConverter.target.type": "string"

tried to convert timestamp converter transform, that also didn't work.

1

There are 1 answers

0
Pratyush Samantaray On

As mentioned in the documentation, date based columns can not pick both date and time.
enter image description here

link>> https://docs.confluent.io/kafka-connectors/oracle-cdc/current/overview.html

After doing some workaround, get to achieve the desired output

sharing the config here>>

"transforms":"cast", "transforms.Cast.type":"org.apache.kafka.connect.transforms.Cast$Value", "transforms.cast.spec": "DOB:string"

now what it will do, it will convert the Date type into String format while polling into kafka topic, and same casting needed to bed used at sink connector, so that target table can also get the similar string data at target end.

sharing output>> source data received at kafka topic: enter image description here

target output received for the same: enter image description here