Tensor conversion requested dtype error while using data.experimental.CsvDataset

16 views Asked by At

I use this simple code to extract csv files with tf.data.dataset.

def load_my_csv(path):
    return tf.data.experimental.CsvDataset(
                path,
                record_defaults=[tf.string, tf.int32, tf.int32, tf.int32, tf.int32],
                header=True,
                field_delim=',',
                select_cols=[0,2,3,4,5,6]
    )

list_csv=tf.data.Dataset.list_files(test_samples + '*.csv',shuffle=False)
dataset = list_csv.interleave(load_my_csv, cycle_length=1)


when I call this command:

ds = ds.map(lambda *items: tf.stack(items))

I get this error:

TypeError: Tensors in list passed to 'values' of 'Pack' Op have types [string, int32, int32, int32, int32] that don't all match.
ValueError: values_1: Tensor conversion requested dtype string for Tensor with dtype int32: <tf.Tensor 'args_1:0' shape=() dtype=int32>

This is clear that the prob is from "record_defaults" tf.string, My first column is a string format and all other are int32 as you can see here a dummy csv sample:

x1,x2,x3,x4,y,yy,yyy
a1,11,111,11111,-1,-11,-111
a1,22,222,2222,-2,-22,-222
a1,33,333,3333,-3,-33,-333
a2,44,444,4444,-4,-44,-444
a2,55,555,5555,-5,-55,-555
a2,66,666,6666,-6,-66,-666
a2,77,777,7777,-7,-77,-777
a2,88,888,8888,-8,-88,-888
a2,99,999,9999,-9,-99,-999

When I remode the first columns ("string" one) all work properly. So the question is why it is not working?

0

There are 0 answers