I use this simple code to extract csv files with tf.data.dataset.
def load_my_csv(path):
return tf.data.experimental.CsvDataset(
path,
record_defaults=[tf.string, tf.int32, tf.int32, tf.int32, tf.int32],
header=True,
field_delim=',',
select_cols=[0,2,3,4,5,6]
)
list_csv=tf.data.Dataset.list_files(test_samples + '*.csv',shuffle=False)
dataset = list_csv.interleave(load_my_csv, cycle_length=1)
when I call this command:
ds = ds.map(lambda *items: tf.stack(items))
I get this error:
TypeError: Tensors in list passed to 'values' of 'Pack' Op have types [string, int32, int32, int32, int32] that don't all match.
ValueError: values_1: Tensor conversion requested dtype string for Tensor with dtype int32: <tf.Tensor 'args_1:0' shape=() dtype=int32>
This is clear that the prob is from "record_defaults" tf.string, My first column is a string format and all other are int32 as you can see here a dummy csv sample:
x1,x2,x3,x4,y,yy,yyy
a1,11,111,11111,-1,-11,-111
a1,22,222,2222,-2,-22,-222
a1,33,333,3333,-3,-33,-333
a2,44,444,4444,-4,-44,-444
a2,55,555,5555,-5,-55,-555
a2,66,666,6666,-6,-66,-666
a2,77,777,7777,-7,-77,-777
a2,88,888,8888,-8,-88,-888
a2,99,999,9999,-9,-99,-999
When I remode the first columns ("string" one) all work properly. So the question is why it is not working?