creating a custom TFDS dataset

Question

I would like to create a custom tensorflow dataset for summarization task. I have a set of reports with three gold summaries for every report. All the data is in (.txt) format. I would like to create a TFDS where the key is the report and the value is the summary. So I will have this format: (report1 , summary11) (report1 , summary12) (report1 , summary13) (report2 , summary21) (report2 , summary22) (report2 , summary23)

Is there any solution that helps me achieve this task. I checked the official documentation on the tensorflow website and it wasn't useful for me.

Thank you !

Answer 1

A custom iterator for traversing through data files can be achieved through the generators. As tf datasets retrieves input from generator you could modify the generator code to return the results

def generator(tuples):
   for tuple in tuples:
      yield tuple[1]