Do while loop with GPDB using talend

Question

I have a very large data set in GPDB from which I need to extract close to 3.5 million records. I use this for a flatfile which is then used to load to different tables. I use Talend, and do a select * from table using the tgreenpluminput component and feed that to a tfileoutputdelimited. However due to the very large volume of the file, I run out of memory while executing it on the Talend server.

I lack the permissions of a super user and unable to do a \copy to output it to a csv file. I think something like a do while or a tloop with more limited number of rows might work for me. But my table doesnt have any row_id or uid to distinguish the rows.

Please help me with suggestions how to solve this. Appreciate any ideas. Thanks!

Answer 1

If your requirement is to load data into different tables from one table, then you do not need to go for load into file and then from file to table.

There is a component named tGreenplumRow which allows you to write direct sql queries (DDL and DML queries) in it.

Below is a sample job,

If you notice, there are three insert statements inside this component. It will be executed one by one separated by semicolon.