We are running MonetDB on an OpenIndiana fileserver (Xenon E5-1620 / 16GB RAM, RAM will be ugraded soon). We merge many small csv files to two big ones.
The first is around 25GB but the second is around 125GB. Importing the first one works fine but the second does not work, the server just shuts down.
Why does this happen? Is there any way around this, except splitting the file? If splitting is unavoidable, is there a way to load the csvs using a loop?
--
How can I do the debugging steps described here when the whole system crashes? Not only the database crashes, the whole system shuts down.
--
We meanwhile found a way around this problem. We split the big file into five smaller ones à 25GB and imported them using a .sql file which contains five COPY INTO statements.