Import paf using php

Question

I’m working with the royal mail PAF database in csv format (approx 29 million lines), and need to split the data into sql server using php.

Can anyone recommend the best method for this to prevent timeout?

Here is a sample of the data: https://gist.github.com/anonymous/8278066

Answer 1

To disable the script execution time limit, start your script off with this:

set_time_limit(0);

Another problem you will likely run into is a memory limit. Make sure you are reading your file line-by-line or in chunks, rather than the whole file at once. You can do this with fgets().

Answer 2

Start your script with

ini_set('max_execution_time', 0);

Answer 3

The quickest way I found was to use SQL Servers BULK INSERT to load the data directly and unchanged, from the csv files, into matching db import tables. Then do my own manipulation and population of application specific tables from those import tables.

I found BULK INSERT will import the main CSVPAF file, containing nearly 31 million address records in just a few minutes.