I have a txt file with about 100 million records (numbers). I am reading this file in Python and inserting it to MySQL database using simple insert statement from python. But its taking very long and looks like the script wouldn't ever finish. What would be the optimal way to carry out this process ? The script is using less than 1% of memory and 10 to 15% of CPU.
Any suggestions to handle such large data and insert it efficiently into database, would be greatly appreciated.
Thanks.
Sticking with python, you may want to try creating a list of tuples from your input and using the execute many statement from the python mysql connector.
If the file is too big, you can use a generator to chunk it into something more digestible.
http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-executemany.html