Im facing a problem with pandas.to_sql().
My db : ibm db2 (bigsql)
table : hadoop table stored as parquet
Im connecting to bigsql and load some data from other db through pandas like this.
read = pd.read_sql(text(item['load_sql']),session.bind) read.to_sql(table_,dst_session.bind,schema='dev',if_exists='append',chunksize=1424, index=False, method='multi')
It's everything ok when chunksize is less then 1425, but when its 1425 or higher im getting an error : (ibm_db_dbi.ProgrammingError) ibm_db_dbi::ProgrammingError: Exception('Statement Execute Failed: [IBM][CLI Driver] Wrong number of parameters. SQLSTATE=07001 SQLCODE=-99999')
I think its not related with sql command string size, becouse with chunksize=1424 is not even 1 mb.
Is there some kind of restriciton to chunksize with db2 / hadoop ? Someone knows why im getting this error ?
Resolved ?: I dont use to_sql() anymore - im creating whole insert command as a string and send it to db. It seems that only limitation in that option is string size but the insertion speed increased and i can put more rows in this command than chunksize allowed before.