Thank you for reading my post in advance.
I'm currently working on a data pipeline that involves extracting binary data from SQL Server 2019, storing it in Blob Storage as multiple Parquet files, and subsequently importing each file into Snowflake.
During the process of loading Parquet files into Snowflake, I encountered the following error:,
Max LOB size (8388608) exceeded, actual size of parsed column is 8615424
This issue arises due to the length limitation for binary columns in Snowflake.
I've been trying to find a solution to migrate varbinary(max) column data from SQL Server to a Snowflake binary column, but haven't been successful so far.
Any suggestions or hints are highly appreciated?
Snowflake scripts to load parquet files
create table if not exists data_table using template (
select array_agg(object_construct(*)) from table(
infer_schema(
location=>'@raw_stage/data_00000.parquet', file_format=>'parquet_format'
)
)
);
copy into data_table
from '@raw_stage/data_00000.parquet'
file_format = 'parquet_format'
on_error = 'ABORT_STATEMENT'
match_by_column_name = case_insensitive;