How to import a PostgreSQL table with arrays into AVRO format?

218 views Asked by At

I have an SQL database with some columns of type TEXT [] (array of strings).

I'm importing the database into AVRO format and saving it to HDFS using the sqoop command:

sqoop import-all-tables
    -Dmapreduce.job.user.classpath.first=true
    --connect jdbc:postgresql://localhost/project
    --username postgres
    --warehouse-dir /project
    --as-avrodatafile
    --compression-codec=snappy
    --outdir output/avsc 
    --m 1

And getting the error:

23/05/08 09:33:51 ERROR orm.ClassWriter: Cannot resolve SQL type 2003
23/05/08 09:33:51 ERROR orm.ClassWriter: No Java type for SQL type 2003 for column genres

I tried to fix it by adding the --map-column-hive 'genres'=ARRAY\<STRING\> flag. Then sqoop printed this flah must be used with --hive-import.

The resulting command

sqoop import-all-tables
    -Dmapreduce.job.user.classpath.first=true
    --connect jdbc:postgresql://localhost/project
    --username postgres
    --warehouse-dir /project
    --as-avrodatafile
    --compression-codec=snappy
    --outdir output/avsc 
    --m 1
    --map-column-hive 'genres'=ARRAY\<STRING\>
    --hive-import

is giving the error: Hive import is not compatible with importing into AVRO format.

Is there a way to import SQL tables with arrays into AVRO format?

I'm using Hive 1.2.1000.2.6.5.0-292

0

There are 0 answers