Oracle table column json data to csv in python

40 views Asked by At

Could you please help me how to achieve below requirement in Python.

I have oracle table as source and will have no of records and one json document in one record at table level.

Table1:

idAsString,data_extract,created_date
123456789,json format data,07-Sept-2022 

json format data

  {
    "id": {
      "timestamp": 1662115897,
      "date": 1662115897000
    },
    "emp": {
      "firstname": "python",
      "attributes": {
        "empid": "12345",
        "type": "F"
      },
      "dept": {
        "code": "001",
        "description": "IT"
      }
    },
    "Technology": [
      {
        "techname": "devops",
        "teamname": "12",
        "attributes": {
          "checkin": {
            "status": "Registered"
          }
        }
      },
      {
        "techname": "mongodb",
        "teamname": "13",
        "attributes": {
          "checkin": {
            "status": "cancelled"
          }
        }
      },
      {
        "techname": "python3",
        "teamname": "11",
        "attributes": {
          "checkin": {
            "status": "confirmed"
          }
        }
      }
    ],
    "docversion": 1,
    "idAsString": "123456789"
  }

I want out as .csv file with all above columns .

output.csv

idAsString,docversion,id,timestamp,date,emp.firstname,attributes.empid,attributes.type,attributes.dept,dept.code,dept.description,Technology.techname,Technology.teamname,Technology.attributes,Technology.attributes.checkin,Technology.attributes.status
123456789,1,'',1662115897,1662115897000,python,12345,F,001,IT,devops,12,Registered
123456789,1,'',1662115897,1662115897000,python,12345,F,001,IT,mongodb,13,cancelled
123456789,1,'',1662115897,1662115897000,python,12345,F,001,IT,python3,11,confirmed

My source is oracle database table and my target is .csv file or target also oracle table.

Below is script which write but nested condition is not giving proper results.

python script

    #!/usr/bin/env python import cx_Oracle
    import config
    import json
    from pathlib import Path
    import pandas as pd
    from pprint import pprint
    from pandas.io.json import json_normalize
    
    data_folder=Path("/etc/data")
    csv_file_name = data_folder/'output.csv'
    
    def OutputTypeHandler (cursor, name, defaultType, size, precision, scale): 
        if defaultType cx_Oracle.CLOB:
            return cursor.var (cx_Oracle. LONG STRING, arraysize = cursor.arraysize)
        elif defaultType cx_Oracle.BLOB:
        return cursor.var (cx_Oracle.LONG_BINARY, arraysize = cursor.arraysize)
    
    db_conn = cx_Oracle.connect (user-config.dbuser, password-config.dbpasswd, dsn-config.dbsid)
    print (db_conn.version)
    db_conn.outputtypehandler = OutputTypeHandler
    cursor db_conn.cursor()
    sql = "select * from scott.Table1"
    cursor.execute(sql)
    
    colnames=[c[0] for c in cursor.description]
    print (colnames)
    for row in cursor:
        json_content= json.loads (row [colnames.index ('data_extract')])
        df_flat = pd.io.json.json_normalize (json_content)  
    
    df flat.to_csv (csv_file_name)
    print()
    print('executed')
    cursor.close()
    db_conn.close()
0

There are 0 answers