Not able to Select,Join on a attribute in dataset Apache spark java

155 views Asked by At

We are not able to perform any of the select join and other operation in apachae spark java for the field uniqueidrowid

Bellow is code using to join two dataset.

  ManufacturerFileContent = ManufacturerFileContent .join(masterFileContent ,masterFileContent .col("uniqueidrowid").equalTo(ManufacturerFileContent .col("uniqueidrowidManufacturer")));

We even tried adding backtick(`) and also renamed the column to different name and we also tried adding separate column monotonically_increasing_id and then joining using that common column all efforts are failed .

    +-----------------------+--------------------+------+--------------------+--------------------------+
    |uniqueidrowid          |               prods|Prefix|             Prod ID| Normalized Manufacturer  |
    +-----------------------+--------------------+------+--------------------+--------------------------+
    |     0x000000000835ae0a|        aa 96K107   |   QQa|            9396K107|                      null|
    |     0x000000000835ae0b|     ff S-641-F-88  |   THa|         dS-641-F-88|                      null|
    |     0x000000000835ae0c|    gg Sff RIG-2549 |   GaE|        SfM RIG-2549|                      null|
    -----------------------------------------------------------------------------------------------------

we tried monotonically increasing to create new column to join but the problem is not solved

masterFileContent = masterFileContent.repartition(1).withColumn("rowidsource", monotonically_increasing_id());

org.apache.spark.sql.AnalysisException: Cannot resolve column name "uniqueidrowid" among (uniqueidrowid, prods, Prefix,...

0

There are 0 answers