We are not able to perform any of the select join and other operation in apachae spark java for the field uniqueidrowid
Bellow is code using to join two dataset.
ManufacturerFileContent = ManufacturerFileContent .join(masterFileContent ,masterFileContent .col("uniqueidrowid").equalTo(ManufacturerFileContent .col("uniqueidrowidManufacturer")));
We even tried adding backtick(`) and also renamed the column to different name and we also tried adding separate column monotonically_increasing_id and then joining using that common column all efforts are failed .
+-----------------------+--------------------+------+--------------------+--------------------------+
|uniqueidrowid | prods|Prefix| Prod ID| Normalized Manufacturer |
+-----------------------+--------------------+------+--------------------+--------------------------+
| 0x000000000835ae0a| aa 96K107 | QQa| 9396K107| null|
| 0x000000000835ae0b| ff S-641-F-88 | THa| dS-641-F-88| null|
| 0x000000000835ae0c| gg Sff RIG-2549 | GaE| SfM RIG-2549| null|
-----------------------------------------------------------------------------------------------------
we tried monotonically increasing to create new column to join but the problem is not solved
masterFileContent = masterFileContent.repartition(1).withColumn("rowidsource", monotonically_increasing_id());
org.apache.spark.sql.AnalysisException: Cannot resolve column name "uniqueidrowid" among (uniqueidrowid, prods, Prefix,...