What I have:
fix_array = [4,5,6]
I put it in the list2 column by:
df = df.withColumn('list2', F.array(map(lambda x: F.lit(x), fix_array)))
So, I have:
| id | list1 | list2 |
--------------------------
| 1 | [1,2,3] | [4,5,6] |
| 2 | [8,9,10]| [4,5,6] |
I want:
| id | new_col1 | new_col2 |
------------------------------
| 1 | 1 | 4 |
| 1 | 2 | 5 |
| 1 | 3 | 6 |
| 2 | 8 | 4 |
| 2 | 9 | 5 |
| 2 | 10 | 6 |
As you might understand, the array in list 2 column is a fix array (computed before) but I don't know the values of the array.
Arrays in list1 and list2 have the same sizes.