Explode 2 columns (2 lists) in the same time in pyspark

1.2k views Asked by At

What I have:

fix_array = [4,5,6]

I put it in the list2 column by:

df = df.withColumn('list2', F.array(map(lambda x: F.lit(x), fix_array)))

So, I have:

| id |  list1  |  list2  |
--------------------------
| 1  | [1,2,3] | [4,5,6] |
| 2  | [8,9,10]| [4,5,6] |

I want:

| id |  new_col1  | new_col2 |
------------------------------
| 1  |      1     |    4     |
| 1  |      2     |    5     |
| 1  |      3     |    6     |
| 2  |      8     |    4     |
| 2  |      9     |    5     |
| 2  |      10    |    6     |

As you might understand, the array in list 2 column is a fix array (computed before) but I don't know the values of the array.

Arrays in list1 and list2 have the same sizes.

0

There are 0 answers