TechQA.

Question

Reading a parquet file using parquet-tool converting string values to scientific notation

score 95 · Answer 1 · 2024-02-02T15:18:28.853000

0

Answer

95

Views

Reading a parquet file using parquet-tool converting string values to scientific notation

95 views Asked by SwapnilM At 02 February 2024 at 15:18

score 46 · Answer 2 · 2024-01-23T16:20:30.933000

Is there a painless way to migrate existing Parquet files to a new schema? I wish to update an AWS Glue table column data type

46 views Asked by Andrew Parsons At 23 January 2024 at 16:20

score 13 · Answer 3 · 2023-12-21T16:40:13.670000

pyspark structural streaming one column dynamic json save to elasticsearch as json object

13 views Asked by Baitanik At 21 December 2023 at 16:40

score 124 · Answer 4 · 2023-12-02T09:27:56.617000

pyspark error while writing data to existing hive table

124 views Asked by dheeraj saini At 02 December 2023 at 09:27

score 197 · Answer 5 · 2023-11-21T10:36:40.120000

ValueError: Unable to parse datatype from schema. Could not parse datatype: interval year

197 views Asked by ZygD At 21 November 2023 at 10:36

score 187 · Answer 6 · 2023-11-08T20:14:58.447000

pyspark - how to add a new element to ArrayType column

187 views Asked by visu At 08 November 2023 at 20:14

score 91 · Answer 7 · 2023-10-25T06:32:38.003000

Need to add headers in existing data frame

91 views Asked by Giri Sankar At 25 October 2023 at 06:32

score 63 · Answer 8 · 2023-10-07T13:02:52.217000

o1566.showString error persists even with os env variables defined with PySpark in jupyter notebook

63 views Asked by user14255281 At 07 October 2023 at 13:02

score 56 · Answer 9 · 2023-10-04T06:00:41.937000

can we merge two parquet files into one which has different datatypes to the columns using the pyspark

56 views Asked by prasanna kumar At 04 October 2023 at 06:00

score 169 · Answer 10 · 2023-09-25T18:10:34.487000

check if rows are already present in pyspark dataset

169 views Asked by x89 At 25 September 2023 at 18:10

score 343 · Answer 11 · 2023-09-17T20:28:08.457000

PySpark DataFrame creation is throwing PySparkTypeError

343 views Asked by Mojdeh Ebrahimi At 17 September 2023 at 20:28

score 55 · Answer 12 · 2023-09-14T08:31:11.033000

PySpark: Get Number of Columns from DataSchema

55 views Asked by LordRofticus At 14 September 2023 at 08:31

score 81 · Answer 13 · 2023-09-10T10:02:18.600000

How to get a timestamp data type column without the seconds in Pyspark?

81 views Asked by bigdataadd At 10 September 2023 at 10:02

score 124 · Answer 14 · 2023-09-09T20:30:32.497000

Pyspark calculate new rows based on previous rows from current and other multiple columns

124 views Asked by prince13i At 09 September 2023 at 20:30

score 32 · Answer 15 · 2023-09-04T08:26:08.200000

pyspark - making a new column lookup_l that contains a list and its elements are values from other columns from same df from current row

32 views Asked by mahak tirole At 04 September 2023 at 08:26

score 270 · Answer 16 · 2023-09-03T17:39:13.570000

How do I grab all data from multiple tables found in a certain table in PySpark?

270 views Asked by user3461502 At 03 September 2023 at 17:39

score 161 · Answer 17 · 2023-08-27T17:47:11.203000

Azure Synapse PySpark - Load Schema from a Schema Definition File

161 views Asked by LordRofticus At 27 August 2023 at 17:47

score 46 · Answer 18 · 2023-08-23T17:43:35.820000

Mulitple Row in spark dataframe using schema

46 views Asked by Blue Clouds At 23 August 2023 at 17:43

score 186 · Answer 19 · 2023-08-04T12:53:28.177000

Selecting a column with backtick in its name - AnalysisException: cannot resolve Column

186 views Asked by Jim Macaulay At 04 August 2023 at 12:53

score 270 · Answer 20 · 2023-07-31T06:07:30.890000

How to replace null value with some value using coalesce in pyspark

270 views Asked by Vivek Mishra At 31 July 2023 at 06:07

TechQA.

List Question

Reading a parquet file using parquet-tool converting string values to scientific notation

Is there a painless way to migrate existing Parquet files to a new schema? I wish to update an AWS Glue table column data type

pyspark structural streaming one column dynamic json save to elasticsearch as json object

pyspark error while writing data to existing hive table

ValueError: Unable to parse datatype from schema. Could not parse datatype: interval year

pyspark - how to add a new element to ArrayType column

Need to add headers in existing data frame

o1566.showString error persists even with os env variables defined with PySpark in jupyter notebook

can we merge two parquet files into one which has different datatypes to the columns using the pyspark

check if rows are already present in pyspark dataset

PySpark DataFrame creation is throwing PySparkTypeError

PySpark: Get Number of Columns from DataSchema

How to get a timestamp data type column without the seconds in Pyspark?

Pyspark calculate new rows based on previous rows from current and other multiple columns

pyspark - making a new column lookup_l that contains a list and its elements are values from other columns from same df from current row

How do I grab all data from multiple tables found in a certain table in PySpark?

Azure Synapse PySpark - Load Schema from a Schema Definition File

Mulitple Row in spark dataframe using schema

Selecting a column with backtick in its name - AnalysisException: cannot resolve Column

How to replace null value with some value using coalesce in pyspark

Popular Questions

Trending Questions