save each line of string as a file in spark

Question

I am working on a use case wherein I need to save each line of text in RDD as a separate file in Google Cloud Storage.

The run platform is Databricks with spark version 3.2.x and language is Scala.

Can you please point me to relevant document that can help me do that?

We have methods to save text but not something that works on such a granularity (each line).

Answer 1

You could control the number of records per file using maxRecordsPerFile property

val df = ...
df.write
    .option("maxRecordsPerFile", 1)
    ...