I have Qubole connected to Azure data lake, and I can start a spark cluster, and run PySpark on it. However, I can't save any native Python output, like text files or CSVs. I can't save anything other than Spark SQL DataFrames. What should I do to resolve this? Thank you in advance!
How to get Python in Qubole to save CSV and TXT files to Azure data lake?
315 views Asked by HT. At
2
There are 2 answers
0
HT.
On
I resolved it. I needed to add the file to the PySpark session using textFile() details and sample code here
For any file I want, I need to add it to the spark session. For example, if I needed to add a .py file from Azure data lake, I need to add it using addPyFile() with the path to the file.
Related Questions in PYTHON
- Cabal sandbox is using a global dependency. Could not resolve
- Haskell lens: let binding of Traversal'
- How can I parse fixed-length, non-delimited integers with attoparsec?
- Pipeline-like operation using TChan
- compile-time vs. run-time cost of Hamlet templates
- Date-time package in haskell - error in the current one, can't find an analog
- How does one debug infinite recursion in Haskell?
- Force GHC using local files
- List with random numbers in Haskell
- Changes in other elements based on listbox selections in threepenny-gui
Related Questions in AZURE
- Cabal sandbox is using a global dependency. Could not resolve
- Haskell lens: let binding of Traversal'
- How can I parse fixed-length, non-delimited integers with attoparsec?
- Pipeline-like operation using TChan
- compile-time vs. run-time cost of Hamlet templates
- Date-time package in haskell - error in the current one, can't find an analog
- How does one debug infinite recursion in Haskell?
- Force GHC using local files
- List with random numbers in Haskell
- Changes in other elements based on listbox selections in threepenny-gui
Related Questions in QUBOLE
- Cabal sandbox is using a global dependency. Could not resolve
- Haskell lens: let binding of Traversal'
- How can I parse fixed-length, non-delimited integers with attoparsec?
- Pipeline-like operation using TChan
- compile-time vs. run-time cost of Hamlet templates
- Date-time package in haskell - error in the current one, can't find an analog
- How does one debug infinite recursion in Haskell?
- Force GHC using local files
- List with random numbers in Haskell
- Changes in other elements based on listbox selections in threepenny-gui
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Popular Tags
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
If I understand your question correctly,I believe you are unable to download the result of pyspark command output into text or CSVs while you are able to do so for the spark sql command command output in a nice tabular format.
Unfortunately, there is no direct field separator for the output text for a Python or Shell command outputs. You will need to get your output comma separated so you can download the raw output and save it as a csv.
If this is not what you meant, Please share more details as to what exactly are you trying to do along with screenshots details. As that will help us answer your question better.