R processx package error Cannot write connection (system error 32, Broken pipe)

239 views Asked by At

I am getting the error below after a long running process on an Ubuntu Docker container. I am using the rocker/tidyverse:3.6.3 base image. My forecast takes about 2 hours to run with multidplyr and builds a fable, or forecast table, with the fable package.

At the end of the script there is a write to Hive where the function below rights the dataframe to Hive. This is where the error is happening as I follow the messages that I have built in script. For shorter process runs the code runs just fine and the table is built in Hive. Unfortunately, I can't provide a reprex because it is internal to my work.

<c_error in rethrow_call(c_processx_connection_write_bytes, con, str):
 Cannot write connection (system error 32, Broken pipe) @processx-connection.c:627 (processx_c_connection_write_bytes)>
 in process

The dependencies for the function are ssh, dplyr, readr, askpass, and magrittr, but the error is for a package that I am unfamiliar with, processx. I believe it is an RStudio supported package because it lives at r-lib.org.

The function being used can be found here; it is too long to paste: https://github.com/Fredo-XVII/RToolShed/blob/master/R/write_df_to_hive3.R

Any help would be greatly appreciated. Thank you!

P.S. I was not able to add #processx as a tag, so if anyone can add it, I would be grateful.

0

There are 0 answers