Is it safe to read data with boto3 from S3 if that data had been written using Stocator in pyspark?

178 views Asked by At

I have an application that uses Stocator as a connector for Spark. This application writes the data to the S3 cos bucket.

Now I am working on a service that's supposed to read that data from S3. According to this thread here, you cannot specify the uri/protocol that boto3 uses. Is it safe to read that data using the default protocol of S3 REST API?

The reason I am asking is that I have been told that reading data using S3A (another protocol) that has been written using Stocator could result in reading duplicates.

0

There are 0 answers