How to handle private files in a microservice

Question

We're working on a backend project and we've started a move to microservices development. We already have a few services in place, one of which is a FileService which stores and fetches files (using underlying Amazon S3 storage). The FileService also provides file checksum, authentication and retry mechanism and is used to share files across services and with the clients.

We are now building a new service and part of this service's private data are files that the service stores and uses for its business logic, and we have a dilemma of whether we should use the FileService to store and fetch the files or handle the storage and fetching of the files internally in the service.

The reason to use the FileService is we're getting all the features implemented in the service for free (retry, checksum etc). The reason not to use it is we want the new service to be able to work autonomously and using the FileService ties the new service to it (it must handle OAuth2 authentication to fetch/upload files, it must deploy the FileService and the AuthService whenever this services is deployed etc).

I wanted to know if someone has best practices for storing private files in a microservices environment, and what is the best approach to it with the pros and cons.

Answer 1

Converting in-process FileService component to microservice will definitely have advantages as well as disadvantages. You've listed several of them, but most importantly you have to create a cost/benefit analysis matrix applicable to your business and domain specifically. There is no "best practices" approach here.

Costs:

• is it okay for you to increase response times? Because now you will have to transfer files twice: s3 -> fs microservice -> client microservice
• how more likely situation of losing a connection between nodes becomes?
• how big your files are? The unreliable connection between microservices may become a problem?
• how frequently do you need to access those files? Maybe you will lose the ability to have local cache to speed up the process?
• are you okay with implementing and supporting separate auth microservice or you can just whitelist this service in your firewall

Benefits:

• you don't have to redeploy all dependent components every time the logic of storing files or doing retries changes.
• you can move to another cloud provider more easily in the future if necessary, again, without redeploying everyting.
• it is reusable in a heterogeneous environment, where other components may be implemented using different technological stacks

Conclusion:

There is no way to answer those questions without actually talking with business people and discussing risks around such transition.