How to deal with issues when storing uploaded files in the file system for a web app?

Question

I am building a web application where the users can create reports and then upload some images for the created reports. Those images will be rendered in the browser when the user clicks a button on the report page. The images are confidential and only authorized users will be able to access them.

I am aware of the pros and cons of storing images in database, in filesystem or a service like amazon S3. For my application, I am inclined to keep the images in the filesystem and paths of the images in the database. That means I have to deal with the problems arising around distributed transaction management. I need some advice on how to deal with these problems.

1- I believe one of the proper solutions is to use technologies like JTA and XADisk. I am not very knowledgeable about these technologies but I believe 2 phase commit is how automicity is achieved. I am using MySQL as the database, and it seems like 2 phase commit is supported by MySQL. Problem with this approach is XADisk does not seem to be an active project and there is not much documentation about it and there is the fact that I am not very knowlegable about the ins and outs of this approach. I am not sure if I should invest in this approach.

2- I believe I can get away with some of the problems arising from the violation of ACID properties for my application. While uploading images, I can first write the files to disk, if this operation succeeds I can update the paths in the database. If database transaction fails, I can delete the files from the disk. I know that is still not bulletproof; an electricity shortage might occur just after the db transaction or the disk might not be responsive for a while etc...I know there are also concurrency issues, for instance if one user tries to modify the uploaded image and another tries to delete it at the same time, there will be some problems. Still the chances for concurrent updates in my application will be relatively low.

I believe I can live with orphan files on the disk or orphan image paths on the db if such exceptional cases occur. If a file path exists in db and not in the file system, I can show a notification to the user on report page and he might try to reupload the image. Orphan files in the file system would not be too much problem, I might run a process to detect such files time to time. Still, I am not very comfortable with this approach.

3- The last option might be to not store file paths in the db at all. I can structure the filesystem such that I can infer the file path in code and load all images at once. For instance, I can create a folder with the name of report id for each report. When a request has been made to load images of the report, I can load the images at once since I know the report id. That might end up with huge number of folders in the filesystem and I am not sure if such a design is acceptable. Concurrency issues will still exist in this scheme.

I would appreciate some advice on which approach I should follow.

Answer 1

I believe you are trying to be ultra-correct, and maybe not that much is needed, but I also faced some similar situation some time ago and explored also different possibilities. I disliked options aligned to your option 1, but about the 2 and 3, I had different successful approaches.

Let's sum up first the list of concerns:

• You want the file to be saved
• You want the file path to be linked to the corresponding entity (i.e the report)
• You don't want a file path to be linked to a file that doesn't exist
• You don't want files in the filesystem not linked to any report

And the different approaches:

1. Using DB

You can assure transactions in the DB pretty much with any relational database, and with S3 you can ensure read-after-write consistency for both new objects and upload of new objects. If you PUT an object and you get a 200 OK, it will be readable. Now, how to put all this together? You need to keep track of the process. I can figure 2 ways:

1.1 With a progress table

1. The upload request is saved to a table with anything need to identify this file, report id, temp uploaded file path, destination path, and a status column
2. You save the file
3. If the file safe fails you can update the record in the table, or delete it
4. If saving the file is successful, in a transaction:
   • update the progress table with successful status
   • update the table where you actually save the relationship report-image
5. Have a cron, but not checking the filesystem, but checking the process table. If there is any file in the filesystem that is orphan, definitely it had been added to the table (it was point 1). Here you can decide if you will delete the file, or if you have enough info, you can continue with the aborted process triggering the point 4.

The same report-image relationship table with some extra status columns.

1.2 With a queue system

Like RabbitMQ, SQS, AMQ, etc

A very similar approach could be done with any queue system instead of a db table. I wont give much details because it depends more on your real infrastructure, but just the general idea.

• The upload request goes to a queue, you send a message with anything you may need to identify this file, report id, and if you want a tentative final path.
• You upload the file
• A worker reads pending messages in the queue and does the work. The message is marked as consumed only when everything goes well.
• If something fails, naturally the message will come back to the queue
• In the next time a message is read, the worker can have enough info to see if there is work to resume, or even a file to delete if resuming is not possible

In both cases, concurrency problems wont be straightforward to manage, but can be managed (relying on DB locks in fist case, and FIFO queues in second cases) but always with some application logic

2. Without DB

To some extent a system without a database would be perfectly acceptable, if we can defend it as a proper convention over configuration design. You have to deal with 3 things:

1. Save files
2. Read files
3. Make sure that the structure of the filesystem is manageable

Lets start with 3:

Folder structure

• In general, something like one folder for report id will be too simple, and maybe hard to maintain, and also ultimately too plain. This will cause issues, because if we have a folder images with one folder per report, and tomorrow you have less say 200k reports, the images folder will have 200k elements, and even an ls will take too much time, same for any programing language trying to access. That will kill you

• You can think about something more sophisticated. Personally like a way that I learnt from Magento 1 more than 10 years ago and I used a lot since then: Using a folder structure following first outside rules, but extended with rules derived extended with the file name itself.

  • We want to save a product image. The image name is: myproduct.jpg
  • first rule is: for product images i use /media/catalog/product
  • then, to avoid many images in the same one, i create one folder per every letter of the image name, up to some number of letters. Lets say 3. So my final folder will be something like /media/catalog/product/m/y/p/myproduct.jpg
  • like this, it is clear where to save any new image. You can do something similar using your reports id, categories, or anything that makes sense for you. The final objective is to avoid too flat structure, and to create a tree that makes sense to you, and also that can be automatized easily.

And that takes us to the next part:

Read and write.

I implemented a similar system before quite successfully. It allowed me to save files easy, and to retrieve them easily, with locations that were purely dynamic. The parts here were:

• S3 (but you can do with any filesystem)
• A small microservice acting as a proxy for both read and write.
• Some namespace system and attached logic.

The logic is quite simple. The namespace lets me know where the file will be saved. For example, the namespace can be companyname/reports/images.

Lets say a develop a microservice for read and write:

For saving a file, it receives:

• namespace
• entity id (ie you report)
• file to upload

And it will do:

• based on the rules I have for that namespace, and the id and file name will save the file in this folder
• it doesn't return the physical location. That remains unknown to the client.

Then, for reading, clients will use a URL that uses also convention. For example you can have something like

https://myservice.com/{NAMESPACE}/{entity_id}

And based on the logic, the microservice will know where to find that in the storage and return the image.

If you have more than one image per report, you can do different things, such as: - you may want to have a third slug in the path such as https://myservice.com/{NAMESPACE}/{entity_id}/1 https://myservice.com/{NAMESPACE}/{entity_id}/2 etc... - if it is for your internal application usage, you can have one endpoint that returns the list of all eligible images, lets say https://myservice.com/{NAMESPACE}/{entity_id} returns an array with all image urls

How I implemented this was with quite simple yml config to define the logic, and very simple code reading that config. That allowed me to have a lot of flexibility. For example save reports in total different paths or servers or s3 buckets if they belong to different companies or are different report types