I'm a DataEngg intern working on a POC project, building an open-source DataLakeHouse with multi-node multi drive MinIO as its storage bucket. Im using Spark as my compute engine. I have 4 nodes, among which one is the host. Idea is to bring in Data to MinIO with certain transformations to the data. My DataLakeHouse model is ready and in testing phase. I tried to upload files of 70MB-100MB and the host-server is not letting me establish connection.
I'm trying to figure out the issue, I notice whenever I upload the file the server went down while the rest nodes are working fine. Right now, Im not able to sshh to my host. Any insights on this?
I tried to change the endpoint URL to check the MinIO console, but it will not let me in as my host is not active.