I am building a embedded device/host that provides a particular service/API in a multi tenant mode. For security reasons strong Isolation is expected. and thus going with 1 container per tenant model.
So in a nut shell, building a device-as-a-service which will run multiple containers using podman/docker. with each container belonging to a particular customer. the device owner controls the platform to start/stop containers from requesting customers, and once created hands over it to the customer and platform has no visibility to what is happening except for general resource utilization metrics.
Now with that in mind what are all the factors that i should consider for building such a multi tenant device. If i have gone with kubernetes i would have used namespace as a isolation model with CNI like calico that can provide network separation, used falco and kyverno for intrusion detection and policy enforement. k8 built in QoS measurement to avoid noisy nebiour situations , kubeplus for multi customer tenancy etc. encrypted storages per namespace using CSI, PV claims model
But going with docker to avoid k8 specfic complications. So my question is what are the things i might be missing when using just docker? There is no one single right answer, so just expecting to hear your experiences/knowledge here from the community.
Docker, does not have built-in support for multi-tenancy. When using Docker for a multi-tenant device, there such several factors to consider to achieve strong isolation and security as follows:
Security consideration on enforcing the policies within the containers.
Network, should be configured to ensure network isolation for each container, this may require additional config outside of Docker's built-in networking capabilities.
Storage, implementing encrypted storage per tenant using Docker may require custom solutions or 3rd party app. Docker lacks the built-in support for storage isolation which is provided by Kubernetes' PersistentVolume (PV) claims model.
Management and Visibility, docker does not provide native support for multi-tenancy management features like tenant namespaces and dedicated tenant controllers.
Lastly, you can look also on this link for additional discussion regarding the docker multi-tenant.