My application needs to run a lot of containers as worker nodes (to do various batch processing jobs) and I'm not really interested in keeping up web servers or databases - just short jobs that can take anywhere between 1 second to 1 hour. My idea is to work against a cloud of nodes without me having to worry about what machine from these nodes has the available resources to process my job (mesos is pretty good at this - as advertised).
I'm playing right now with DC/OS and I was wondering if any of the other clustering technologies offer this feature: given I need 1CPU, 2GB RAM and 2GB of disk - run X docker container against my nodes.
I like the idea of swarm due to the fact that I'm very familiar with docker itself and I believe it's the easiest to setup and automate (scale up or down). I like kubernetes (no experience however) because it's free and I'm pretty sure it will stay that way for a long time. I like DC/OS because it bundles a lot but I'm not sure of their future plans and I'm used to projects cutting off features to include them in a plan that charges your soul for x number of nodes.
What are your thoughts?
Kubernetes, Swarm, and Mesos can all technically schedule jobs for you and handle constraining resources for you.
Unlike the other two, Mesos was designed primarily to handle distribution, task, and resource management at a lower level. Focusing on these bits led to greater power and flexibility, but also more complexity at a lower level. That's why DC/OS exists, to give you a bundle of microservice tools that work well as a higher level platform.
Mesos was also designed to allow you to bring your own scheduler to handle task lifecycle needs, which tend to be needed for stateful tasks. Kubernetes and Swarm were designed primarily to handle the stateless services use case and then adapted later to handle stateful services and jobs, with the included scheduler.
DC/OS is built on Mesos and comes with built-in schedulers for jobs and services, while still allowing you to build your own custom scheduler if needed.
Kubernetes recently added support for custom schedulers as well, but its significantly less mature than the Mesos implementation and ecosystem and also still revolves around using the core pods & replica set primitives, which may be empowering or limiting, depending on your needs.
Mesosphere recently built a new dcos-commons framework to make it trivial to build JVM-based Mesos schedulers, as well. So that may boost your productivity on DC/OS. https://github.com/mesosphere/dcos-commons
Mesos & DC/OS also gives you more options on containerization. You can use Docker images and Docker containers, if you like. Or you can use the Mesos container runtime with or without Docker images, which gives you more flexibility in terms of workloads and packaging.
DC/OS and Kubernetes both have package managers, as well, which can be useful for installing dependencies like Spark, Kafka, or Cassandra. But DC/OS tends to have more robust data services, because they're built with their own custom schedulers, whereas the Kubernetes ecosystem tends to make complex lifecycle managing Docker container wrappers around their systems due to the late arrival of custom schedulers. Docker also sort of includes package management if you consider docker images "packages". The difference is that DC/OS and Kubernetes package higher level abstractions (apps & pods) which may include multiple containers. More recently, Docker has added "stacks" which are higher level abstractions, but I don't think there's any external repository mechanism or much package management around them.
Swarm is definitely the simplest, but its original API was designed to be the same as the node API, which was great for familiarity and onboarding, but rather limiting as a higher level abstraction. They've since effectively rewritten the swarm API and bundled it into docker-engine as "swarm-mode". This bundling of the orchestration engine and container runtime makes it easier for the user to install and manage but also combines what was previously two different abstraction levels. So instead of being just a dependency of orchestration engines, Docker engine now competes with them as well, going against the unix philosophy of doing one thing well and making for a bit of a political mess in the respective open source communities. Twitter, hacker news, and chat conversations escalated into talk of forking docker which lead to K8s experimenting on alternatives, DC/OS supporting Docker images without using Docker engine, and Docker extracting containerd.
They all work fine. Selecting one kind of depends on your needs. I generally recommend DC/OS because it tackles a larger set of problems and is made up of many distinct microservice tools and layers, allowing you support multiple use cases by programming against the layer than makes the most sense. Disclosure tho, I do work for Mesosphere! ;)