Migrating from spring-batch-admin to spring cloud dataflow

Question

We are planning to migrate from spring-batch-admin to spring-cloud-dataflow as batch admin will be moving to into the Spring Attic with an end of life date to be December 31, 2017.

As per the documentation in Migration Doc, “In the Spring Batch use case, each batch job is packaged as an independent Spring Boot über jar that is registered with Spring Cloud Data Flow”

In which case, if there are 50 batch jobs, 50 über jars have to created? Doing so would be a maintenance nightmare, creating many jars and maintaining them would be difficult.

Any workaround for the same? so we would be able to pack all batch jobs in single über jar and we can randomly launch any job when required?

Any help would be highly appreciated, thanks in advance

Answer 1

TL;DR;
Watch my talk on migrating data processing to microservices found here: https://www.youtube.com/watch?v=COzkPkHZMG8. If after that you still feel that it's a bad thing, skip to the bottom where I propose an alternate (not recommended) approach.

Why this is a good thing
Let me take a minute to walk through why we thing this is a better solution going forward and why I'd discourage you from taking a monolithic approach. After all of that, I'll propose an alternative that I do not recommend, but should work.

Breaking up the monolith
If you think about the process most enterprises have for delivering batch functionality from a developer's laptop to production, it is typically a slow process with infrequent releases. The process is slow, there may be multiple groups the code needs to get through (development, some from of external QA, possibly some form of change control process, then finally some type of operations team to actually deploy the code). Typically, the smaller the piece of code that needs to get through that process the easier it is to get through that process.

In this example, for a system that contains 50 batch jobs, in order to change one, you need to go through that process with all the jobs. Breaking this up actually simplifies maintenance in that you can change and deploy jobs independently. A developer only needs to focus on the batch job at hand instead.

Migrating to über jars
Another advantage of moving from a single WAR file with all your jobs packaged in it is flexibility. You can run these jobs however you want on whatever infrastructure you want. Want to run it locally or on bare metal via the java -jar foo.jar command? Go for it. Want to run it on CloudFoundry via a cf push? You bet. Want to dockerize the app and run it on Kubernetes? You can! While you can do the same thing without going to an über jar approach, it's more nuanced since the infrastructure may change from environment to environment. In the case of an über jar, all you need to guarantee is the java version.

Also the artifact management piece is a well solved problem. Pushing über jars to a Maven repository is a simple process that is well vetted across the java landscape. How to manage WAR files really isn't. You can push them to a Maven repository, but it's not ideal. By moving to über jars, your release process becomes very standardized across all of the jobs (and all of your apps in general).

Finally, moving to an über jar shouldn't be that hard. It should really be just a packaging exercise assuming your jobs are well defined. If they aren't, this is a good opportunity to do some healthy restructuring so that they are more modular in the first place (good engineering practice).

The alternative approach
I want to start off here by saying that I DO NOT recommend this approach. However, it should work.

Instead of creating one über jar for each job, create an über jar with all 50 jobs in it. You'll need to create your own CommandLineRunner that looks at an environment variable to determine which job to run when launched and turn off the Spring Boot functionality to automatically execute your jobs on startup.

From there, you'd configure your 50 jobs via 50 task definitions within Spring Cloud Data Flow. Each one passing the environment variable indicating the job to run. From there, you'd be able to independently execute/monitor/etc each of the 50 jobs and still get your monolithic artifact.

Answer 2

Adding to the "alternative approach", you can give an option when executing single monolithic 50-jobs jar file to execute exact job you want.

java -jar single.jar -Dspring.batch.job.names=jobName