I am not experienced with stream processing and wonder, if the problem I am facing can and should be solved with a stream processing framework, or if I am heading the wrong direction here.
I have some finished processing applications handling large volumes of data, all accepting some inputs and generating some outputs in real time. Many of them are parts of a computational chain, where more than one of these processing applications are run. These chains change over time as the inputs change over time.
Some of these chains or applications are more resource hungry than others (in terms of CPU and memory load) and should therefore be reasonably distributed across a computing cluster depending on the currently available resources and the chain to be processed.
The applications themselves must not be altered except for the interfaces.
Is this a feasably task to accomplish with a stream processing framework? I am not sure, because my take on these frameworks is that you split single steps (such as aggregation) of processing across multiple machines, not entire long-running applications.
If they are not the right tool for me, which kind of software am I looking for instead?
Thanks