I am in a bit of a jam.
I am working on upgrading our software to have Kettle 6.1. Specifically, we need the feature of S3FileOutput. Meanwhile, our application was already using the aws-sdk for other things.
So I am running into a problem: Pentaho Kettle requires version 1.0.something of aws-sdk. Our application, on the otherhand needs 1.9.6 of the aws-sdk.
To give more details, the feature of Kettle we require is in the pentaho-big-data-legacy plugin. Even if I upgrade to the latest version of Kettle, pentaho-big-data-legacy still uses the old version of the aws-sdk.
I've read a bit about plugins having special classloaders, so one option I was considering is that maybe I am not downloading the right dependency. However, when I tried downloading pentaho-big-data-plugin instead of pentaho-big-data-legacy, I got weird errors, so I stopped going down this path.
I was wondering if there is any way I could put the Kettle Libs in one folder, and my application libs in another folder, and then set some sort of a PENTAHO environment variable to pick up the libraries from the alternative folder.
Another option is if I could somehow set the pentaho classloader, but I don't know if this is possible.
What are my options for having 2 versions of the aws-sdk in my application, with regards to Kettle?
Maven can do much more than download dependencies.
The Maven Shade plugin can help with your current predicament. During a build, it can rename packages.
You would make a project that builds a "fat jar" (or "uber jar") with Pentaho Kettle and its version of the aws-sdk re-packaged as appropriate. That dependency would be handled before your project is built, so you are free to use whatever version of aws-sdk you like since there is no longer a conflict on package names.