We're encountering some errors when starting Nifi 1.16.3 after calling encrypt-config.sh in a secure environment. For background, Nifi is deployed and managed using an in-house deployment product, and is used as a component within our application stack. In general everything works fine when using Nifi 1.15.1, however, we are encountering issues when using Nifi 1.16.3. In short, when deploying an instance of Nifi in our system, we first create all settings necessary, including a nifi.properties and bootstrap.conf files. At this first stage, we set an initial password of nififtw! to sensitive.props.key, which at this moment in time is plain text. We execute encrypt-config command:
JAVA_HOME=/tech/java/openjdk1.8.0_322
/bin/encrypt-config.sh --key AAAF0313BCC242CC9EC1DD30AD74FB2E --niFiProperties /servers/nifi/default/conf/nifi.properties --loginIdentityProviders /servers/nifi/default/conf/login-identity-providers.xml --bootstrapConf /servers/nifi/default/conf/bootstrap.conf
At this stage, we don't parse in the flow.xml as it doesn't exist yet, however, the protected values in nifi.properties, i.e sensitive.props.key, are encrypted based on the --key. Nifi starts as expected. In a later moment, when updating our system, we invoke encrypt-config.sh one more time, but now we attempt to change the propsKey to the same value of they hexkey used. Note that at this stage, we have a flow.xml and we parse in the path as an argument:
JAVA_HOME=/tech/java/openjdk1.8.0_322
/bin/encrypt-config.sh --key AAAF0313BCC242CC9EC1DD30AD74FB2E --niFiProperties /servers/nifi/default/conf/nifi.properties --loginIdentityProviders /servers/nifi/default/conf/login-identity-providers.xml --flowXml /databases/nifi/default/flow.xml.gz --propsKey AAAF0313BCC242CC9EC1DD30AD74FB2E --bootstrapConf ey/servers/nifi/default/conf/bootstrap.conf
No errors are encountered at this stage and the value of sensitive.props.key in nifi.properties is changed accordingly. However, when attempting to start nifi server, we encounter the error bellow: nifi-bootstrap.log
2022-11-13 06:32:06,488 INFO [main] org.apache.nifi.bootstrap.Command Launched Apache NiFi with Process ID 82873
2022-11-13 06:32:14,269 INFO [NiFi Bootstrap Command Listener] org.apache.nifi.bootstrap.RunNiFi Apache NiFi now running and listening for Bootstrap requests on port 32908
2022-11-13 06:34:06,217 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
2022-11-13 06:34:06,218 ERROR [NiFi logging handler] org.apache.nifi.StdErr Shutting down...
2022-11-13 06:34:06,945 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will not restart NiFi
nifi-app.log
2022-11-13 06:34:06,209 INFO [main] org.eclipse.jetty.server.Server Started @119858ms
2022-11-13 06:34:06,209 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL]
at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:78)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.decrypt(StandardFlowComparator.java:281)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareProperties$3(StandardFlowComparator.java:291)
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compareProperties(StandardFlowComparator.java:289)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:267)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareComponents$1(StandardFlowComparator.java:114)
at java.util.HashMap.forEach(HashMap.java:1290)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:467)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compare$5(StandardFlowComparator.java:472)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.lambda$compareComponents$1(StandardFlowComparator.java:114)
at java.util.HashMap.forEach(HashMap.java:1290)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compareComponents(StandardFlowComparator.java:112)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:472)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:94)
at org.apache.nifi.registry.flow.diff.StandardFlowComparator.compare(StandardFlowComparator.java:79)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.compareFlows(VersionedFlowSynchronizer.java:387)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:167)
at org.apache.nifi.controller.serialization.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:43)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1524)
at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:107)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:819)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:542)
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:67)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1008)
at org.apache.nifi.NiFi.<init>(NiFi.java:170)
at org.apache.nifi.NiFi.<init>(NiFi.java:82)
at org.apache.nifi.NiFi.main(NiFi.java:330)
Caused by: javax.crypto.BadPaddingException: pad block corrupted
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$BufferedGenericBlockCipher.doFinal(Unknown Source)
at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2168)
at org.apache.nifi.encrypt.CipherPropertyEncryptor.decrypt(CipherPropertyEncryptor.java:74)
... 62 common frames omitted
As shown in the errors above, the decryption fails when attempting to start the nifi server.
We suspect that when we try to encrypt the flow.xml with a new sensitive.props.key, the re-encryption doesn't happen, even though the value of propsKey in nifi.properties has been replaced and encrypted accordingly to the new password we parse.
After we get the failure described, if we manually change the propsKey in nifi.properties to what it was before (the encrypted version of 'nififtw!'), we manage to start nifi successfully.
Note that the password nififtw! is just one example. The issue happens with any password we might use for the first call of encrypt-config, even if we call encrypt-config again specifying a new propsKey.
We've attempted to modify the arguments used when executing the encrypt-config.sh, i.e. by creating the modified flow.xml and nifi.properties in a /tmp/ directory and manually replacing the older versions, which hasn't worked either.
This issue only arrises in Nifi version 1.16.3.0. We were using version 1.15.1.0 and version 1.11.3.0 and the same logic works perfectly fine. This is how my nifi.properties looks like:
# security properties #
nifi.sensitive.props.key=<encrypted-key>
nifi.sensitive.props.key.protected=aes/gcm/128
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.additional.keys=
Any idea of what might be causing this issue?
The error that you encountered in nifi 1.16 is appearing because of some changes introduced in nifi starting from 1.16 version.
Starting from nifi 1.16 , NIFI start writing on flow.xml.gz as well as JSON format ( prior to 1.16 this JSON was not there ) flow.json.gz , see the link https://www.mail-archive.com/[email protected]/msg15332.html
When 1.16 nifi start it creates both the file, writes onto it and then encrypt it using props key from nifi.properties , when your code run encrypt-config.sh tool it only changes the flow.xml.gz with new props key and JSON flow file still encrypted with old props key, While next time nifi going to start it try to read props key from nifi.properties , which is the new ( changed props key ) and it successfully able to decrypt the flow.xml.gz but it can not able to decrypt flow.json.gz because JSON file is still encrypted with old props key which is overwritten by new props key on nifi.properties
Solution : from 1.16 onwards "./bin/nifi.sh set-sensitive-properties-key NewSensitivePropertiesKey" tool improved to cover both xml as well as json version of flow file while changing the props key, see JIRA ticket https://issues.apache.org/jira/browse/NIFI-9711 , So while changing the props key you can try using nifi.sh in place of encrypt-config.sh
Regards
Vikas