I have been engaging with the Spring Cloud ecosystem for the past 3 months. We have created different streams with both prebuilt and custom Spring Cloud Stream apps. During this period, we have come across different errors and found solutions from documentation and forums.
However, one thing I'm still unsure about is the root cause of the below error, that I'm getting during the deployment of the streams.
An error occurred
Could not install AppDeployRequest [[AppDeploymentRequest@4319efb5 commandlineArguments = list[[empty]], deploymentProperties = map['spring.cloud.deployer.appName' -> 's3-sink', 'spring.cloud.deployer.bootVersion' -> '2', 'spring.cloud.deployer.count' -> '1', 'spring.cloud.deployer.group' -> 's3-pipeline-2'], definition = [AppDefinition@2e9b8419 name = 's3-sink-v8', properties = map['management.endpoints.web.exposure.include' -> 'health,info,bindings', 'spring.cloud.dataflow.stream.name' -> 's3-pipeline-2', 'management.metrics.tags.instance.index' -> '${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}', 'spring.cloud.stream.kafka.streams.binder.brokers' -> 'PLAINTEXT://kafka-broker:9092',
Below is the detailed error that I got from docker log -f --tail 125 the dataflow-server.
2023-10-31 15:40:21.579 INFO 1 --- [nio-7577-exec-1] o.s.s.s.DefaultStateMachineService : Acquiring machine with id alert-test-1
2023-10-31 15:40:21.581 INFO 1 --- [eTaskExecutor-4] o.s.c.s.s.s.StateMachineConfiguration : Entering state StateMachineState [getIds()=[INSTALL, INSTALL_INSTALL], toString()=AbstractState [id=INSTALL, pseudoState=null, deferred=[], entryActions=[], exitActions=[], stateActions=[], regions=[], submachine=INSTALL_INSTALL INSTALL_EXIT / / uuid=0d372249-1980-4bc3-9a1b-450e6bec435d / id=alert-test-1], getClass()=class org.springframework.statemachine.state.StateMachineState]
2023-10-31 15:40:21.582 INFO 1 --- [eTaskExecutor-4] o.s.c.s.s.s.StateMachineConfiguration : Entering state ObjectState [getIds()=[INSTALL_INSTALL], getClass()=class org.springframework.statemachine.state.ObjectState, hashCode()=1195742278, toString()=AbstractState [id=INSTALL_INSTALL, pseudoState=org.springframework.statemachine.state.DefaultPseudoState@3f2ab544, deferred=[], entryActions=[org.springframework.cloud.skipper.server.statemachine.InstallInstallAction@5d965dbd], exitActions=[], stateActions=[], regions=[], submachine=null]]
2023-10-31 15:40:21.601 ERROR 1 --- [eTaskExecutor-4] o.s.c.s.s.statemachine.AbstractAction : Action execution failed class=[class org.springframework.cloud.skipper.server.statemachine.InstallInstallAction]
org.springframework.cloud.skipper.SkipperException: Could not install AppDeployRequest [[AppDeploymentRequest@ddd395a commandlineArguments = list[[empty]], deploymentProperties = map['spring.cloud.deployer.appName' -> 's3-sink', 'spring.cloud.deployer.bootVersion' -> '2', 'spring.cloud.deployer.group' -> 'alert-test-1'], definition = [AppDefinition@49bd0084 name = 's3-sink-v1', properties = map['management.endpoints.web.exposure.include' -> 'health,info,bindings,prometheus', 'spring.cloud.dataflow.stream.name' -> 'alert-test-1', 'management.metrics.tags.instance.index' -> '${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}', 'spring.cloud.stream.kafka.streams.binder.brokers' -> 'PLAINTEXT://kafka-broker:9092', 'wavefront.application.service' -> '${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}-${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}', 'spring.cloud.stream.bindings.input.group' -> 'alert-test-1', 'management.metrics.tags.application.guid' -> '${spring.cloud.application.guid:unknown}', 'cloud.aws.region.static' -> 'ap-southeast-2', 'server.port' -> '20013', 'management.metrics.tags.application.name' -> '${vcap.application.application_name:${spring.cloud.dataflow.stream.app.label:unknown}}', 'wavefront.application.name' -> '${spring.cloud.dataflow.stream.name:unknown}', 'spring.kafka.streams.properties.metrics.recording.level' -> 'DEBUG', 'management.metrics.tags.application.type' -> '${spring.cloud.dataflow.stream.app.type:unknown}', 'spring.cloud.dataflow.stream.app.label' -> 's3-sink', 'management.metrics.tags.stream.name' -> '${spring.cloud.dataflow.stream.name:unknown}', 'management.metrics.tags.application' -> '${spring.cloud.dataflow.stream.name:unknown}-${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}', 's3.consumer.bucket' -> 'aija-dst-2', 'spring.cloud.stream.kafka.streams.binder.zknodes' -> 'zookeeper:2181', 'cloud.aws.credentials.accessKey' -> 'AKIAV7ITGOFTEZPUQIUW', 'cloud.aws.credentials.secretKey' -> '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', 's3.consumer.key-expression' -> 'headers['targetCompleteFileName']', 'spring.cloud.dataflow.stream.app.type' -> 'sink', 'spring.cloud.stream.kafka.binder.zknodes' -> 'zookeeper:2181', 'spring.cloud.stream.bindings.input.destination' -> 'alert-test-1.s3-processor', 'cloud.aws.stack.auto' -> 'false', 'management.endpoints.web.exposure.auth.enabled' -> 'true', 'spring.cloud.stream.kafka.binder.brokers' -> 'PLAINTEXT://kafka-broker:9092']], resource = org.springframework.cloud.stream.app:s3-sink-kafka:jar:3.2.1]] to platform [default]. Error Message = [App with deploymentId [alert-test-1.s3-sink-v1] with state [deployed] doesn't match expected state [unknown]]
Caused by: java.lang.IllegalStateException: App with deploymentId [alert-test-1.s3-sink-v1] with state [deployed] doesn't match expected state [unknown]
at org.springframework.util.Assert.state(Assert.java:76) ~[spring-core-5.3.29.jar:5.3.29]
at org.springframework.cloud.deployer.spi.local.LocalAppDeployer.validateStatus(LocalAppDeployer.java:351) ~[spring-cloud-deployer-local-2.9.0-SNAPSHOT.jar:na]
at org.springframework.cloud.deployer.spi.local.LocalAppDeployer.deploy(LocalAppDeployer.java:132) ~[spring-cloud-deployer-local-2.9.0-SNAPSHOT.jar:na]
at org.springframework.cloud.skipper.server.deployer.DefaultReleaseManager.install(DefaultReleaseManager.java:142) ~[spring-cloud-skipper-server-core-2.11.0-SNAPSHOT.jar:2.11.0-SNAPSHOT]
... 53 common frames omitted
2023-10-31 15:40:21.606 INFO 1 --- [eTaskExecutor-4] o.s.s.support.LifecycleObjectSupport : stopped org.springframework.statemachine.support.DefaultStateMachineExecutor@7dfa153c
2023-10-31 15:40:21.606 INFO 1 --- [eTaskExecutor-4] o.s.s.support.LifecycleObjectSupport : stopped INSTALL_INSTALL INSTALL_EXIT / / uuid=0d372249-1980-4bc3-9a1b-450e6bec435d / id=alert-test-1
2023-10-31 15:40:21.606 INFO 1 --- [eTaskExecutor-4] o.s.c.s.s.s.StateMachineConfiguration : Entering state ObjectState [getIds()=[ERROR], getClass()=class org.springframework.statemachine.state.ObjectState, hashCode()=851729950, toString()=AbstractState [id=ERROR, pseudoState=null, deferred=[], entryActions=[org.springframework.cloud.skipper.server.statemachine.ErrorAction@63dd81ee], exitActions=[], stateActions=[], regions=[], submachine=null]]
2023-10-31 15:40:21.607 INFO 1 --- [eTaskExecutor-4] o.s.c.s.s.s.SkipperStateMachineService : setting future exception
This is my stream properties:
app.s3-processor.cloud.aws.credentials.accessKey=******
app.s3-processor.cloud.aws.credentials.secretKey=******
app.s3-processor.cloud.aws.region.static=ap-southeast-2
app.s3-processor.cloud.aws.stack.auto=false
app.s3-processor.error.test=0
app.s3-processor.management.endpoints.web.exposure.include=health,info,bindings,prometheus
app.s3-processor.management.metrics.tags.application.guid=${spring.cloud.application.guid:unknown}
app.s3-processor.management.metrics.tags.application.name=${vcap.application.application_name:${spring.cloud.dataflow.stream.app.label:unknown}}
app.s3-processor.management.metrics.tags.application.type=${spring.cloud.dataflow.stream.app.type:unknown}
app.s3-processor.management.metrics.tags.application=${spring.cloud.dataflow.stream.name:unknown}-${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}
app.s3-processor.management.metrics.tags.instance.index=${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}
app.s3-processor.management.metrics.tags.stream.name=${spring.cloud.dataflow.stream.name:unknown}
app.s3-processor.server.port=20012
app.s3-processor.source.bucket=aija-src-2
app.s3-processor.spring.cloud.stream.instanceCount=1
app.s3-processor.spring.cloud.stream.kafka.binder.brokers=PLAINTEXT://kafka-broker:9092
app.s3-processor.spring.cloud.stream.kafka.binder.zknodes=zookeeper:2181
app.s3-processor.spring.cloud.stream.kafka.streams.binder.brokers=PLAINTEXT://kafka-broker:9092
app.s3-processor.spring.cloud.stream.kafka.streams.binder.zknodes=zookeeper:2181
app.s3-processor.spring.kafka.streams.properties.metrics.recording.level=DEBUG
app.s3-processor.target.bucket=aija-dst-2
app.s3-processor.wavefront.application.name=${spring.cloud.dataflow.stream.name:unknown}
app.s3-processor.wavefront.application.service=${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}-${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}
app.s3-sink.cloud.aws.credentials.accessKey=******
app.s3-sink.cloud.aws.credentials.secretKey=******
app.s3-sink.cloud.aws.region.static=ap-southeast-2
app.s3-sink.cloud.aws.stack.auto=false
app.s3-sink.consumer.bucket=aija-dst-2
app.s3-sink.consumer.key-expression=headers['targetCompleteFileName']
app.s3-sink.management.endpoints.web.exposure.include=health,info,bindings,prometheus
app.s3-sink.management.metrics.tags.application.guid=${spring.cloud.application.guid:unknown}
app.s3-sink.management.metrics.tags.application.name=${vcap.application.application_name:${spring.cloud.dataflow.stream.app.label:unknown}}
app.s3-sink.management.metrics.tags.application.type=${spring.cloud.dataflow.stream.app.type:unknown}
app.s3-sink.management.metrics.tags.application=${spring.cloud.dataflow.stream.name:unknown}-${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}
app.s3-sink.management.metrics.tags.instance.index=${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}
app.s3-sink.management.metrics.tags.stream.name=${spring.cloud.dataflow.stream.name:unknown}
app.s3-sink.server.port=20013
app.s3-sink.spring.cloud.stream.bindings.input.consumer.bucket=aija-dst-2
app.s3-sink.spring.cloud.stream.bindings.input.consumer.key-expression=headers['targetCompleteFileName']
app.s3-sink.spring.cloud.stream.instanceCount=1
app.s3-sink.spring.cloud.stream.kafka.binder.brokers=PLAINTEXT://kafka-broker:9092
app.s3-sink.spring.cloud.stream.kafka.binder.zknodes=zookeeper:2181
app.s3-sink.spring.cloud.stream.kafka.streams.binder.brokers=PLAINTEXT://kafka-broker:9092
app.s3-sink.spring.cloud.stream.kafka.streams.binder.zknodes=zookeeper:2181
app.s3-sink.spring.kafka.streams.properties.metrics.recording.level=DEBUG
app.s3-sink.wavefront.application.name=${spring.cloud.dataflow.stream.name:unknown}
app.s3-sink.wavefront.application.service=${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}-${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}
app.s3-src.cloud.aws.credentials.accessKey=******
app.s3-src.cloud.aws.credentials.secretKey=******
app.s3-src.cloud.aws.region.static=ap-southeast-2
app.s3-src.cloud.aws.stack.auto=false
app.s3-src.file.consumer.markers-json=false
app.s3-src.management.endpoints.web.exposure.auth.enabled=false
app.s3-src.management.endpoints.web.exposure.include=health,info,bindings,prometheus
app.s3-src.management.metrics.export.prometheus.enabled= true
app.s3-src.management.metrics.tags.application.guid=${spring.cloud.application.guid:unknown}
app.s3-src.management.metrics.tags.application.name=${vcap.application.application_name:${spring.cloud.dataflow.stream.app.label:unknown}}
app.s3-src.management.metrics.tags.application.type=${spring.cloud.dataflow.stream.app.type:unknown}
app.s3-src.management.metrics.tags.application=${spring.cloud.dataflow.stream.name:unknown}-${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}
app.s3-src.management.metrics.tags.instance.index=${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}
app.s3-src.management.metrics.tags.stream.name=${spring.cloud.dataflow.stream.name:unknown}
app.s3-src.server.port=20011
app.s3-src.spring.cloud.stream.instanceCount=1
app.s3-src.spring.cloud.stream.kafka.binder.brokers=PLAINTEXT://kafka-broker:9092
app.s3-src.spring.cloud.stream.kafka.binder.zknodes=zookeeper:2181
app.s3-src.spring.cloud.stream.kafka.streams.binder.brokers=PLAINTEXT://kafka-broker:9092
app.s3-src.spring.cloud.stream.kafka.streams.binder.zknodes=zookeeper:2181
app.s3-src.spring.kafka.streams.properties.metrics.recording.level=DEBUG
app.s3-src.supplier.auto-create-local-dir=false
app.s3-src.supplier.delete-remote-files=true
app.s3-src.supplier.local-dir=false
app.s3-src.supplier.remote-dir=aija-src-2
app.s3-src.wavefront.application.name=${spring.cloud.dataflow.stream.name:unknown}
app.s3-src.wavefront.application.service=${spring.cloud.dataflow.stream.app.label:unknown}-${spring.cloud.dataflow.stream.app.type:unknown}-${vcap.application.instance_index:${spring.cloud.stream.instanceIndex:0}}
deployer.s3-processor.bootVersion=2
deployer.s3-processor.count=1
deployer.s3-sink.bootVersion=2
deployer.s3-sink.count=1
deployer.s3-src.bootVersion=2
deployer.s3-src.count=1
spring.cloud.dataflow.skipper.platformName=default
The weird thing here is that with the same properties, I have created and deployed streams without any issues. Even it will work this time If I destroy the current and create new one or try restarting the dataflow-server and Skipper. It doesn't seem like I missed something at the properties level; instead, it appears to be related to cache or similar issues.
Please ask if any further information is required and help resolve this. Thanks in advance."