How can I scale CloudFoundry applications "down" without the risk of restarting all of them?

Question

This is a question regarding the Swisscom Application Cloud.

I have implemented a strategy to restart already deployed CloudFoundry applications without using cf restart APP_NAME. I wanted to be able to:

• restart running applications without needing access the app manifest and
• avoid them suffering any down-time.

The general concept looks like this:

1. cf scale APP_NAME -I 2

   • increasing the instance count of the app from 1 to 2
   • wait for all app instances to be running

2. cf restart-app-instance APP_NAME 0

   • restart the "old" app instance
   • wait for all app instances to be running again

3. cf scale easyasset-repower-staging -I 1

   • decrease the instance count of the app back from 2 to 1

This generally works and usually does what I expect it to do. The problem I am having occurs at Stage (3), where sometimes instead of just scaling the instance count back, CloudFoundry will also restart all (remaining) instances.

I do not understand:

• Why does this happen only sometimes (all apps restart when scaling down)?
• Shouldn't CloudFoundry keep the the remaining instances up and running?
• If cf scale is not able to keep perfectly fine running app instances alive - when is it useful?

Please Note: I am well aware of the Bluegreen / Autopilot plugins for zero-down-time deployment of applications in CloudFoundry and I am actually using them for our deployments from our build server, but they require me to provide a manifest (and additional credentials), which in this case I don't have access to (unless I can somehow extract it from a running app via cf create-app-manifest?).

Update 1: Looking at the plugins again I found bg-restage, which apparently does approximately what I want, but I have no idea how reliable that one is.

Update 2: I have concluded that it's probably an obscure issue (or bug) in CloudFoundry and that there are no guarantees given by cf scale that existing instances are to remain running. As pointed out above, I have since realised that it is indeed very much possible to generate the app manifest on the fly (cf create-app-manifest) and even though I couldn't use the bg-restage plugin without errors, I reverted back to the blue-green-deploy plugin, which I can now hand a freshly generated manifest to avoid this whole cf scale exercise.

Comment Questions:

Why do you have the need to restart instances of your application?

We are caching some values from persistent storage on start-up. This restart is happening when changes to that data was detected.

information about the health-check

We are using all types of health checks, depending on which app is to be re-started (http, process and port). I have observed this issue only for apps with health checkhttp. I also have ahttp-endpoint` defined for the health check.

Are you trying to alter the memory with cf scale as well?

No, I am trying to keep all app configuration the same during this process.

Answer 1

When you have two running instances, the command

cf scale <APP> -i 1

will kill instance #1 and instance #0 will not be affected.