NoDelayProvisionStrategy won't provision when starting with 0 instances in auto-scaling fleet

[jzila]

I've found that if I let my fleet drop to 0 instances, and then wait a while, the plugin seems to lose track of what's going on and won't provision instances until I get up to 2 jobs pending. This only seems to happen with the NoDelayProvisionStrategy.

Apr 10, 2020 6:55:15 AM INFO com.amazon.jenkins.ec2fleet.NoDelayProvisionStrategy apply
currentDemand -2 availableCapacity 3 (availableExecutors 0 connectingExecutors 1 plannedCapacitySnapshot 2 additionalPlannedCapacity 0)
...
Apr 10, 2020 6:56:25 AM INFO com.amazon.jenkins.ec2fleet.NoDelayProvisionStrategy apply
currentDemand -1 availableCapacity 2 (availableExecutors 0 connectingExecutors 1 plannedCapacitySnapshot 1 additionalPlannedCapacity 0)

[jzila]

The issue seems to be with plannedCapacitySnapshot, which just seems to be incorrect.

[jzila]

Actually it also seems like currentDemand is incorrect.

I have:
2 builds in the queue, 1 executor running a third build, and 1 executor connecting. The logs report:

Apr 10, 2020 7:16:25 AM INFO com.amazon.jenkins.ec2fleet.NoDelayProvisionStrategy apply
currentDemand -1 availableCapacity 3 (availableExecutors 0 connectingExecutors 1 plannedCapacitySnapshot 2 additionalPlannedCapacity 0)

[terma]

thanks for logs, I spent a lot of time but still can't reproduce this beast, is that stable fleet, I mean you created it a lot of time ago and didn't change settings? just go down to zero and fail, is that correct assumption?

[jzila]

It's been stable for days and it's still giving me this issue.

currentDemand 0 availableCapacity 1 (availableExecutors 0 connectingExecutors 0 plannedCapacitySnapshot 1 additionalPlannedCapacity 0)

[jzila]

Happy to give you any logs you need.

[jzila]

Actually I've rebooted Jenkins on the newest version and it seems to be picking up builds for new nodes now. Feel free to ignore this issue for now, I'll report back in a few days and close if there's no more repro.

[terma]

Ok, I assume Jenkins on the newest version means Jenkins version not a plugin?

[jzila]

Ok, I assume Jenkins on the newest version means Jenkins version not a plugin?

Yep, that's correct.

[jzila]

OK this has been working fine. Closing.

[cccCody]

Did you ever figure out a root cause for this, or did it just start working? I think I'm seeing the same thing and filed #425