Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Instance scheduler is failing to start instance at its scheduled time #332

Closed
lady-wanderer opened this issue Nov 1, 2022 · 10 comments
Closed

Comments

@lady-wanderer
Copy link

Summary:

  1. Lambda does not experience errors on days the instance fails to start.
  2. On days the instance fails to start, the StartInstances API is never called for the affected EC2 Instance.
  3. The Schedule period is correctly constructed, but the Function does not seem to follow the rule set in the function. Most invocations in the month of October are on Monday and Wednesday, with some on Thursday and Friday.

I have a schedule configured from 00:00 est to 21:00 pm est, Monday-Friday. Most days, the solution works and stops and starts instances at the correct time. However, sometimes the solution will not start an instance after it has been stopped. Subsequent runs after the start time, do not trigger the instance to start either.

@labkey-jony
Copy link

@lady-wanderer Are you on the newest version of the scheduler?

My org's seeing a similar issue where EC2 instances are not starting up in the mornings. However, we're on 1.3.1 and we're hoping bumping to 1.4.1 will remedy this.

@gockle
Copy link
Member

gockle commented Nov 21, 2022

Hi @lady-wanderer is this still an issue, if so can you provide the information which version of the solution is being used, schedule, period configurations.

@lady-wanderer
Copy link
Author

lady-wanderer commented Nov 21, 2022 via email

@lady-wanderer
Copy link
Author

@lady-wanderer Are you on the newest version of the scheduler?

My org's seeing a similar issue where EC2 instances are not starting up in the mornings. However, we're on 1.3.1 and we're hoping bumping to 1.4.1 will remedy this.

Upgrading should remedy this.

@labkey-jony
Copy link

labkey-jony commented Nov 22, 2022

Confirmed on my end as well. I had to both upgrade to 1.4.1 and also make sure the KMS:CreateGrant permission was added to my scheduler role since we're using encrypted EBS volumes and set my resource for that permission to cover those as well. Something like "Resource": ["arn:aws:kms:us-west-2:AWS_ACCOUNT_ID:key/*"]

@easante817
Copy link

I have added the KMS:CreateGrant but still not working. wondered if I did it wrong

@hearde hearde added the bug label Jun 16, 2023
@jmd-au

This comment was marked as resolved.

@CrypticCabub
Copy link
Member

Hi @jmd-au

have you tried adding permission to access the KMS key to your scheduler role as described in the implementation guide here?
https://docs.aws.amazon.com/solutions/latest/instance-scheduler-on-aws/plan-your-deployment.html

If so, is there anything that is particularly confusing or unclear?
We are looking to improve the troubleshooting documentation around this particular issue so any extra information on what customers need to be able to understand the solution as easily as possible is much appreciated!

@jmd-au
Copy link

jmd-au commented Jun 21, 2023

A colleague came to the same conclusion (needing to add kms:CreateGrant to the policy), I added it yesterday and now the scheduler is functioning as expected.

The main thing is that for something that should be done to conform with best practice (encryption at rest), I would suggest that this should be highlighted more than an single line in the documentation.

Another suggestion is that this could somewhat easily be included through a boolean in the CloudFormation template. e.g: Are you using KMS encryption on your EBS volumes for EC2? - which would then conditionally add the statement to the policy.

@CrypticCabub
Copy link
Member

v3.0.0 now includes a CloudFormation parameter to provide the scheduler with CreateGrant permissions to KMS keys

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants