Instance scheduler is failing to start instance at its scheduled time

[lady-wanderer]

Summary:

1. Lambda does not experience errors on days the instance fails to start.
2. On days the instance fails to start, the StartInstances API is never called for the affected EC2 Instance.
3. The Schedule period is correctly constructed, but the Function does not seem to follow the rule set in the function. Most invocations in the month of October are on Monday and Wednesday, with some on Thursday and Friday.

I have a schedule configured from 00:00 est to 21:00 pm est, Monday-Friday. Most days, the solution works and stops and starts instances at the correct time. However, sometimes the solution will not start an instance after it has been stopped. Subsequent runs after the start time, do not trigger the instance to start either.

[labkey-jony]

@lady-wanderer Are you on the newest version of the scheduler?

My org's seeing a similar issue where EC2 instances are not starting up in the mornings. However, we're on 1.3.1 and we're hoping bumping to 1.4.1 will remedy this.

[gockle]

Hi @lady-wanderer is this still an issue, if so can you provide the information which version of the solution is being used, schedule, period configurations.

[lady-wanderer]

This is no longer an issue. I went to version 1.4.1 and it fixed it. Sent from my T-Mobile 5G Device Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Nikhil Reddy ***@***.***> Sent: Monday, November 21, 2022 9:29:57 AM To: aws-solutions/aws-instance-scheduler ***@***.***> Cc: Alice Brassfield ***@***.***>; Mention ***@***.***> Subject: Re: [aws-solutions/aws-instance-scheduler] Instance scheduler is failing to start instance at its scheduled time (Issue #332) Hi @lady-wanderer<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flady-wanderer&data=05%7C01%7Calice.brassfield%40jdidata.com%7C97cd406227b94bf4b48808dacbccde03%7Caafc3fafdb8c41f98a6ff64f50ec0d5e%7C0%7C0%7C638046378024060452%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6RjQw95wkQytBnVuP3ZvX%2BAKBipCIkZRiazzDIPZ5xM%3D&reserved=0> is this still an issue, if so can you provide the information which version of the solution is being used, schedule, period configurations. — Reply to this email directly, view it on GitHub<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faws-solutions%2Faws-instance-scheduler%2Fissues%2F332%23issuecomment-1322147594&data=05%7C01%7Calice.brassfield%40jdidata.com%7C97cd406227b94bf4b48808dacbccde03%7Caafc3fafdb8c41f98a6ff64f50ec0d5e%7C0%7C0%7C638046378024060452%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G1a%2FV0Dfa1a5pI88Qz6%2BEE6wUGaNUlXR%2B%2BOITplTzbA%3D&reserved=0>, or unsubscribe<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMC6426Z5MN36XAACL6XYPLWJOBOLANCNFSM6AAAAAARUHERFM&data=05%7C01%7Calice.brassfield%40jdidata.com%7C97cd406227b94bf4b48808dacbccde03%7Caafc3fafdb8c41f98a6ff64f50ec0d5e%7C0%7C0%7C638046378024060452%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XFjyR91O1qhyKLP1lqvHd5z%2FQ62uq%2F%2BBn9wGj3RQsaM%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[lady-wanderer]

@lady-wanderer Are you on the newest version of the scheduler?

My org's seeing a similar issue where EC2 instances are not starting up in the mornings. However, we're on 1.3.1 and we're hoping bumping to 1.4.1 will remedy this.

Upgrading should remedy this.

[labkey-jony]

Confirmed on my end as well. I had to both upgrade to 1.4.1 and also make sure the KMS:CreateGrant permission was added to my scheduler role since we're using encrypted EBS volumes and set my resource for that permission to cover those as well. Something like "Resource": ["arn:aws:kms:us-west-2:AWS_ACCOUNT_ID:key/*"]

[easante817]

I have added the KMS:CreateGrant but still not working. wondered if I did it wrong

[CrypticCabub]

Hi @jmd-au

have you tried adding permission to access the KMS key to your scheduler role as described in the implementation guide here?
https://docs.aws.amazon.com/solutions/latest/instance-scheduler-on-aws/plan-your-deployment.html

If so, is there anything that is particularly confusing or unclear?
We are looking to improve the troubleshooting documentation around this particular issue so any extra information on what customers need to be able to understand the solution as easily as possible is much appreciated!

[jmd-au]

A colleague came to the same conclusion (needing to add kms:CreateGrant to the policy), I added it yesterday and now the scheduler is functioning as expected.

The main thing is that for something that should be done to conform with best practice (encryption at rest), I would suggest that this should be highlighted more than an single line in the documentation.

Another suggestion is that this could somewhat easily be included through a boolean in the CloudFormation template. e.g: Are you using KMS encryption on your EBS volumes for EC2? - which would then conditionally add the statement to the policy.

[CrypticCabub]

v3.0.0 now includes a CloudFormation parameter to provide the scheduler with CreateGrant permissions to KMS keys