Fix removeUntaggedEc2: check iit-billing-tag before deleting EKS clusters

[nogueiraanderson]

Summary

Fixes critical logic bug in removeUntaggedEc2 Lambda where EKS clusters with valid billing tags were being deleted.

Problem

The Lambda was deleting EKS clusters that didn't match the skip pattern (pe-.*) without checking if they had valid billing tags.

Old (broken) logic:

• EKS cluster doesn't match pe-.* → Delete entire cluster (ignored billing tag)

New (fixed) logic:

• EKS instance has valid iit-billing-tag → Keep it (billing tag trumps everything)
• EKS cluster matches pe-.* → Keep it (protected cluster)
• EKS cluster has neither → Delete cluster

Changes

1. New Function: has_valid_billing_tag()

Validates iit-billing-tag values with support for:

• Category strings: "pmm-staging", "jenkins-pmm-slave" → Valid
• Unix timestamps: "1759837138" → Valid only if timestamp is in future (UTC)
• Expired timestamps: Treated as invalid → Instance/cluster will be deleted

2. Updated: is_eks_managed_instance()

New priority order:

1. Check billing tag first - If valid, skip (keep instance)
2. Then check skip pattern - If matches pe-.*, skip (protected cluster)
3. Neither - Mark cluster for deletion

3. Updated: is_instance_to_terminate()

Uses has_valid_billing_tag() for consistent validation across EKS and regular EC2.

Test Coverage

Comprehensive test suite with 12 scenarios in test_removeUntaggedEc2_logic.py:

Regular EC2:

• ✓ Category tag → Kept
• ✓ Future timestamp → Kept
• ✗ Expired timestamp → Terminated
• ✗ No tag, >10min → Terminated
• ○ No tag, <10min → Kept (grace period)

EKS - Protected (pe-*):

• ✓ pe-crossplane no tag → Kept (skip pattern)
• ✓ pe-infra with tag → Kept (billing tag)

EKS - Non-Protected:

• ✓ pmm-test with tag → Kept (billing tag)
• ✓ pmm-temp with future timestamp → Kept (billing tag)
• ✗ pmm-ha no tag → Cluster deleted
• ✗ pmm-expired with expired timestamp → Cluster deleted

Run tests: python3 cloud/aws-functions/test_removeUntaggedEc2_logic.py

Behavior Summary

Regular EC2:

• Has valid billing tag → Skip
• No tag + >10min → Terminate
• No tag + <10min → Skip (grace period)

EKS Instances:

• Has valid billing tag → Skip (instance is legitimate)
• No billing tag BUT matches pe-.* → Skip (protected cluster)
• No billing tag AND doesn't match pattern → Delete entire cluster

Key Points:

• Billing tag can be category string OR Unix timestamp
• Timestamps validated for expiration (UTC)
• EKS instances never terminated individually - whole cluster deleted via CloudFormation
• Pattern pe-.* whitelists Platform Engineering clusters

Files Changed

• cloud/aws-functions/removeUntaggedEc2.py - Fixed Lambda function
• cloud/aws-functions/test_removeUntaggedEc2_logic.py - Test suite (no AWS connection needed)
• cloud/aws-functions/README_removeUntaggedEc2.md - Documentation

Deployment

Before deploying, run validation:

python3 cloud/aws-functions/test_removeUntaggedEc2_logic.py

Deploy:

cd cloud/aws-functions
zip removeUntaggedEc2.zip removeUntaggedEc2.py

aws lambda update-function-code \
  --function-name removeUntaggedEc2 \
  --zip-file fileb://removeUntaggedEc2.zip \
  --region eu-west-1 \
  --profile percona-dev-admin