EC2 Driver Relies on undefined refresh! method of Aws::Credentials in at least some cases

[timvisher]

EC2 Driver Relies on undefined refresh! method of Aws::Credentials in at least some cases

Some credentials classes are non-refreshing and when those are in use
test-kitchen may trigger an inappropriate call.

Version

$ chef -v
Chef Workstation version: 21.4.365
Chef Infra Client version: 16.12.3
Chef InSpec version: 4.29.3
Chef CLI version: 4.0.0
Chef Habitat version: 1.6.288
Test Kitchen version: 2.11.2
Cookstyle version: 7.10.0

I tested this down to the latest 20 release as well.

Environment

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

Doesn't seem to have anything to do with the contents of the kitchen run.

Scenario

Use EC2 test platforms and have a decent error message if the clock is skewed.

Steps to Reproduce

I'm happy to go to the trouble of providing a little Vagrantfile/.kitchen.yml reproduction repo if there's interest in fixing this. Didn't want to go that far until someone told me I should. :)

1. Launch an Ubuntu 20.04 instance

2. Install chef-workstation like:

   curl -L https://omnitruck.chef.io/install.sh |
     bash -s -- -c stable -P chef-workstation -v 21
   

3. Setup a .kitchen.yml file with an EC2 platform. Something like:

   - name: ec2-20.04
     driver:
       name: ec2
       image_search:
         owner-id: "099720109477"
         name: "ubuntu/images/*/ubuntu-*-20.04*"
   

4. Login to the Ubuntu instance, skew the clock, and go for a create.
   Something like:

   sudo timedatectl set-ntp no
   sudo timedatectl set-time 12:17:13 # skewed past 15 minutes
   aws sts get-caller-identity
   # An error occurred (SignatureDoesNotMatch) when calling the GetCallerIdentity operation: Signature expired: 20210429T121720Z is now earlier than 20210429T170527Z (20210429T172027Z - 15 min.)
   kitchen create test-ec2-2004
   # -----> Starting Test Kitchen (v2.11.2)
   # -----> Creating <test-ec2-2004>...
   # >>>>>> ------Exception-------
   # >>>>>> Class: Kitchen::ActionFailed
   # >>>>>> Message: 1 actions failed.
   # >>>>>>     Failed to complete #create action: [undefined method `refresh!' for #<Aws::SharedCredentials:0x0000000002b97290>] on test-ec2-2004
   # >>>>>> ----------------------
   # >>>>>> Please see .kitchen/logs/kitchen.log for more details
   # >>>>>> Also try running `kitchen diagnose --all` for configuration
   

5. Unskew the clock and retry the create action:

   sudo timedatectl set-ntp yes
   sudo systemctl restart systemd-timesyncd.service
   aws sts get-caller-identity
   # …
   kitchen create test-ec2-2004
   # -----> Starting Test Kitchen (v2.11.2)
   # -----> Creating <test-ec2-2004>...
   #        Detected platform: ubuntu version 16.04 on x86_64. Instance Type: m5.large. Default username: ubuntu (default).
   # …
   

Expected Result

Anything but a call to an undefined method by the AWS SDK.

Actual Result

E, [2021-04-29T12:17:23.206405 #25650] ERROR -- Kitchen: ------Exception-------
E, [2021-04-29T12:17:23.206671 #25650] ERROR -- Kitchen: Class: Kitchen::ActionFailed
E, [2021-04-29T12:17:23.206909 #25650] ERROR -- Kitchen: Message: 1 actions failed.
>>>>>>     Failed to complete #create action: [undefined method `refresh!' for #<Aws::SharedCredentials:0x0000000002ba4a58>] on test-ec2-2004
E, [2021-04-29T12:17:23.207086 #25650] ERROR -- Kitchen: ----------------------
E, [2021-04-29T12:17:23.207162 #25650] ERROR -- Kitchen: ------Backtrace-------
E, [2021-04-29T12:17:23.207218 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/command.rb:181:in `report_errors'
E, [2021-04-29T12:17:23.207274 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/command.rb:172:in `run_action'
E, [2021-04-29T12:17:23.207332 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/command/action.rb:35:in `block in call'
E, [2021-04-29T12:17:23.207385 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/2.7.0/benchmark.rb:293:in `measure'
E, [2021-04-29T12:17:23.207442 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/command/action.rb:33:in `call'
E, [2021-04-29T12:17:23.207617 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/cli.rb:52:in `perform'
E, [2021-04-29T12:17:23.207664 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/cli.rb:198:in `block (2 levels) in <class:CLI>'
E, [2021-04-29T12:17:23.207700 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
E, [2021-04-29T12:17:23.207798 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
E, [2021-04-29T12:17:23.208033 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
E, [2021-04-29T12:17:23.209106 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'
E, [2021-04-29T12:17:23.209277 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/bin/kitchen:11:in `block in <top (required)>'
E, [2021-04-29T12:17:23.209557 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/errors.rb:170:in `with_friendly_errors'
E, [2021-04-29T12:17:23.209841 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/bin/kitchen:11:in `<top (required)>'
E, [2021-04-29T12:17:23.210094 #25650] ERROR -- Kitchen: /usr/bin/kitchen:354:in `load'
E, [2021-04-29T12:17:23.210174 #25650] ERROR -- Kitchen: /usr/bin/kitchen:354:in `<main>'
E, [2021-04-29T12:17:23.210233 #25650] ERROR -- Kitchen: ----End Backtrace-----

Additional context

I'm willing to bet dollars to pay that this is actually an upstream issue
in https://github.com/aws/aws-sdk-ruby but I'm not sure whether it would
make more sense to defend against it here while simultaneously opening a
bug report there.

[tas50]

Issue moved to test-kitchen/kitchen-ec2 #576 via ZenHub