ssh_keys not working on droplet create #80
Comments
|
Ah, @shortdudey123 thanks for opening an issue. We don't include the ssh keys that a droplet was created with in the response, which is why it returns However, the keys should have been used to actually create the droplet - with valid ids/keys does that call work? |
No, the ssh_keys array does not get passed onto the droplet for creation. I am unable to ssh to the new droplet. If i create one through the UI using the same ssh key it works. |
|
Hey @shortdudey123, This is what I use to pull my premade keys from DO. If you are trying to add a key that you haven't added to DO previously you could accomplish this inside the userdata section of the droplet kit. edit: put the wrong username :P |
|
@CloudCowboyCo the ssh key i am trying to pass in the array already exists on DO. That is verified with the first line of the output in my original post :) |
|
Sorry, I saw the value as nil when returned. I should have read more closely. This is how I accomplish spinning up a droplet with my keys already on the system https://github.com/CloudCowboyCo/do-cocaine/blob/master/droplet_deploy.rb Let me know if this helps you out. |
|
Thanks, i am using the knife-digital_ocean gem to create the droplets (already ruled that out since i know that the ssh key is being passed to stuff in the |
|
@shortdudey123 our logs show that the API request was received with a ssh key and that the droplet was created with it. It might help to open a ticket with some of the details (https://cloud.digitalocean.com/support/tickets/new) and we can help you debug further. |
|
Done, thanks |
|
Hey @shortdudey123 - did you find out what was the problem? Out of the blue I started to see exactly the same issues with kitchen-digitalocean (it uses droplet_kit under the hood). |
|
@jwadolowski nope, by the time digital ocean support got back to be I was unable to reproduce the issue |
|
@jwadolowski what are the symptoms you're seeing? Don't believe this is a droplet kit specific issue: we've verified that DropletKit is passing the correct params and Droplets are being created with ssh keys. Note that after a droplet (VM) is created, it may take some time for the OS and |
|
one thing i had thought about doing, but never did was to do a root password reset then use that to login through the web shell interface and see if the pub key is in the |
|
@phillbaker actually symptoms are the same as originally reported by @shortdudey123 - SSH key is not passed through to my droplet. It's definitely not the case related to boot time and To narrow it down a little bit I wrote and executed this (it is essentially what happens in kitchen-digitalocean): require 'droplet_kit'
client = DropletKit::Client.new(access_token: ENV['DIGITALOCEAN_ACCESS_TOKEN'])
client.ssh_keys.find(id: ENV['DIGITALOCEAN_SSH_KEY_IDS'])
droplet = DropletKit::Droplet.new(
name: 'droplet-kit-test',
image: 'centos-7-0-x64',
size: '1gb',
region: 'fra1',
ssh_key_ids: ENV['DIGITALOCEAN_SSH_KEY_IDS'],
private_networking: true,
ipv6: false
)
d = client.droplets.create(droplet)
d.ssh_keysLast line returns Env variables: |
|
Good point @shortdudey123. I've just logged in and root's |
|
Thanks for checking the authorized keys file @jwadolowski, that's helpful. I can escalate that issue with our internal team. Note that we don't include the ssh keys that a droplet was created with in the response, which is why |
|
Thanks @phillbaker! Looking forward to an update. |
|
Reopening due to @jwadolowski being able to replicate this issue |
|
@jwadolowski @shortdudey123 some follow up that would help us debug:
|
Unfortunately not. Here's the
Yes, just used
Didn't check that, but will do shortly.
When it comes to kitchen-digitalocean this is the shortest way to reproduce this issue:
---
<% chef_versions = %w( 12 ) %>
<% platforms = %w( centos-6-5-x64 centos-7-0-x64 debian-7-0-x64 ) %>
driver:
name: digitalocean
provisioner:
name: chef_zero
platforms:
<% platforms.each do |p| %>
<% chef_versions.each do |chef_version| %>
- name: <%= p %>-chef-<%= chef_version %>
driver:
image: <%= p %>
driver_config:
region: fra1
size: 1gb
require_chef_omnibus: <%= chef_version %>
<% end %>
<% end %>
suites:
- name: default
run_list:
- recipe[do_test::default]Right after that please execute Output I see at the moment: Correct one looks like this: Totally strange thing about it is that it works kinda non deterministic. Sometimes it works, sometimes it doesn't. I've been using the same flow for months (if not years now) without any issues. Yesterday, out of the blue, it just stopped working.
Actually I've raised one already (ticket ID: 966628) but it refers to droplet that no longer exists. Just created another one for affected droplets that are still running (ticket ID: 968368). |
Here's the curl command I just executed: curl \
-X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DIGITALOCEAN_ACCESS_TOKEN}" \
-d "{ \"name\":\"curl-test\", \"region\":\"fra1\", \"size\":\"1gb\", \"image\":\"centos-7-0-x64\",\"ssh_keys\":[\"${DIGITALOCEAN_SSH_KEY_IDS}\"], \"ipv6\":false, \"private_networking\":true}" \
"https://api.digitalocean.com/v2/droplets"and surprisingly... it works perfectly fine. |
|
I've been able to reproduce this issue using Something we've been digging into is looking at Since both reports are on Centos 7, can you try reproducing this on other versions of Centos or other distributions? |
|
Sure thing. Will get back to you with test results as soon as possible. |
|
Just wrote a couple of scripts to make testing easier. Here's require 'droplet_kit'
amount = 3
images = %w(
centos-5-8-x64
centos-6-5-x64
centos-7-0-x64
fedora-22-x64
debian-7-0-x64
debian-8-x64
ubuntu-14-04-x64
)
images.each do |image|
for i in 1..amount
client = DropletKit::Client.new(
access_token: ENV['DIGITALOCEAN_ACCESS_TOKEN']
)
droplet = DropletKit::Droplet.new(
name: "droplet-kit-#{image}-#{i}",
image: image,
size: '512mb',
region: 'fra1',
ssh_key_ids: ENV['DIGITALOCEAN_SSH_KEY_IDS'],
private_networking: true,
ipv6: false
)
client.droplets.create(droplet)
end
endScript I've used for testing purposes: #!/usr/bin/env bash
tugboat droplets | cut -d' ' -f1,3 | tr -d ',' | while read line; do
droplet_name=$(echo ${line} | cut -d' ' -f1)
droplet_ip=$(echo ${line} | cut -d' ' -f2)
ssh_out=$(ssh -o StrictHostKeyChecking=no -oBatchMode=yes -l root $droplet_ip 'uname -a' 2>/dev/null; echo $?)
echo "${droplet_name}: $ssh_out"
doneAccording to
Output: Unfortunately my SSH key was not injected into any of these droplets. To be 100% sure I've cherry picked a few of them, but my script was right: |
|
Same procedure, but pure #!/usr/bin/env bash
images=("centos-5-8-x64" "centos-6-5-x64" "centos-7-0-x64" "fedora-22-x64" "debian-7-0-x64" "debian-8-x64" "ubuntu-14-04-x64")
for i in ${images[@]}; do
curl \
-X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ${DIGITALOCEAN_ACCESS_TOKEN}" \
-d "{ \"name\":\"curl-${i}\", \"region\":\"fra1\", \"size\":\"512mb\", \"image\":\"${i}\",\"ssh_keys\":[\"${DIGITALOCEAN_SSH_KEY_IDS}\"], \"ipv6\":false, \"private_networking\":true}" \
"https://api.digitalocean.com/v2/droplets"
doneHad to update my test script a little bit to prevent #!/usr/bin/env bash
tugboat droplets | cut -d' ' -f1,3 | tr -d ',' | while read line; do
droplet_name=$(echo ${line} | cut -d' ' -f1)
droplet_ip=$(echo ${line} | cut -d' ' -f2)
ssh_out=$(ssh -n -o StrictHostKeyChecking=no -oBatchMode=yes -l root $droplet_ip 'uname -a' 2>/dev/null >/dev/null; echo $?)
echo "${droplet_name}: $ssh_out"
doneFinal output: Did some manual login attempts to confirm that's true and indeed I was able to log in to every single droplet created by |
|
@jwadolowski glad you were able to reproduce this! thanks for doing the extended testing to verify it |
|
@jwadolowski I think the parameter name is So this looks like it wouldn't work? ssh_key_ids: ENV['DIGITALOCEAN_SSH_KEY_IDS'],Would you mind updating and trying one more time? |
|
Sorry about that, too much copying and pasting. Updated Ruby code now contains correct hash key and mimics the same logic as in require 'droplet_kit'
amount = 3
images = %w(
centos-5-8-x64
centos-6-5-x64
centos-7-0-x64
fedora-22-x64
debian-7-0-x64
debian-8-x64
ubuntu-14-04-x64
)
images.each do |image|
for i in 1..amount
client = DropletKit::Client.new(
access_token: ENV['DIGITALOCEAN_ACCESS_TOKEN']
)
droplet = DropletKit::Droplet.new(
name: "droplet-kit-#{image}-#{i}",
image: image,
size: '512mb',
region: 'fra1',
ssh_keys: ENV['DIGITALOCEAN_SSH_KEY_IDS'].to_s.split(/, ?/),
private_networking: true,
ipv6: false
)
client.droplets.create(droplet)
end
endRun that a few times, but all was good all the time: Did yet another try and created droplet using test kitchen: Decided to run my test just for this droplet and this happened: Since it happens mostly on This means something may be wrong with The fact we've been using that for months without issues concerns me most :) How did it happen that a few days ago we just stumbled upon such problems? Will let |
|
@jwadolowski thanks for the update! We're investigating issues around cloud-init and centos7 specifically, so there may be something there. |
|
@phillbaker, I did a few more tests (results are available here) and indeed it seems to be CentOS 7 specific. |
|
@phillbaker did you guys manage to track this bug down? It's kinda annoying that we have reiterate a couple of times before we get usable droplet. |
|
Hi @phillbaker, I managed to partially solve the problem (at least the part related to Test Kitchen DigitalOcean driver), but something's still wrong (possibly with cloud-init). You can find all the details here: test-kitchen/kitchen-digitalocean#45 (comment) Droplet that got stuck is still running and I'm going to raise a ticket in DO portal, so you can take a look at that. |
|
@jwadolowski, thanks for the followup. @kitschysynq is probably the best person to contact about that. |
I am trying to create a droplet, however, the ssh_keys array does not appear to get passed along during creation
The text was updated successfully, but these errors were encountered: