Authenticate as proxy for automation

[Aguay-val]

Describe the bug

I would like to run several command through target, with ansible, but the behavior make me think it should not work like this.

With the command below :

ssh -o "ProxyCommand=ssh -t -W %h:%p -q my-passhport.fqdn" my-target

I'm able to authenticate through passhport to reach my target.
But when the first step is done, the second, ssh connection to my-target is waiting my ssh key not the passhport one.

I though passhport would make the connection to my-target but it seems that i have to authenticate myself.

I don't know if it's clear.

To Reproduce

Create a user1
Create target1

Copy passhport ssh key to target1.
Grant access to user1 on target1.

Try my command :

ssh -o "ProxyCommand=ssh -t -W %h:%p -q my-passhport.fqdn" my-target

Expected behavior

My expected behavior is that i don't have to copy my ssh key on the target because i'm using the passhport gateway.

[Raphux]

Hi there,

sorry for the delayed answer. After spending a few hours on trying to make it works, I couldn't find a way.
The solution proposed in this post is only valid if you have your ssh pub key in the ~/.ssh/authorized_keys on the target host.

I'll search a little bit more, but I'm not very optimist at the moment.

[Kinvaras]

Hi @Aguay-val , @Raphux , I'm stuck with the exact same problem, do you have any update on this ? Thanks

[exploitation-nehsdigital]

We wrote a specific script that generates our inventory these way.
No need to push our personnal keys on targets

[tg_all]
traget_name ansible_host=target_ip ansible_user=target_user

[tg_all:vars]
ansible_ssh_common_args='-o ProxyCommand="ssh -t -W %h:%p -q -p passhport_port passhport@passhport_ip"'

[Raphux]

That is a really nice workaround !

[golflima]

Hi,

I just followed the doc in https://github.com/LibrIT/passhport/blob/6ca5f7ead83437ba51cd5adb5354850f05003bca/docs/installation-and-configuration/make-passhport-more-transparent-for-user.rst
and ... I have an issue when I try to connect through Ansible :(

I generate the PaSSHport configuration in aim to have targets named like <ansible_user>@<inventory_hostname>, and I added in my all group in the inventory these vers:

    ansible_ssh_common_args: -o 'SetEnv PASSHPORT_TARGET={{ ansible_user }}@{{ inventory_hostname }}' -o 'SendEnv PASSHPORT_TARGET'
    ansible_ssh_host: "{{ passhport_url }}"
    ansible_ssh_port: "{{ passhport_port }}"
    ansible_ssh_user: passhport
    passhport_port: 22
    passhport_url: bastion.domain.tld

Sadly, within Ansible I get the error :

$ ansible -i inventory/ -m ping mytargetname
mytargetname | FAILED! => {
    "msg": "failed to resolve remote temporary directory from ansible-tmp-1622046353.6898282-1624-75583294829257: `( umask 77 && mkdir -p \"` echo ~/.ansible/tmp `\"&& mkdir \"` echo ~/.ansible/tmp/ansible-tmp-1622046353.6898282-1624-75583294829257 `\" && echo ansible-tmp-1622046353.6898282-1624-75583294829257=\"` echo ~/.ansible/tmp/ansible-tmp-1622046353.6898282-1624-75583294829257 `\" )` returned empty string"
}

Thanks to PaSSHport, I was able to retrieve the commands played by Ansible via the logs :) So I know Ansible is connecting correctly to the target via PaSSHport, and I know Ansible is correctly sending the commands.

So I tried manually.

ssh -o 'SetEnv PASSHPORT_TARGET=xxx@server' -o 'SendEnv PASSHPORT_TARGET' [email protected]
Connecting to: xxx@server...
$

This works ! I have access to my target directly, and I can do whatever I want. For that, great I'm happy :D

$ ssh -o 'SetEnv PASSHPORT_TARGET=xxx@server' -o 'SendEnv PASSHPORT_TARGET' [email protected] 'echo $HOME'
Host key verification failed.

But if I append a command it's broken :(

$ ssh -o 'SetEnv PASSHPORT_TARGET=xxx@server' -o 'SendEnv PASSHPORT_TARGET' -t [email protected] 'echo $HOME'
xxx@sever_ip: Permission denied (publickey).
Connection to bastion.domain.tld closed.

And it is not better (but different) with -t ...

Any idea ?

So close to have a fully working integration of PaSSHport with Ansible in our infra...

[elg]

I will have to do some tests around this.
However, AFAIK openssh has multiple method to handle connection. An connection in interactive mode is totally different than direct commands. Actually direct commands doesn't allows "return" char (\n I guess) and disconnect directly.

Are you sure that ansible is using this exact syntax ?
Anyway I'll do tests.

[golflima]

Ansible uses exactly this syntax: (you can see it by appending -vvvvv (yeah, I may have added extra vs!) to the command)

ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="passhport"' -o ConnectTimeout=10 -o 'SetEnv PASSHPORT_TARGET=user@server' -o 'SendEnv PASSHPORT_TARGET' -o ControlPath=/home/user/.ansible/cp/ecebc325eb bastion.domain.tld '/bin/sh -c '"'"'echo "`pwd`" && sleep 0'"'"''

Thanks for your answer, I'll try to investigate more tomorrow.

[elg]

I did some test, everything works fine here. I didn't test the whole complex line from ansible but I assume it will works too.

That is what I did:

1. Install last PaSSHport version (updated on a running passhport)
2. Add "AcceptEnv PASSHPORT_TARGET" at the end of /etc/ssh/sshd_config of my passhport server
3. systemctl reload sshd
4. add a user and his pubkey to passhport
5. add this user to a target (I did on 2 differents target to be sure)
6. from this user terminal (another machine) : ssh -o 'SetEnv PASSHPORT_TARGET=Krank' -o SendEnv=PASSHPORT_TARGET -t passhport@bastion 'echo $HOME'

I got the result in a second.
I didn't try with "@" in the target name as you did... I will right now :)

[golflima]

Ahem :D

OK, I found the problem. I did a test on a server with SSHD not bind on 22 (used by git), and, for some reason, when you run a direct command, the port is not set correctly.
I retried on a server with SSHD on 22 and it works fine (with and without -t). But as soon it's not the default port, the configuration of the target is ignored :(

Now I need to figure out why I have a problem with Ansible.

[elg]

Nice to hear! Your target is defined with a different port and it's not working? If it's the case it's definitively a bug that I could correct rapidly.

[golflima]

Yeah, I have SSH for git running on 22 on it, and the real SSHD running on 2020.
I did a passhport-admin target show user@server and the config is accurate, so it looks like a bug indeed.
Glad you can fix it rapidly :)

I'll continue to fight with Ansible then :D

[golflima]

OK, this is very strange, I retried the command ran by Ansible manually, and it works fine (on a server with SSHD on 22 of course).
I wonder if the fact Ansible sends multiple commands in few time can be problematic with PaSSHport running in dev server, I'm going to setup a WSGI server in front for it.
Maybe a cache problem too...

Thanks for your support :)

[elg]

Correction done... Sorry for the delay: I changed a box port, and didn't notice a firewall on everything else than 22... took me some time to be able to reconnect to it :)

Let me know if it's OK now.

Oups didn't saw your last message. Some tips:

1. WSGI is mandatory in production.
2. You'll have to put some persistent socket. @phlb shows me that once. I dont remember the syntax but you have to put it in the target options. It decrease dramatically the times of ansible playbook execution. I should add it to documentation... If I find time tomorrow I'll do it.

[golflima]

Hi, thank you for the quick fix and your tips !

It's now working like a charm :)

The Ansible / WSGI part will be done next week due to lack of time, but I will keep you updated, and I'll post here the working procedure.

[golflima]

I installed uwsgi for PaSSHport (btw, the official doc says it's not mandatory to use Apache or NGINX in front of uWSGI as long the app doesn't serve static files over http, and since you only expose an API, it should be fine and also easier to setup (less things to install to run PaSSHport)). and now Ansible is working :D

But ... As soon as I run commands on multiple hosts, it fails.

For example :

• ansible -i inventory/ -m ping server1 works
• ansible -i inventory/ -m ping all doesn't work (same error output as before)

My understanding is that Ansible throws too much connections simultaneously and PaSSHport can't handle them.

I'm now playing with ansible_command_timeout and ansible_ssh_pipelining to improve the stability, but I also consider having more workers on uWSGI. I keep you updated.

Edit: With ansible_ssh_pipelining: 'True' and ansible_command_timeout: 10 I have something working on multiple hosts, but it's not very stable yet.

[golflima]

OK, I managed to have a complex playbook running through PaSSHport with Ansible on a single host correctly.

But, I struggle to have it working on multiple hosts.

My Ansible configuration:

    ansible_command_timeout: 10
    ansible_scp_if_ssh: 'True'
    ansible_ssh_common_args: -o 'SetEnv PASSHPORT_TARGET={{ ansible_user }}@{{ inventory_hostname }}' -o 'SendEnv PASSHPORT_TARGET'
    ansible_ssh_host: "{{ passhport_url }}"
    ansible_ssh_pipelining: 'True'
    ansible_ssh_port: "{{ passhport_port }}"
    ansible_ssh_user: passhport
    passhport_port: 22
    passhport_url: bastion.domain.tld

And, for PaSSHport, I have uWSGI working with 10 processes (it was also working with 5, I just increased to test).

On multiple hosts, I have strange behaviors :

• Random hosts are failing: on 6 servers, I have 3 responding well, the 3 others are facing get the error user@server: Permission denied (publickey)... surprizingly, it's the same user@server for all servers! Just like it was in fact reusing the same connexion... Weird
• Sometimes, all hosts are failing
• Then, it doesn't work anymore when I try on a single host
• Then, after a moment or if I rm -rf ~/.ansible, it works again randomly...

And once, I ran a complex playbook on multiple servers, and I saw a part of deployment done on the wrong server (thanks --diff) !

So, for now, I'm confident in using PaSSHport with Ansible for a single host, and I'm looking for another solution to run it for multiple hosts, I'm thinking about connecting through PaSSHport to an intermediate server where we will run Ansible (or maybe inside a CI/CD) so we would still have logs at least, but we loose the access control ...

[elg]

Hi,
I'm really sorry I missed that.
I tested passhport with ansible without any issues. However, I've put some specific option to passhport-connect.sh (in passhport/passhport directory) Line 15:
OPTIONS="-C -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=/tmp/ socket-%u-to-%r@%h:%p"

I hope you didn't give up and you figured out without me :)

Again sorry for the delay, the github notice was in my spams.