run-tree-smoketest: create a tmpdir to use with Ansible

[miabbott]

The 'smoketest' job was failing while running the sanity test because
it was unable to copy a file from the host under test to the host
running the playbook. The error message suggested to use a temp
directory under /tmp as a mitigation.

This creates a random temp directory and sets an env var to tell
Ansible to use said temp directory.

[miabbott]

I didn't test this locally...

[miabbott]

@jlebon @cgwalters Anyone with merge privs want to have a look?

[jlebon]

Hmm, odd why this would fail now though. Wonder if it's a recent ansible update.

The error message says "did not have permissions on the remote directory", so this sounds like it's about access to a directory on the remote, rather than locally, no?

[cgwalters]

I have a feeling it's related instead to:

17:52:47 [DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and 
17:52:47 make sure become_method is 'sudo' (default). This feature will be removed in 
17:52:47 version 2.6. Deprecation warnings can be disabled by setting 
17:52:47 deprecation_warnings=False in ansible.cfg.

If we're not becoming root on the remote system correctly then that looks like an error message we'd get.

Or to say it conversely, the problem seems to relate to permissions remotely, not locally. But that said I'm happy to just try this out.

[miabbott]

@cgwalters that deprecation warning is coming from running sig-atomic-buildscripts/centos-ci/setup/setup-system.yml

---
- hosts: all
sudo: no
...

In the sanity test we are using become:

- name: Improved Sanity Test - Pre-Upgrade
  hosts: "{{ subjects | default('all') }}"
become: true
...

[miabbott]

The error message says "did not have permissions on the remote directory", so this sounds like it's about access to a directory on the remote, rather than locally, no?

@jlebon you are right. I don't know how I confused that.

There doesn't seem to be a env var for configuring the remote tmp directory, though. I guess I'll have to change this to write out an ansible.cfg that has the remote_tmp setting.

[miabbott]

Force pushed a new solution ⬆️

[jlebon]

Looks sane to me! Worth a try.

[cgwalters]

Perhaps not coincidentally I started hitting this recently running the ostree installed tests which also use Ansible.

Using -vvvv, I see:

<127.0.0.3> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1529420382.18-58464918296754 `" && echo ansible-tmp-1529420382.18-58464918296754="` echo /root/.ansible/tmp/ansible-tmp-
1529420382.18-58464918296754 `" ) && sleep 0'                                                                                                                                                                     
fatal: [/srv/libvirt/images-gold/Fedora-AtomicHost-28-20180505.0.x86_64.qcow2]: UNREACHABLE! => {                                                                                                                 
    "changed": false, 
    "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg 
to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /root/.ansible/tmp/ansible-tmp-1529420382.18-58464918296754 `\" && echo ansible-tmp-1529420382.18-58464918296754=\"` echo /root/.
ansible/tmp/ansible-tmp-1529420382.18-58464918296754 `\" ), exited with result 1, stderr output: mkdir: cannot create directory ‘/root’: Permission denied\n",                                                    
    "unreachable": true
}

However, when I log into the machine after Ansible has failed, /root/.ansible does exist. I need to trace things more. It's weird that it doesn't happen in ostree's CI.

[cgwalters]

Or wait...maybe it is about the local directory, since I'm running as non-root in my pet container.

[cgwalters]

Ah: https://bugzilla.redhat.com/show_bug.cgi?id=1580530

[cgwalters]

(And yeah, updating my f28 dev container fixed it)

[miabbott]

Strangely, the smoketest job is no longer failing because of this problem but because of #329