Feature request: Add rsync command

[tangrufus]

• I've read the guidelines for Contributing to Roots Projects
• This request isn't a duplicate of an existing request
• This is not a personal support request that should be posted on the Roots Discourse community

Summary

Add trellis rsync which transfers files between remote servers and local machines.

Example:

rsync -aP --numeric-ids -e "ssh -T -o Compression=no -x -p 22" '[email protected]:/srv/www/example.com/shared/uploads/' /Users/me/Desktop/example.com/uploads/

Discussion

What subcommands / flags should we provide?

For reference: AWS CLI provides 3 different subcommands for moving files between s3 buckets and local machines:

aws s3 mv <source> <destation>
aws s3 sync <source> <destation>
aws s3 cp <source> <destation>

Possible solutions:

# rsync without --delete
trellis rsync cp <env> <site> <source> <destation>

trellis rsync cp production example.com shared/uploads/ /Users/me/Desktop/example.com/uploads/
==> rsync -aP --numeric-ids -e "ssh -T -o Compression=no -x -p 22" '[email protected]:/srv/www/example.com/shared/uploads/' /Users/me/Desktop/example.com/uploads/

trellis rsync cp --user=admin production example.com /absolute/path/ /Users/me/Desktop/example.com/uploads/
==> rsync -aP --numeric-ids -e "ssh -T -o Compression=no -x -p 22" '[email protected]:/absolute/path/' /Users/me/Desktop/example.com/uploads/

# rsync with --delete
trellis rsync sync <env> <site> <source> <destation>

[swalkinshaw]

I wonder if we could get most of these benefits just by using SSH aliases? Example: if we generated aliases like example.com_production this could work:

rsync somefile example.com_production:somedir

[swalkinshaw]

@tangrufus related roots/trellis#1259

I'm liking the idea of supporting this more, but ideally we could get remote <-> remote working easily too 🤔

# copy from local to remote
trellis copy production source/uploads {site}/shared/uploads

# copy from remote to remote
trellis copy production admin@old_host:{site}/shared/uploads {site}/shared/uploads

# sync from remote to remote
trellis copy --sync production admin@old_host:{site}/shared/uploads {site}/shared/uploads

# maybe we could omit the last path as well and it would use the same dest as the source path?
trellis copy --sync production admin@old_host:{site}/shared/uploads

I'd prefer just one top-level command with the best default. So here the default is rsync's cp behaviour and you'd use an explicit flag like --sync if you wanted that behaviour.

{site} is an idea to support a couple templating variables that trellis-cli would replace. Maybe just site, web_root, home, nginx, etc. edit: although this is brittle since those paths are customizable in Trellis 🤔 we might have to run playbooks to get the values

And also shown here is an idea to easily support remote <-> remote. We'd detect if the pattern <user>@<host>:path were used and do the SSH forwarding transparently.

What do you think?

[MikeiLL]

Love it!

# copy from local to remote
trellis copy production source/uploads {site}/shared/uploads

I'd prefer just one top-level command with the best default. So here the default is rsync's cp behaviour and you'd use an explicit flag like --sync if you wanted that behaviour.

I'm not clear what --sync would do, and in what look like 100 options not getting clarification. I generally just use -avP (archive, which preserves most things, verbose and progress.) Maybe we could have a flags option for advanced requirements.

{site} is an idea to support a couple templating variables that trellis-cli would replace. Maybe just site, web_root, home, nginx, etc. edit: although this is brittle since those paths are customizable in Trellis 🤔 we might have to run playbooks to get the values

I'm not understanding what those would be for.

And also shown here is an idea to easily support remote <-> remote. We'd detect if the pattern <user>@<host>:path were used and do the SSH forwarding transparently.

Would users need to have .ssh/config entries or could the main parameter(s) contain full ssh strings (like with port added) followed by colon and path?

maybe we could omit the last path as well and it would use the same dest as the source path?
trellis copy --sync production admin@old_host:{site}/public_html/idiotic_path/wp-content/uploads

Yes, since the new_host path will usually be /srv/www/{site}/shared/uploads.

Exciting.

[swalkinshaw]

I'm not clear what --sync would do, and in what look like 100 options not getting clarification. I generally just use -avP (archive, which preserves most things, verbose and progress.) Maybe we could have a flags option for advanced requirements.

--sync would also delete things if they existed on the destination vs a straight copy. But yeah, we could look into just passing through rsync options directly.

I'm not understanding what those would be for.

Just so people wouldn't have to type out /srv/www/example.com/ ({site} would be the equivalent) manually for example; it's a nice to have.

Would users need to have .ssh/config entries or could the main parameter(s) contain full ssh strings (like with port added) followed by colon and path?

Not 100% sure yet, but full ssh strings should work I'd think.

[tangrufus]

The --sync idea looks good. However, trellis copy isn't explicit enough for the direction. I suggest:

trellis rsync push [--sync] <env> <site> <source> <destation>
trellis rsync pull [--sync] <env> <site> <source> <destation>

To tackle the templating variables problem, I suggest we do not run rsync directly, but use ansible's synchronize module:

# copy from local to remote
# trellis rsync push production example source/uploads {site}/shared/uploads
- name: Synchronization of src on the control machine to dest on the remote hosts
  ansible.posix.synchronize:
    src: source/uploads
    dest: "{{site}}/shared/uploads"

# copy from remote to local
# trellis rsync pull production example {site}/shared/uploads source/uploads 
- name: Synchronization of src on the inventory host to the dest on the localhost in pull mode
  ansible.posix.synchronize:
    mode: pull
    src: "{{site}}/shared/uploads"
    dest: source/uploads 

---

But, 2 issues:

1. in contrast to other commands, <site> argument becomes required
2. remote to remote rsync is not supported by ansible's synchronize module (possible by using a complicated playbook, but I think that shouldn't become part of trellis-cli core)

[MikeiLL]

trellis rsync push [--sync] <env> <site> <source> <destination>
trellis rsync pull [--sync] <env> <site> <source> <destination>

I may be missing the obvious, but if we are talking about sharing data between B and C from A, when would it be necessary to push from B rather than just pull to C?

There would need to be, in any case:

• public ssh key on server A that matches local
• public ssh key on server B that matches local
• correct path to source data
• correct path to destination data
• username, port, address for server A
• username, port, address for server B

Right?

I'm thinking about use cases:

• Migrating from a non-bedrock site to new trellis-managed server
• Copying between staging, production, development servers of same
• Syncing between staging, production, development servers of same
• Migrating from one bedrock server to another with same and , different IP
• Migrating from one to another (example.com to another.com)

I wonder if Trellis-cli can also handle migrating databases and running search-replace, but that's obviously out of the scope of this discussion. We generally WP-Migrate-DB-Pro these days.

[swalkinshaw]

The --sync idea looks good. However, trellis copy isn't explicit enough for the direction. I suggest:

🤔 It's always just copy FROM TO (or copy SOURCE DEST). I think push/pull and source/dest could be more confusing? I personally find push/pull more confusing when you mix in remote. It's simple when you're just doing local<->remote, but otherwise confusing?

To tackle the templating variables problem, I suggest we do not run rsync directly, but use ansible's synchronize module:

Interesting idea. Why does this make site required? We just default to the first site if none is specified, couldn't we do that here as well? The lack of remote<->remote is more of a problem.

[HXavS]

@MikeiLL

I wonder if Trellis-cli can also handle migrating databases and running search-replace, but that's obviously out of the scope of this discussion. We generally WP-Migrate-DB-Pro these days.

I am pretty sure the Trellis-cli can do it pretty easily. You can use the mysqldump in /usr/bin or just use wp db export on the vagrant host to create a copy of the the database. Backup the database being overwritten. use wp cli to do a database import and then wp search-replace $remote_url $local_url to correct urls.

Here is a handy script I found a while back that does just that. https://github.com/dale42/wp-site-sync/blob/master/wp-site-sync.sh
Only requires SSH and WP-CLI. It also uses rsync.