Built-in restore functionality

[lawndoc]

Currently, stack-back only provides backup functionality, and restores have to happen manually with the restic CLI. We will be adding the functionality to restore backups to stack-back in a future release.

This discussion is for external users and collaborators to provide input on what the restore functionality should look like in stack-back.

[faekiva]

Potentially it could just store a config file on disk, representing the current snapshot versions as seen in rcb snapshots, like

version: 0.0.1

restore_all: false
items:
  - path: /volumes
    id: 19928e1c
    restore: false
  - path: /databases/mysql/all_databases.sql
    id: 7a642f37
    restore: false

The container could update the ids automatically as the backups occurred. On container startup, if any of the individual restore keys or restore-all is set to true, the appropriate path(s) are restored, the keys setting to false as they're restored.

[lawndoc]

This is a bit more complex than I was expecting. I worry that keeping a config file in sync with the current state would introduce risk for something to go wrong. Permission issues could prevent the file from being written to, and you'd also have the recursive problem where the config file to restore the backup would also need to be backed up.

What do you think about just adding a single command like rcb restore abcd1234 that stops all containers, restores the specified snapshot, and starts the containers back up?

[faekiva]

I think it makes sense to have both available, shouldn't be too much extra effort to do the rcb command once the config version is done, particularly since we can pretty much just expose the restic restore command since it's all in one snapshot. The config format changes work for me, it'd be a "nice to have" to allow adding path specific restores via the flat file but since that could pretty easily be added in a non-breaking way I'm happy to put that as a potential future enhancement to keep the code size small.

In terms of the config, when I'm looking at the code for the application I'm not seeing any dependencies for the application itself at first glance, just the requirements.txt for the test. Since we'd be allowing users to modify the config, I think it'd be worth having a robust YAML library to do (de)serialization if we go for YAML as the config language. I'd personally pick PyYAML since it seems like the defacto standard, but I'm open to other thoughts.

We could also do it with JSON instead of YAML to avoid the library, but I think YAML's nicer, and theoretically the ability to use anchors and things could be useful for users down the line.

If we end up going the dependency route, do you have a preference as to how they're managed? I have a weak preference for using something like poetry or uv instead of managing venvs manually, but ultimately it's your call.

[faekiva]

I discovered requests and docker libraries in use and now I'm extra wondering how dependencies are being managed :)

[lawndoc]

I like your idea of eventually having the option to restore an individual volume/database. I'm actually new to restic, so I wasn't sure yet if there was a way to pick specific files or directories out of a snapshot. Either way, I do think we should focus on just having full snapshot restore functionality for now, since that will be the most common and stable way to restore a compose project. Let's add more granular restore capability in a future update.

I'm fine with adding PyYAML as a dependency, I like YAML too.

As far as where the Python dependencies are currently defined, docker is specified in setup.py, which actually pulls in requests as a dependency when restic-compose-backup is packaged at build time. If you have a compelling argument for changing how dependencies are managed, I'll hear you out. Otherwise I kind of feel like "it ain't broke so don't fix it" :)

[faekiva]

As far as where the Python dependencies are currently defined, docker is specified in setup.py, which actually pulls in requests as a dependency when restic-compose-backup is packaged at build time.

That makes sense! It might be worth adding requests as a dependency in setup.py anyways, just to protect from the application relying on a docker package implementation detail (ie: using requests) function.

If you have a compelling argument for changing how dependencies are managed, I'll hear you out. Otherwise I kind of feel like "it ain't broke so don't fix it" :)

Honestly it's largely a developer experience thing for me. For VS Code intellisense to work properly I needed to make a virtual env that had all of the dependencies in it, I think technically I could do it with setup.py but I wasn't able to find a great guide for doing it, since I don't want to have the actual cli installed in the venv, just the dependencies. I'm also personally running mypy to make sure I don't do anything stupid with a type mismatch, which means I have dev dependencies of types-docker and types-requests to make sure that I can easily track what the data that's being returned looks like.

It's also nice using uv specifically that for the virtual env I don't need to worry about installing the right python version that the application uses with a separate application, since it integrates with a .python-version file in the repo. (it'd also likely be worth pinning the python version even if we don't use that route, right now the Dockerfile is just using the latest version which means the code might break randomly).

It's also broadly recommended to use a pyproject.toml by python and using something like uv or poetry helps manage it.

Because of all of that, I have do have the relevant files for it in a basic version of it set up on my fork right now for my own use regardless, and I think it'd be useful to have it upstream and the build system updated to use the same tooling. That being said, I can definitely just rebase my changes after I make them onto a branch that doesn't have those things on it so it can be merged upstream, so it's not a huge deal if you decide to keep things as is! I know that my workflow is not everyone's workflow :)

[lawndoc]

to protect from the application relying on a docker package implementation detail (ie: using requests) function

I think you make a good point here. I'm fine with adding requests as a dependency explicitly.

It's also broadly recommended to use a pyproject.toml by python

This is already something I was considering updating, so this change is fine with me too. I also think pinning the Python version is a good idea.

For dependency management, I've tended to stick to the standard Python tooling (pip + venv). I think you're a more experience Python dev than I am, though, so if you think it's worth tweaking the repo to work better with mypy and uv then I trust you.

Long term, I'd like to have dependency management automated with GitHub Actions or Dependabot (with integration tests). As long as there isn't anything added that is going to make that challenging, I don't have a problem with it.

[lawndoc]

Just wanted to call out a few options in the restic restore command that might be relevant for the rcb restore functionality as defaults or passthrough arguments.

--overwrite if-changed will only restore files that haven't changed (based on file size + modification timestamp). --overwrite always is default in restic, but that seems a bit wasteful to me. It's probably kept to always for stability reasons, but IMO if-changed seems safe enough. Either way, we could pass through this argument from the rcb restore wrapper to restic restore to let the user choose this behavior.

--delete will remove files from the target directory that do not exist in the snapshot during restore. The default behavior in restic is to not delete extra files.

--dry-run --verbose=2 shows which files would be restored or deleted when using --overwrite if-changed or --delete

[lawndoc]

@faekiva I just looked at your fork and it looks like you already handle passthrough args. Disregard this message :)

[MissMusic]

I'm looking at a simple way to restore full docker stacks on a new host in a DR scenario, if you already manage your compose files on a central place like with komodo and Git, or any similar tool, support to trigger the basic restic restore through rcb where stack-back do all the normal backup-steps, but restores the volumes from latest snapshot instead of backing up (stop containers with the stop flag, restore volumes, re-start containers) it would be so nice.
Is there any plans or updates on the restore feature?

[lawndoc]

Hi @MissMusic I'm a bit strapped for time right now in my personal life being a new father and studying for the CISSP certification. I'm prioritizing bug fixes and ecosystem stability at the moment. I plan to have some time freed up in March when I hope to be done with my certification exam. I'll be able to take a closer look at this around that time.

In the meantime, feel free to contribute any user stories, diagrams, or code samples related to this feature. Thanks!

[lawndoc]

Off the top of my head, the tricky part of what you described is that restic has no concept of a mapping between a volume or database in the restic respository and the container it came from.

Backup

flowchart LR
  subgraph Compose Project
    subgraph Container1
      volume1
      volume2
    end
    subgraph Database1
      mysql.dump
    end
  end
  subgraph Restic Repository
    v1b[volume1 backup]
    v2b[volume2 backup]
    db[mysql.dump backup]
end
volume1 --> v1b
volume2 --> v2b
mysql.dump --> db

Loading

Restore

flowchart LR
  subgraph project[Compose Project]
    subgraph Container1
      volume1
      volume2
    end
    subgraph Database1
      mysql.dump
    end
  end
  subgraph Restic Repository
    v1b[volume1 backup]
    v2b[volume2 backup]
    db[mysql.dump backup]
end
v1b --???-->project
v2b--???-->project
db--???-->project

Loading

We could have stack-back create that mapping and store it in the restic repository, but there's some additional pieces to consider on how to handle inconsistencies.

[MissMusic]

Since the docker stack already contain enough data in service-tags for stack-back to select and back up, I don't really get why one can't re-use that same logic for an in-place restore. Restoring a database is more complex as the location for where the database was found and the dump backup taken is less straight forward.

For many containers tile based backups are enough, could the restore-in-place feature be scope limited to non-database setups? At least in a DR scenario that would free up quite some time to be able to focus on manually restoring the databases.

If stack-back is relying on docker container/volume GUIDs to ID the data in the repository I can understand the issue to restore if the stack is re-deployed with newly generated IDs. Switching to a service+volume name combo in the repository would solve that but include new risks like duplicate names. A workaround could be an optional manual stack-back ID provided in an env in the docker config. The user opting in for that would have to make sure it is unique for the source repository, it could still be combined with service+volume name combo to minimize risks.

For the moment I personally switched to Backrest, As a DR I can deploy that in a stack together with a base config file with my repository info and a backup job for Backrests own config pre-configured (I never need to update this file, just keep it safe for re-deploy). After re-deploy I can in-place restore the latest Backrest config. It gets all later configured backup jobs back, and then I can in-place restore all of those in turn to get all the data for all the stacks on a host back in place matching my compose files.

Far from as clean or hands-off as it could be to just re-deploy a stack with stack-back and a restore-command, but I don't need to do DR too often, and I'll just need to hope the restore works the day I need it to.

Except the in-place restore, stack-back had all the features I really needed, neatly packaged with all it's configuration contained within the docker-stack config, something I really liked.

Best of luck with the both the CISSP exam and the kid!

[lawndoc]

Since the docker stack already contain enough data in service-tags for stack-back to select and back up, I don't really get why one can't re-use that same logic for an in-place restore.

It's because the docker stack configuration may change between backups, creating an inconsistency between what is in the stack config and what is in the restic repo. Restoring from the restic repo in that state may cause a volume or database that was previously backed up to become orphaned because it's no longer in the current config and now doesn't have a place to restore to.

I gave it some thought and I think we might be able to calculate an expected config from the volume + database paths in a snapshot and then verify that the current stack's config matches what is coming from the restic repo. If the config doesn't match, a restore won't happen and the diff could be logged for clarification on what needs to be fixed.

I really want to make sure that the restore behavior is consistent, intuitive, and fool-proof... The last thing you want when you're restoring a backup is a "gotcha".

[MissMusic]

That makes sense. I have many stacks with just a few services in each, so from my point of view stacks are quite static. But I guess there's nothing stopping you putting everything you run in to one stack, and if you do it like that, stack changes would become common.