Refactor Snapshot Service

[LesnyRumcajs]

I'll be honest, the setup for snapshot service is looking pretty confusing, a mix of ruby, bash and bash embedded into ruby. Perhaps we should improve upon this code? My eyes bleed reading this, it's not comprehensive at all. And we are talking about a really small amount of code.

If we want to keep using Ruby/Python or whatever else scripting language here - I think we should reconsider how bash is wrapped, and come up with somewhat clearer abstractions, e.g. an Object that has a bunch of methods that then do some bash stuff under the hood. It would be much nicer than interpolating a bunch of commands into a single string.

This stuff is looking pretty unreadable:

add_timestamps_cmd = <<~CMD
  awk '{
  if ($0 !~ /^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{6}Z/)
    print strftime("[%Y-%m-%d %H:%M:%S]"), $0;
  else
    print $0;
  fflush();
  }'
CMD

I appreciate the awk knowledge, but I'd prefer to have a readable alternative. If it's not easier and more comprehensive to do it in Ruby - at least some comments to avoid having to decipher this would be much appreciated.

All of that said, if it's too much effort to invest - it would at least be nice to have this code properly documented.

Originally posted by @ruseinov in #403 (comment)

[LesnyRumcajs]

@ruseinov Feel free to consult the new design (+ priority of implementing it) with the author of the service.

My two cents - this service worked well enough, regardless of its beauty, and is, to my understanding, going to be deprecated in favour of what Chainsafe Infrastructure is doing (@lemmih please correct me if I'm wrong) and so there's no point in any major refactoring.

[LesnyRumcajs]

Forest snapshot service discontinued in favour of the infra snapshot service.