Provide flexible command timeouts

[christophermaier]

Commands that are expected to take a long time to run (e.g., synchronous creation of infrastructure) can fail with Cog simply because they exceed the built-in timeout for a command.

Currently, users can recompile Cog with a higher command timeout (e.g.

cog/config/config.exs

Line 72 in 731f733

interactive_timeout: {60, :sec},

), but that is a.) inconvenient (it should be exposed as an environment variable at the very least), and b.) too coarse, since that timeout applies to every command, not just the long-lived ones we're interested in here.

Providing a way to alter the timeout for specific commands would be nice. We could expose this in a bundle's config.yaml file, but should also have a way for Cog operators to modify this; command authors might not specify a timeout, or might specify one that is inappropriate for the operator's particular circumstances and use-cases.

(Initially raised in public Cog Slack by Peter Norton.)

[christophermaier]

Another alternative put forth by @kevsmith is to provide a pipeline language construct that allows users to declare a timeout at runtime.

[pcn]

The ability to have user-specified timeouts on the pipeline seems useful, since it lets the user say "I think this whole thing should only take X seconds" however I think it's actually a different problem than allowing a bundle's developer to be able to say that a particular command's expected timeout is expected to be more or less than the default based on their knowledge of the infrastructure being developed.

If the developer knows that some bundle's command should take 5 minutes, the user should be able to test 10 minutes or 1 minute if they know something that the developer doesn't (or if they want to test something on behalf of the developer, perhaps) but requiring a user to have this knowledge when a bundle-related configuration could be set seems like un-necessary work.

[pcarranza]

Please! 👍

If I have to pick, allowing users to define at runtime.

[nmohoric]

I think this is pretty important.
Maybe we consider doing this in multiple phases, the ability to override based on an environment variable should be pretty easy so could be knocked out first.

Bundle and user level timeouts will be much more work (and the latter will take a lot more consideration on the user experience side) so I would prefer to scope those out separately.