DockerSwarmOperator high load when running

[pklejch]

Hello everyone,

we are using DockerSwarmOperator without enable_logging, some of our tasks run for hours and following part makes this operator very inefficient.

while True:
    if self._has_service_terminated():
        self.log.info("Service status before exiting: %s", self._service_status())
        break

This operator constantly asks Docker API whether the service has ended, if you run a lot of Docker services this makes a huge difference.

I think that simple sleep (with configurable sleep period and reasonable default value) in the while loop would make a ton of difference. I think that sleeping for few seconds between checks is a good compromise and if you run a job that take 3 hours to complete you don't need to detect end of the job right away.

What do you think ? I am willing to make a PR.

[pankajastro]

I feel it makes sense to have a configurable sleep period here.