Add exception handling to task running

[alexgleith]

Not sure if this is naive, but I have some tasks failing, and Statistician currently crashes, rather than marking the task as an error and continuing.

I think this is the right place to catch exceptions and gracefully return an error notification.

[codecov]

Codecov Report

Merging #34 (7643227) into develop (9ee6375) will decrease coverage by 0.12%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           develop      #34      +/-   ##
===========================================
- Coverage    62.01%   61.88%   -0.13%     
===========================================
  Files           20       20              
  Lines         1935     1939       +4     
===========================================
  Hits          1200     1200              
- Misses         735      739       +4     

Impacted Files	Coverage Δ
odc/stats/proc.py	26.00% <0.00%> (-0.54%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9ee6375...7643227. Read the comment docs.

[supermarkion]

Should be worth to have a try. Better than no try-catch.

[emmaai]

I have my doubt on this "catching everything then move on" approach. And I'm not sure about the context, if it's the only resort to fix the problem, then yes, if we could do more work to avoid crash, then we should.

[alexgleith]

I have my doubt on this "catching everything then move on" approach. And I'm not sure about the context, if it's the only resort to fix the problem, then yes, if we could do more work to avoid crash, then we should.

I agree that crashes shouldn't happen, but if they do, we have a choice of how we handle them. Currently, it's basically by aborting the whole job. What should happen is that Task should fail, be logged and the next Task can be tried.

This is especially important for streaming from SQS, which has built-in retries and failed task logging using the deadletter queue.