Polling error #12
Description
After running for a couple of hours in daemonized mode, we stopped receiving data. Running in debug mode we can see it continues polling again and again the same query. It's been in this cycle for more than two hours. This is part of the log:
2017-01-30 14:17:50,108 - INFO - requests.packages.urllib3.connectionpool - MainThread - Starting new HTTPS connection (1): rest.logentries.com
2017-01-30 14:17:51,000 - DEBUG - requests.packages.urllib3.connectionpool - MainThread - "GET /query/SOME_KEY:1:SOME_OTHER_KEY?log_keys=ANOTHER_KEY&query=where(...)+calculate(percentile(95):RT) HTTP/1.1" 200 412
2017-01-30 14:17:51,018 - DEBUG - leexportpy.request_helper - MainThread - links attribute found in a subsequent continue request. Should continue polling.
2017-01-30 14:17:52,020 - DEBUG - leexportpy.request_helper - MainThread - Continuing getting response...
2017-01-30 14:17:52,022 - DEBUG - leexportpy.request_helper - MainThread - Making get request to the url: https://rest.logentries.com/query/SOME_KEY:1:SOME_OTHER_KEY?log_keys=ANOTHER_KEY&query=where(...)+calculate(percentile(95):RT)
2017-01-30 14:17:52,025 - INFO - requests.packages.urllib3.connectionpool - MainThread - Starting new HTTPS connection (1): rest.logentries.com
I checked the code and this happens because in request_helper.get_continuity_final_response() the response has the word 'links' on it, so it tries again but it does not have a limit on retrying. For some reason the LogEntries query continues to receive responses with word 'links', so it falls into an apparently infinite cycle.
I added a count limit for the retries to ignore this query after some failed tries. With this improvement we now have little gaps every now and then (say 3-8 gaps every hour in total for the 18 metrics currently configured).
This has happened many times after running for some hours (it works well for a while), whether it's daemonized or not. I'm running on Ubuntu 16.10, Python 2.7.12 and leexportpy 0.1.8.
Thank You.