Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

skip polling providers still processing #5181

Merged
merged 16 commits into from
Jun 28, 2024
Merged

skip polling providers still processing #5181

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
8106574
skip polling providers still processing
lcouzens Jun 21, 2024
fe63c74
add additional or fo failed update
lcouzens Jun 24, 2024
76ab892
requested changes
lcouzens Jun 25, 2024
ab8fda1
Merge branch 'main' into COST-tweak-polling-batch
lcouzens Jun 25, 2024
48c463a
Merge branch 'main' into COST-tweak-polling-batch
myersCody Jun 26, 2024
1da7a87
make wait timer configurable
lcouzens Jun 27, 2024
c34d9a1
comments
lcouzens Jun 27, 2024
51e18dc
unit tests
lcouzens Jun 27, 2024
73399b1
Merge branch 'main' into COST-tweak-polling-batch
lcouzens Jun 27, 2024
0abfa4f
Update orchestrator.py
lcouzens Jun 27, 2024
7356d7f
unit coverage
lcouzens Jun 27, 2024
fea633c
one more!
lcouzens Jun 27, 2024
86e0b93
fix initial ingest
lcouzens Jun 28, 2024
7998787
unit tests
lcouzens Jun 28, 2024
933e1c6
logging
lcouzens Jun 28, 2024
e6e0bda
Merge branch 'main' into COST-tweak-polling-batch
lcouzens Jun 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ PROMETHEUS_MULTIPROC_DIR='/tmp'
CURRENCY_URL=https://open.er-api.com/v6/latest/USD
UNLEASH_TOKEN='*:*.dbffffc83b1f92eeaf133a7eb878d4c58231acc159b5e1478ce53cfc'
POLLING_TIMER=60 # Set how often you can trigger downloads per provider
PROCESSING_WAIT_TIMER=3 # Set how many days before a long processing provider can be polled again
LARGE_PROCESSING_WAIT_TIMER=7 # Set how many days before a long LARGE customer processing provider can be polled again
MAX_GROUP_BY_OVERRIDE=3 # Set maximum aloud group bys
TAG_ENABLED_LIMIT=200 # Set the max amount of tags per account
DELAYED_TASK_TIME=30 # Set the seconds before a delayed summary task should expire
Expand Down
94 changes: 94 additions & 0 deletions deploy/clowdapp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -492,6 +492,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: KAFKA_CONNECT
Expand Down Expand Up @@ -671,6 +675,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down Expand Up @@ -1150,6 +1158,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down Expand Up @@ -1326,6 +1338,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down Expand Up @@ -1508,6 +1524,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down Expand Up @@ -1690,6 +1710,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down Expand Up @@ -1874,6 +1898,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: TRINO_HOST
Expand Down Expand Up @@ -2060,6 +2088,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: TRINO_HOST
Expand Down Expand Up @@ -2246,6 +2278,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: TRINO_HOST
Expand Down Expand Up @@ -2430,6 +2466,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: TRINO_HOST
Expand Down Expand Up @@ -2614,6 +2654,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: TRINO_HOST
Expand Down Expand Up @@ -2798,6 +2842,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: TRINO_HOST
Expand Down Expand Up @@ -2984,6 +3032,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: DELAYED_TASK_TIME
value: ${DELAYED_TASK_TIME}
- name: DELAYED_TASK_POLLING_MINUTES
Expand Down Expand Up @@ -3174,6 +3226,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: DELAYED_TASK_TIME
value: ${DELAYED_TASK_TIME}
- name: DELAYED_TASK_POLLING_MINUTES
Expand Down Expand Up @@ -3364,6 +3420,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: DELAYED_TASK_TIME
value: ${DELAYED_TASK_TIME}
- name: DELAYED_TASK_POLLING_MINUTES
Expand Down Expand Up @@ -3552,6 +3612,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down Expand Up @@ -3734,6 +3798,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down Expand Up @@ -3916,6 +3984,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down Expand Up @@ -4098,6 +4170,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: DELAYED_TASK_TIME
value: ${DELAYED_TASK_TIME}
- name: DELAYED_TASK_POLLING_MINUTES
Expand Down Expand Up @@ -4284,6 +4360,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: DELAYED_TASK_TIME
value: ${DELAYED_TASK_TIME}
- name: DELAYED_TASK_POLLING_MINUTES
Expand Down Expand Up @@ -4470,6 +4550,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: DELAYED_TASK_TIME
value: ${DELAYED_TASK_TIME}
- name: DELAYED_TASK_POLLING_MINUTES
Expand Down Expand Up @@ -4658,6 +4742,10 @@ objects:
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down Expand Up @@ -5554,6 +5642,12 @@ parameters:
- displayName: Provider count for batch polling
name: POLLING_BATCH_SIZE
value: "100"
- displayName: Timer to prevent triggering tasks while still processing in days
name: PROCESSING_WAIT_TIMER
value: "3"
- displayName: Timer to prevent triggering tasks while still processing in days
name: LARGE_PROCESSING_WAIT_TIMER
value: "7"
- displayName: Enable Tags Limit
name: TAG_ENABLED_LIMIT
value: "200"
Expand Down
6 changes: 6 additions & 0 deletions deploy/kustomize/base/base.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -515,6 +515,12 @@ parameters:
- displayName: Provider count for batch polling
name: POLLING_BATCH_SIZE
value: "100"
- displayName: Timer to prevent triggering tasks while still processing in days
name: PROCESSING_WAIT_TIMER
value: "3"
- displayName: Timer to prevent triggering tasks while still processing in days
name: LARGE_PROCESSING_WAIT_TIMER
value: "7"
- displayName: Enable Tags Limit
name: TAG_ENABLED_LIMIT
value: "200"
Expand Down
4 changes: 4 additions & 0 deletions deploy/kustomize/patches/masu.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,10 @@
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: KAFKA_CONNECT
Expand Down
4 changes: 4 additions & 0 deletions deploy/kustomize/patches/scheduler.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,10 @@
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down
4 changes: 4 additions & 0 deletions deploy/kustomize/patches/worker-celery.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down
4 changes: 4 additions & 0 deletions deploy/kustomize/patches/worker-cost-model-penalty.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down
4 changes: 4 additions & 0 deletions deploy/kustomize/patches/worker-cost-model-xl.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down
4 changes: 4 additions & 0 deletions deploy/kustomize/patches/worker-cost-model.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down
4 changes: 4 additions & 0 deletions deploy/kustomize/patches/worker-download-penalty.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,10 @@
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: TRINO_HOST
Expand Down
4 changes: 4 additions & 0 deletions deploy/kustomize/patches/worker-download-xl.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,10 @@
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: TRINO_HOST
Expand Down
4 changes: 4 additions & 0 deletions deploy/kustomize/patches/worker-download.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,10 @@
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: TRINO_HOST
Expand Down
4 changes: 4 additions & 0 deletions deploy/kustomize/patches/worker-hcs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,10 @@
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TRINO_HOST
value: ${TRINO_HOST}
- name: TRINO_PORT
Expand Down
4 changes: 4 additions & 0 deletions deploy/kustomize/patches/worker-ocp-penalty.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,10 @@
value: ${POLLING_TIMER}
- name: POLLING_BATCH_SIZE
value: ${POLLING_BATCH_SIZE}
- name: PROCESSING_WAIT_TIMER
value: ${PROCESSING_WAIT_TIMER}
- name: LARGE_PROCESSING_WAIT_TIMER
value: ${LARGE_PROCESSING_WAIT_TIMER}
- name: TAG_ENABLED_LIMIT
value: ${TAG_ENABLED_LIMIT}
- name: TRINO_HOST
Expand Down