[k8s] move filtering to stream time when calculating allocated GPU quantity by each node

[SeungjinYang]

The two users of get_all_pods_in_kubernetes_cluster does the same postprocessing of the pods

• filter out pods that are not in PENDING or RUNNING
• filter out pods according to should_exclude_pod_from_gpu_allocation
  and then postprocesses the pods calculate a collections.defaultdict(int).

By moving the filtering and postprocessing logic into the stream processing, we can reduce the memory needs of executing this call especially in larger k8s contexts with a lot of pods.

Tested (run the relevant ones):

• Code formatting: install pre-commit (auto-check on commit) or bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
• Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
• Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

[kevinmingtarja]

/quicktest-core --kubernetes
/smoke-test --kubernetes

[kevinmingtarja]

LGTM overall, thanks @SeungjinYang! Just one question regarding the allocated_qty calculation.