Bulk Scale test failed with NoHttpResponse with multiple replicas

[chandrams]

Describe the bug
Bulk Scale test failed with multiple replicas with the below issue

org.apache.http.NoHttpResponseException: kruize.openshift-tuning.svc.cluster.local:8080 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)

How to reproduce it

• Clone this PR and run the scalability test after replacing the server (where thanos setup) is present in the below command:

./bulk_scale_test.sh -i quay.io/vinakuma/autotune_operator:jobsave3  -w 3 -d 1 -r /tmp/results/bulk_scale_test_jobsave3 -a 3 -o 1 -c 2 -s "2025-03-10T02:00:00.000Z" --url="http://thanos-query-frontend-thanos-bench.apps.<server>/"

Expected behavior
No exceptions/errors should be seen and recommendations should be generated

Relevant logs

2025-03-0723:45:06.854 ERROR [qtp1789268516-62][BulkService.java(212)]-Not able to fetch recommedations from database due to Cannot invoke "java.util.Map.forEach(java.util.function.BiConsumer)" because the return value of "com.autotune.analyzer.serviceObjects.BulkJobStatus.getExperimentMap()" is null
2025-03-0723:45:07.343 ERROR [qtp1789268516-65][BulkService.java(212)]-Not able to fetch recommedations from database due to Cannot invoke "java.util.Map.forEach(java.util.function.BiConsumer)" because the return value of "com.autotune.analyzer.serviceObjects.BulkJobStatus.getExperimentMap()" is null
2025-03-0723:45:20.437 ERROR [qtp1789268516-58][BulkService.java(212)]-Not able to fetch recommedations from database due to Cannot invoke "java.util.Map.forEach(java.util.function.BiConsumer)" because the return value of "com.autotune.analyzer.serviceObjects.BulkJobStatus.getExperimentMap()" is null
2025-03-0723:45:44.307 ERROR [qtp1789268516-57][BulkService.java(212)]-Not able to fetch recommedations from database due to Cannot invoke "java.util.Map.forEach(java.util.function.BiConsumer)" because the return value of "com.autotune.analyzer.serviceObjects.BulkJobStatus.getExperimentMap()" is null
2025-03-0723:46:26.954 INFO [pool-10-thread-1][DataSourceMetadataOperator.java(301)]-filterBuilder: namespace!=''
2025-03-0723:46:26.955 INFO [pool-10-thread-1][DataSourceMetadataOperator.java(301)]-filterBuilder: workload!=''
2025-03-0723:46:26.955 INFO [pool-10-thread-1][DataSourceMetadataOperator.java(301)]-filterBuilder: container!=''
2025-03-0723:46:26.955 INFO [pool-10-thread-1][DataSourceMetadataOperator.java(215)]-namespaceQuery: sum by (namespace) ( avg_over_time(kube_namespace_status_phase{namespace!='' ,org_id="org-1",cluster_id="eu-1-1"}[15d]))
2025-03-0723:46:26.955 INFO [pool-10-thread-1][DataSourceMetadataOperator.java(216)]-workloadQuery: sum by (namespace, workload, workload_type) ( avg_over_time(namespace_workload_pod:kube_pod_owner:relabel{workload!='' ,org_id="org-1",cluster_id="eu-1-1"}[15d]))
2025-03-0723:46:26.955 INFO [pool-10-thread-1][DataSourceMetadataOperator.java(217)]-containerQuery: sum by (container, image, workload, workload_type, namespace) (  avg_over_time(kube_pod_container_info{container!='' ,org_id="org-1",cluster_id="eu-1-1"}[15d]) *  on (pod, namespace,prometheus_replica) group_left(workload, workload_type)   avg_over_time(namespace_workload_pod:kube_pod_owner:relabel{workload!~"" ,org_id="org-1",cluster_id="eu-1-1"}[15d]))
2025-03-0723:46:28.198 ERROR [qtp1789268516-64][BulkService.java(212)]-Not able to fetch recommedations from database due to Cannot invoke "java.util.Map.forEach(java.util.function.BiConsumer)" because the return value of "com.autotune.analyzer.serviceObjects.BulkJobStatus.getExperimentMap()" is null
2025-03-0723:46:29.744 ERROR [qtp1789268516-59][BulkService.java(212)]-Not able to fetch recommedations from database due to Cannot invoke "java.util.Map.forEach(java.util.function.BiConsumer)" because the return value of "com.autotune.analyzer.serviceObjects.BulkJobStatus.getExperimentMap()" is null
2025-03-0723:46:36.521 ERROR [qtp1789268516-61][BulkService.java(212)]-Not able to fetch recommedations from database due to Cannot invoke "java.util.Map.forEach(java.util.function.BiConsumer)" because the return value of "com.autotune.analyzer.serviceObjects.BulkJobStatus.getExperimentMap()" is null
2025-03-0723:46:51.417 ERROR [qtp1789268516-61][RecommendationEngine.java(1941)]-MetricProfile does not exist or is not valid: resource-optimization-local-monitoring
2025-03-0723:46:51.466 ERROR [qtp1789268516-61][RecommendationEngine.java(1941)]-MetricProfile does not exist or is not valid: resource-optimization-local-monitoring
2025-03-0723:46:51.467 ERROR [qtp1789268516-57][RecommendationEngine.java(1941)]-MetricProfile does not exist or is not valid: resource-optimization-local-monitoring
2025-03-0723:46:51.471 ERROR [qtp1789268516-64][RecommendationEngine.java(1941)]-MetricProfile does not exist or is not valid: resource-optimization-local-monitoring
2025-03-0723:47:12.999 INFO [pool-10-thread-3][DataSourceMetadataOperator.java(301)]-filterBuilder: namespace!=''
2025-03-0723:47:12.999 INFO [pool-10-thread-2][DataSourceMetadataOperator.java(301)]-filterBuilder: namespace!=''
2025-03-0723:47:12.999 INFO [pool-10-thread-2][DataSourceMetadataOperator.java(301)]-filterBuilder: workload!=''
2025-03-0723:47:12.999 INFO [pool-10-thread-3][DataSourceMetadataOperator.java(301)]-filterBuilder: workload!=''
2025-03-0723:47:12.999 INFO [pool-10-thread-2][DataSourceMetadataOperator.java(301)]-filterBuilder: container!=''
2025-03-0723:47:12.999 INFO [pool-10-thread-3][DataSourceMetadataOperator.java(301)]-filterBuilder: container!=''
2025-03-0723:47:12.999 INFO [pool-10-thread-3][DataSourceMetadataOperator.java(215)]-namespaceQuery: sum by (namespace) ( avg_over_time(kube_namespace_status_phase{namespace!='' ,org_id="org-1",cluster_id="eu-1-1"}[15d]))
2025-03-0723:47:12.999 INFO [pool-10-thread-2][DataSourceMetadataOperator.java(215)]-namespaceQuery: sum by (namespace) ( avg_over_time(kube_namespace_status_phase{namespace!='' ,org_id="org-1",cluster_id="eu-1-2"}[15d]))
2025-03-0723:47:13.000 INFO [pool-10-thread-3][DataSourceMetadataOperator.java(216)]-workloadQuery: sum by (namespace, workload, workload_type) ( avg_over_time(namespace_workload_pod:kube_pod_owner:relabel{workload!='' ,org_id="org-1",cluster_id="eu-1-1"}[15d]))
2025-03-0723:47:13.000 INFO [pool-10-thread-2][DataSourceMetadataOperator.java(216)]-workloadQuery: sum by (namespace, workload, workload_type) ( avg_over_time(namespace_workload_pod:kube_pod_owner:relabel{workload!='' ,org_id="org-1",cluster_id="eu-1-2"}[15d]))
2025-03-0723:47:13.000 INFO [pool-10-thread-3][DataSourceMetadataOperator.java(217)]-containerQuery: sum by (container, image, workload, workload_type, namespace) (  avg_over_time(kube_pod_container_info{container!='' ,org_id="org-1",cluster_id="eu-1-1"}[15d]) *  on (pod, namespace,prometheus_replica) group_left(workload, workload_type)   avg_over_time(namespace_workload_pod:kube_pod_owner:relabel{workload!~"" ,org_id="org-1",cluster_id="eu-1-1"}[15d]))
2025-03-0723:47:13.000 INFO [pool-10-thread-2][DataSourceMetadataOperator.java(217)]-containerQuery: sum by (container, image, workload, workload_type, namespace) (  avg_over_time(kube_pod_container_info{container!='' ,org_id="org-1",cluster_id="eu-1-2"}[15d]) *  on (pod, namespace,prometheus_replica) group_left(workload, workload_type)   avg_over_time(namespace_workload_pod:kube_pod_owner:relabel{workload!~"" ,org_id="org-1",cluster_id="eu-1-2"}[15d]))
2025-03-0723:47:14.226 ERROR [qtp1789268516-60][BulkService.java(212)]-Not able to fetch recommedations from database due to Cannot invoke "java.util.Map.forEach(java.util.function.BiConsumer)" because the return value of "com.autotune.analyzer.serviceObjects.BulkJobStatus.getExperimentMap()" is null
2025-03-0723:47:15.433 ERROR [qtp1789268516-61][BulkService.java(212)]-Not able to fetch recommedations from database due to Cannot invoke "java.util.Map.forEach(java.util.function.BiConsumer)" because the return value of "com.autotune.analyzer.serviceObjects.BulkJobStatus.getExperimentMap()" is null
org.apache.http.NoHttpResponseException: kruize.openshift-tuning.svc.cluster.local:8080 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)

Environment:

• Kubernetes Cluster openshift