Distributor queue full errors

[brajeshpalod]

Hey Team,

We are running a microservice mode deployment for tempo and are getting a lot of errors for queue full for the distributor tempo_distributor_queue_pushes_failures_total and gone through the below discussion #3053. I have tried to override the distributor queue size with metrics_generator_forwarder_queue_size. However, Increasing the queue size parameters has not increased the queue in distributors. I have checked via metrics tempo_distributor_queue_length which is still getting on 100 as max for queue even when I have configured for 500 in metrics_generator_forwarder_queue_size as override.

Could you please suggest me how to override the distributor push queue from default value?