Key_Shared subscription with autoscaling enabled

[praveengopalan]

In my design, I am using a Key_Shared subscription with autoscaling enabled. However, when new pods are created during scaling, I notice that messages start getting stuck in the topic and the backlog count keeps increasing.

[lhotari]

Which Pulsar version are you using? In Pulsar 4.0+, PIP-379 minimizes the problem that you are describing. The PIP-379 implementation is enabled by default. (In some server configurations, it might be disabled, so you should ensure that subscriptionKeySharedUseClassicPersistentImplementation is set to use the default value false).

PIP-379 related documentation: https://pulsar.apache.org/docs/4.1.x/concepts-messaging/#preserving-order-of-message-delivery-by-key

Please check "Troubleshooting issues when message delivery is blocked for a key in Key_Shared subscriptions AUTO_SPLIT mode" for more details for your case.

PIP-379 announcement in Pulsar 4.0 blog post: https://pulsar.apache.org/blog/2024/10/24/announcing-apache-pulsar-4-0/#enhanced-key_shared-subscription-scale-without-compromising-message-order