Replication broken after upgrade from 2.11.2 to 3.0.2

[jmhublar]

Greetings,

I'm hoping the community can set me on the right track here. . .

We upgraded our Pulsar clusters to version 3.0.2 and have been encountering issues with namespace-scoped replication ever since. The replication between clusters (cluster_a and cluster_b ) has started failing, and we noticed this shortly after the upgrade. We continue to observe this with 3.0.7

Problem Details:

It seems like the brokers in our cluster_a cluster aren't passing the OAuth2 parameters required to authenticate with the other instance in cluster_b . Upon closer inspection, we noticed that the cluster configuration in cluster_a lacks the necessary authentication parameters:

{
  "serviceUrl": "http://pulsar-proxy.apps-01.sbx.cluster_b.example.com:8080",
  "serviceUrlTls": "https://pulsar-proxy.apps-01.sbx.cluster_b.example.com:8443",
  "brokerServiceUrl": "pulsar://pulsar-proxy.apps-01.sbx.cluster_b.example.com:6650",
  "brokerServiceUrlTls": "pulsar+ssl://pulsar-proxy.apps-01.sbx.cluster_b.example.com:6651",
  "authenticationPlugin": "",
  "authenticationParameters": "",
  ...
}

The errors we’re seeing on the cluster_b cluster are like this:

WARN  org.apache.pulsar.proxy.server.ProxyConnection - [/10.191.231.31:14887] Unable to authenticate
javax.naming.AuthenticationException: No anonymous role, and no authentication provider configured

To address this, we tried updating the cluster configuration for cluster_a to include OAuth2 parameters, hoping that it would resolve the issue. It is worth noting that cluster_a and cluster_b use the same auth endpoint and tokens. Below is a representation of the updated configuration:

{
  "serviceUrl": "http://pulsar-proxy.apps-01.sbx.cluster_b.example.com:8080",
  "serviceUrlTls": "https://pulsar-proxy.apps-01.sbx.cluster_b.example.com:8443",
  "brokerServiceUrl": "pulsar://pulsar-proxy.apps-01.sbx.cluster_b.example.com:6650",
  "brokerServiceUrlTls": "pulsar+ssl://pulsar-proxy.apps-01.sbx.cluster_b.example.com:6651",
  "authenticationPlugin": "org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2",
  "authenticationParameters": "{\"issuerUrl\": \"https://oauth.example.com/[REDACTED]\",\"privateKey\": \"file:///pulsar/privatekey/proxy-oauth.json\",\"audience\": \"pulsar\"}",
  "brokerClientTlsEnabled": true,
  "tlsAllowInsecureConnection": true,
  ...
}

Despite this, we’re still seeing the same authentication errors when attempting replication.
Questions:

• Does this configuration seem correct? We're essentially trying to mirror the broker's internal settings for OAuth2.
• If we make changes to the cluster configuration (like updating OAuth2 parameters), do we need to reinitialize or reset the replication configuration on the cluster for those changes to take effect?

Any insights or advice from those who've encountered similar issues would be greatly appreciated. Thanks in advance for your help!

[david-streamlio]

From what I can tell, it looks like you are using Pulsar proxies in front of your brokers. The WARNING message indicates that you have NOT configured the authentication parameters on the proxies via the conf/proxy.conf file.

See the documentation for more details.