Azure Functions KafkaTrigger works with Event Hub (Kafka surface) locally, but rarely triggers when deployed

[strongmindsnan]

Description

The problem
I have an Azure Function in Java that I am trying to migrate from an Event Hub trigger to a Kafka trigger. For testing purposes, I create an Event Hub with Kafka surface enabled to test against before deploying to use the real Kafka instance from Confluent.

When I deploy the Event Hub on Azure and run the function locally (mvn azure-functions:run) with my personal account, it consumes messages from a local producer correctly every time. But when the same function is deployed to Azure and assigned a managed identity, it (mostly) ignores messages. It is run by a managed identity that has send, read, and owner rights to the event hub.

Code
The Function:

@FunctionName("KafkaLicensesConsumer")
    public void consumeKafka(
            @KafkaTrigger(
                    name = "KafkaLicensesConsumer",
                    brokerList = "BOOTSTRAPSERVERS",
                    topic = "<TOPICS>_NAME",
                    consumerGroup = "<TOPIC>_CONSUMER_GROUP",
                    dataType = "binary",
                    protocol = BrokerProtocol.SASLSSL,
                    authenticationMode = BrokerAuthenticationMode.PLAIN,
                    username = "SASL_USERNAME",
                    password = "SASL_PASSWORD"
            ) byte[] eventData,
            final ExecutionContext context
    ) {
        //doStuff
        }
    }

Event Hub namespace, hub, and authorization rule in Bicep:

resource eventHub_namespace 'Microsoft.EventHub/namespaces@2021-11-01' = {
  name: eventHub.namespace
  location: location
  sku: {
    name: 'Standard'
    tier: 'Standard'
    capacity: 1
  }
  properties: {
    zoneRedundant: false
    isAutoInflateEnabled: true
    maximumThroughputUnits: 1
    kafkaEnabled: true
  }
}
resource eventHub_namespace_eventHub_licensesTopic 'Microsoft.EventHub/namespaces/eventhubs@2021-11-01' = {
  name: '${eventHub.namespace}/${eventHub.licensesTopic}'
  location: location
  properties: {
    messageRetentionInDays: 7
    partitionCount: 1
    status: 'Active'
  }
  dependsOn: [
    eventHub_namespace
  ]
}

resource eventHub_namespace_eventHub_licensesTopic_eventHub_authorizationRule 'Microsoft.EventHub/namespaces/eventhubs/authorizationRules@2021-11-01' = {
  name: '${eventHub.namespace}/${eventHub.licensesTopic}/${eventHub.authorizationRule}'
  location: location
  properties: {
    rights: [
      'Listen'
      'Send'
      'Manage'
    ]
  }
  dependsOn: [
    eventHub_namespace_eventHub_licensesTopic
  ]
}

Function App in bicep:

resource functionApp 'Microsoft.Web/sites@2021-03-01' = {
  name: functionAppName
  location: location
  kind: 'functionapp,linux'
  tags: resourceTags
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${myFunctionAppMsiResourceId}': {
      }
    }
  }
  properties: {
    httpsOnly: 'true'
    serverFarmId: serverfarm.id
    siteConfig: {
      linuxFxVersion: 'Java|8'
      functionAppScaleLimit: 100
      appSettings: [
        //settings here ]
    }
  }
  dependsOn: [
    storageAccount
  ]
}

Things I have confirmed:

• BrokerList, topic and auth config is identical locally and in Azure.
• The local and deployed functions use different consumer groups to not interfere with each other.
• Setting partition count on the Event Hub to 1 let the deployed function receive 4 messages (the 3rd with 10 minutes delay), then it began ignoring them again.
• The identity used by the function has Send, Receive and Owner rights to the Event Hub.
• When I used a regular Kafka broker deployed in Azure Container Instances instead of an Event Hub with Kafka surface, the deployed function received messages as it should. However, due to the existing test setup in the project, this is not a viable option.
• When deployed, the old Event Hub trigger is triggered by every message in the same Hub that the Kafka trigger is (mostly) ignoring.
• Increasing SessionTimeoutMs in host.json does not change the behavior.
• After the Functions of the project are deployed, they are restarted and synced programmatically.
• The Event Hub sometimes shows more outgoing than incoming messages, which I assume should indicate that more than one consumer is listening, but the Function is not triggering: Event Hub messages overview

How to reproduce

• Deploy the above resources to bicep, adding values as appropriate.
• Use the Kafka console producer or a Java producer to send a message to the event hub.
• Confirm in the Azure Portal that the Function does not react

Has it worked previously?

It works once in a while, mainly after certain config changes (added "manage event hubs" rights to the managed identity, or reduced partitions to 1).

Checklist

Please provide the relevant information for the following items:

• If using Apache Kafka Java clients or a framework that uses Apache Kafka Java clients, version: org.apache.kafka 2.8.2
• [ x] Namespace and EventHub/topic name: "dev23590-metadataingester", "licenses".
• [ x] Consumer or producer failure Consumer failure
• [ x] Timestamps in UTC 10/17/2023 @ 9:11am
• [ x] group.id or client.id consumer-group-1, also tried other names.
• [x ] Standalone repro Willing/able to send scenario to repro issue>
• [ x] Operating system: on Azure
• [x ] Critical issue

If this is a question on basic functionality, please verify the following:

• [ x] Port 9093 should not be blocked by firewall ("broker cannot be found" errors)
• [ x] Pinging FQDN should return cluster DNS resolution (e.g. $ ping namespace.servicebus.windows.net returns ~ ns-eh2-prod-am3-516.cloudapp.net [13.69.64.0]):
• [x ] Namespace should be either Standard or Dedicated tier, not Basic (TopicAuthorization errors)