-
Notifications
You must be signed in to change notification settings - Fork 216
Description
Release version
v4.5.0
Describe the bug
We have configured the event hub logger for apim using the system-assigned identity. While running the Publisher to Dev environment is failing with the below exception on the EH logger. I could not able to find any reference in the documentation around eventhub setup using system identity; the wiki and few other issues reported on logger were talking about the logger configured using the connectionString.
exception log:
System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/**-int-dv-apim-rg/providers/Microsoft.ApiManagement/service/**-int-dv-apim/loggers/apim-dv-eventhub-logger?api-version=2022-04-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"One or more fields contain incorrect values:","details":[{"code":"ValidationError","target":"One or more Properties ['{0}'] specified are missing.","message":"64c7d150fb42680fe*******"}]}}'.
We noticed when the extractor ran that the associated extracted json for the logger only contains the following
loggerInformation.json, but it was missing the other properties within the credentials.
`{
"properties": {
"credentials": {
"name": "i**-mgmt-pr-evh-02"
},
"description": "adding a new logger with system assigned managed identity",
"isBuffered": true,
"loggerType": "azureEventHub"
}
}`
With the above-extracted json, we noticed the publisher to dev was failing with the missing connectionString property was missing; since we did not use connectionstring to set up the logger and it was using the system identity. I tried to update the JSON logger as below by adding the endpoint and identityClientID (generated named value for the logger), but it failed with the error mentioned in the exception log above.
loggerInformation_manual_update.json
`{
"properties": {
"credentials": {
"identityClientId": "{{64c7d150fb42680**********}}",
"name": "***-mgmt-pr-evh-02",
"endpointAddress": "***-mgmt-pr-evhns.servicebus.windows.net/***-mgmt-pr-evh-02"
},
"description": "adding a new logger with system assigned managed identity",
"isBuffered": true,
"loggerType": "azureEventHub"
}
}`
Documentation referred for setting up EH logger using system identity https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell#logger-with-system-assigned-managed-identity-credentials
a
Expected behavior
We expect the publisher to dev environment should not fail while pushing the changes back to Dev environment and eventually to higher environments
Actual behavior
System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/gi-int-dv-apim-rg/providers/Microsoft.ApiManagement/service/gi-int-dv-apim/loggers/apim-dv-eventhub-logger?api-version=2022-04-01-preview failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"One or more fields contain incorrect values:","details":[{"code":"ValidationError","target":"One or more Properties ['{0}'] specified are missing.","message":"64c7d150fb42680fe4*****"}]}}'.
Reproduction Steps
- EH logger added using system identity for steps mentioned here https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell#logger-with-system-assigned-managed-identity-credentials
- Ran extractor and publisher job. Publisher was using the json file loggerInformation.json, where the publisher fails on missing connectionstring property
- Manually updated the loggerInformation.json, see above mentioned block for loggerInformation_manual_update.json, where with this config, publisher has failed with the exception as provided above
Activity
guythetechie commentedon Aug 2, 2023
@Mohid-A - according to the documentation, here's the proper JSON structure for Event Hub with managed identities. You were close, except that
identityClientId
expects the hard-coded valueSystemAssigned
; not the actual client ID.I would have expected the client ID as well and agree it's confusing. Please don't shoot the messenger. :)
A-Muqees commentedon Aug 2, 2023
@guythetechie - I tried that option still no luck
2023-08-02T14:14:48.1720521Z System.Net.Http.HttpRequestException: HTTP request to URI https://management.azure.com/subscriptions/***/resourceGroups/gi-***-dv-apim-rg/providers/Microsoft.ApiManagement/service/***-dv-apim/loggers/apim-dv-eventhub-logger?api-version=2022-04-01-preview failed with status code 500. Content is '{"error":{"code":"InternalServerError","message":"Request processing failed due to internal error.","details":null}}'.
A-Muqees commentedon Aug 2, 2023
@waelkdouh @guythetechie any suggestions on the fix when you get a chance?
A-Muqees commentedon Aug 8, 2023
Hi @guythetechie, we are stuck on this issue trying to figure out the fix. Do you have any thoughts on this? Also can't find any reference in apiops documentation/wiki on EH logger using managed identity. The documentation only includes the explanation of implementation using connectionstring. thank you
guythetechie commentedon Aug 8, 2023
@Mohid-A - I don't think we've tested this scenario specifically (Event Hubs with managed identities). I'll test tomorrow and respond.
guythetechie commentedon Aug 10, 2023
I was able to successfully create it with this configuration:
Also note that I'm not passing a resource ID; again, don't know if it makes a difference.
A-Muqees commentedon Aug 15, 2023
@guythetechie I tried with suggested approach still resulted in failure. I have set the logging levels to DEBUG, still see no information around the root cause of the internal server error.
Here's the updated loggerInformation.json I tried on the publish-to-dev job, also tried removing the logger name and id in the subsequent runs too
Exception logs
Also for your testing, did you create the logger as mentioned here https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell#logger-with-system-assigned-managed-identity-credentials
waelkdouh commentedon Sep 21, 2023
@Mohid-A if you resolved your issue please close the issue. As @guythetechie demonstrated he is not facing the issue when he tried to replicate it.
A-Muqees commentedon Oct 11, 2023
Hi @waelkdouh I am still facing the issue and it's not resolved yet. I checked with @guythetechie if he followed the below steps provided by Microsoft and waiting for this response.
https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell#logger-with-system-assigned-managed-identity-credentials
waelkdouh commentedon Oct 11, 2023
We will get back to you when we get a chance to take a look. Please note that this OS an open source project. So we can't promise any timelines.
Patrick-Chojnacki-Snow commentedon Feb 26, 2024
Hello @waelkdouh Has their been any progress with this issue as we are experiencing it aswell? Thank you!
waelkdouh commentedon Feb 26, 2024
Can you please post the logs? Make sure you scrub them please for your privacy.
waelkdouh commentedon Feb 26, 2024
@Mohid-A did you ever resolve this issue? @guythetechie cousint reproduce the issue.
Patrick-Chojnacki-Snow commentedon Feb 26, 2024
2024-02-22T17:32:16.7591736Z info: Extractor[0]
2024-02-22T17:32:16.7592773Z Writing logger information file /tmp/apiops-extractor-sandbox/loggers/apim-logger-csdoopsaoai/loggerInformation.json...
2024-02-22T17:32:16.7593950Z info: Extractor[0]
The extractor tools extract all my APIM information including the loggers, but for the logger types EventHub it is missing these properties.
"endpointAddress": "didier-events.servicebus.windows.net/didier-hub",
"identityClientId": "{{65d7af00000000000032}}",
A-Muqees commentedon Feb 27, 2024
@waelkdouh I still do not have success on this. I have not got any response back on the below question
I checked with @guythetechie if he followed the below steps provided by Microsoft and waiting for this response.
https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell#logger-with-system-assigned-managed-identity-credentials
waelkdouh commentedon Feb 28, 2024
I will check with him again. But looking at the history of this conversation it seems like he was successful with his test.
guythetechie commentedon Feb 28, 2024
@Mohid-A - I just tried again successfully with these settings.
One other thing to check: have you given your APIM instance permissions on the Event Hub? It needs the Azure Event Hubs Data Sender role.

Patrick-Chojnacki-Snow commentedon Mar 1, 2024
We confirmed that when we send the REST API call it retrieves the correct properties, so that means APIM and EventHub are able to communicate properly and have the correct roles. The issue is on the tool itself, when writing the properties on the loggerInformation.json file.
Patrick-Chojnacki-Snow commentedon Mar 12, 2024
Hi Everyone, Just wondering if we made any progress with this issue and what next steps might be looking like. Thank you
guythetechie commentedon Mar 12, 2024
@Patrick-Chojnacki-Snow - the extractor will not extract the connection string and other sensitive information. The APIM REST API doesn't expose it via a GET request. You will have to pass that information by overriding your publisher configuration file. Something like this:
Patrick-Chojnacki-Snow commentedon Mar 15, 2024
We tried with that config but it did not work
joshuaauger commentedon Mar 20, 2024
@guythetechie Following up on what @Patrick-Chojnacki-Snow said, we merged the suggestions into
loggers/<logger>/loggerInformation.json
as we aren't using the configuration file. We then saw this issue: