Evaluation failed when AIFoundry created Identity Based

[duongthaiha]

• Package Name: azure-ai-evaluation
• Package Version: 1.1.0
• Operating System: Any
• Python Version: 3.11

Describe the bug
When AI Foundry is created with storage access using the Identity-based access
When evaluation function from the SDK call ie with GroundednessEvaluator then evaluation failed with the error EvaluationException: (InternalError) The workspace datastore secrets request failed with HTTP 400

To Reproduce
Steps to reproduce the behavior:

1. Create AI Foundry and AI Project using Storage Access with IdentityBase with disabled SharedKeyAccess
2. Execute step follow in example [note book](https://github.com/Azure-Samples/azureai-samples/blob/5269dfb7843d50186217d64a9f593e41e70d1061/scenarios/evaluate/Simulators/Simulate_Evaluate_Groundedness/Simulate_Evaluate_Groundedness.ipynb)

3. Last step will fail with the error as stated above

Expected behavior
When AI Foundry and project is created using identity base then it will works the same as credential based

Screenshots

Additional context

The error occur when _eval_run.py line 424
self._management_client.workspace_get_default_datastore(self._workspace_name, True)
This will throw and error when trying to get the secret from data store as it is not available

Here is the sample code that used

import logging
import os
import sys
import uuid
from azure.identity import DefaultAzureCredential

# Set up logging
logger = logging.getLogger('azure')
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
logging_enable=True
print(
    f"Logger enabled for ERROR={logger.isEnabledFor(logging.ERROR)}, "
    f"WARNING={logger.isEnabledFor(logging.WARNING)}, "
    f"INFO={logger.isEnabledFor(logging.INFO)}, "
    f"DEBUG={logger.isEnabledFor(logging.DEBUG)}"
)

# Ensure Azure credentials are set up
credential = DefaultAzureCredential()
token = credential.get_token("https://management.azure.com/.default")
print(f"Token acquired: {token.token[:10]}...")

# Ensure project scope and model config are correctly set
assert "subscription_id" in project_scope, "Subscription ID is missing in project_scope"
assert "resource_group_name" in project_scope, "Resource group name is missing in project_scope"
assert "project_name" in project_scope, "Project name is missing in project_scope"
assert "azure_endpoint" in model_config, "Azure endpoint is missing in model_config"
assert "azure_deployment" in model_config, "Azure deployment is missing in model_config"

# Initialize the evaluator
groundedness_evaluator = GroundednessEvaluator(model_config=model_config)

# Run the evaluation
eval_output = evaluate(
    data=output_file,
    evaluators={
        "groundedness": groundedness_evaluator,
    },
    azure_ai_project=project_scope,
)
print(eval_output)

[duongthaiha]

@ralph-msft Hi I think i saw your commit that fix this bug. By any chance you know when this can be release or how to get the bug fix please?
Thank you very much in advance

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @luigiw @needuv @singankit.