You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Test file naming: rename the live-only WebSocket realtime tests so 'realtime' names the feature (client.beta.voice_agents.realtime) and 'livetest' -- one token -- marks that the module only runs against the live service, instead of 'live' sitting next to 'realtime'/'voice_agent' where it could be misread as describing a streaming agent (and be confused with the unrelated azure-ai-voicelive package):
test_voice_agent_realtime_live.py -> test_voice_agent_realtime_livetest.py
test_voice_agent_realtime_live_async.py -> test_voice_agent_realtime_livetest_async.py
Renamed TestVoiceAgentRealtimeLive -> TestVoiceAgentRealtimeLivetest and the
async counterpart to match.
Sample file naming: the same 'live' ambiguity existed in four samples, so 'live' is replaced with 'realtime' there too:
sample_voice_agent_live_text_conversation.py -> sample_voice_agent_realtime_text_conversation.py
sample_voice_agent_live_text_conversation_async.py -> sample_voice_agent_realtime_text_conversation_async.py
sample_voice_agent_live_audio_conversation_async.py -> sample_voice_agent_realtime_audio_conversation_async.py
sample_voice_agent_live_function_tool.py -> sample_voice_agent_realtime_function_tool.py
Updated every cross-reference (docstrings, USAGE lines, default agent-name
strings, CHANGELOG.md, tests/README.md) accordingly. Verified black-clean,
all files still parse, and all 6 renamed tests collect under their new
names and pass live against a real Foundry endpoint (function-tool/text/
lifecycle scenarios); the async trio only passes once a local azure-core
Brotli decompression gap is worked around, a pre-existing environment issue
unrelated to this rename (confirmed via A/B testing against the original
file).
Also adds sdk/ai/azure-ai-projects/test-resources.json (this package had no test-resources file for any feature area before), modeled on sdk/voicelive/azure-ai-voicelive/test-resources.json: provisions a Cognitive Services AIServices account with a default Foundry project, plus a role assignment granting the built-in Foundry User role to the test principal (needed since these tests authenticate via AAD, unlike voicelive which uses key auth). Documented in tests/README.md.
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.
All SDK Contribution checklist:
The pull request does not introduce [breaking changes]
CHANGELOG is updated for new features, bug fixes or other significant changes.
Title of the pull request is clear and informative.
There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.
…ources.json
Test file naming: rename the live-only WebSocket realtime tests so 'realtime'
names the feature (client.beta.voice_agents.realtime) and 'livetest' -- one
token -- marks that the module only runs against the live service, instead of
'live' sitting next to 'realtime'/'voice_agent' where it could be misread as
describing a streaming agent (and be confused with the unrelated
azure-ai-voicelive package):
test_voice_agent_realtime_live.py -> test_voice_agent_realtime_livetest.py
test_voice_agent_realtime_live_async.py -> test_voice_agent_realtime_livetest_async.py
Renamed TestVoiceAgentRealtimeLive -> TestVoiceAgentRealtimeLivetest and the
async counterpart to match.
Sample file naming: the same 'live' ambiguity existed in four samples, so
'live' is replaced with 'realtime' there too:
sample_voice_agent_live_text_conversation.py -> sample_voice_agent_realtime_text_conversation.py
sample_voice_agent_live_text_conversation_async.py -> sample_voice_agent_realtime_text_conversation_async.py
sample_voice_agent_live_audio_conversation_async.py -> sample_voice_agent_realtime_audio_conversation_async.py
sample_voice_agent_live_function_tool.py -> sample_voice_agent_realtime_function_tool.py
Updated every cross-reference (docstrings, USAGE lines, default agent-name
strings, CHANGELOG.md, tests/README.md) accordingly. Verified black-clean,
all files still parse, and all 6 renamed tests collect under their new
names and pass live against a real Foundry endpoint (function-tool/text/
lifecycle scenarios); the async trio only passes once a local azure-core
Brotli decompression gap is worked around, a pre-existing environment issue
unrelated to this rename (confirmed via A/B testing against the original
file).
Also adds sdk/ai/azure-ai-projects/test-resources.json (this package had no
test-resources file for any feature area before), modeled on
sdk/voicelive/azure-ai-voicelive/test-resources.json: provisions a
Cognitive Services AIServices account with a default Foundry project, plus a
role assignment granting the built-in Foundry User role to the test
principal (needed since these tests authenticate via AAD, unlike voicelive
which uses key auth). Documented in tests/README.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.
This publishes the account key as a plain deployment output even though these tests create clients with AAD credentials and no package code consumes FOUNDRY_PROJECT_API_KEY. Deployment outputs are retained and can be surfaced by tooling, so remove this unnecessary credential exposure.
The freshly provisioned project has no realtime model deployment, but every renamed live test requires FOUNDRY_VOICE_MODEL_NAME and uses that deployment to create an agent. An automated live pipeline using this template will therefore fail against fresh resources; provision a realtime-capable deployment and output its name, or explicitly bind the pipeline to a pre-provisioned project.
…re-ai-projects
Adds sdk/ai/azure-ai-projects/tests.yml (a package-level pipeline entry point, coexisting
with the shared sdk/ai/tests.yml used for cross-package weekly static analysis), modeled on
sdk/voicelive/azure-ai-voicelive/tests.yml and sdk/core/azure-core-tracing-opentelemetry/tests.yml.
It extends archetype-sdk-tests.yml (the actual live-test stage template, not the
analyze-only one) and wires in the test-resources.json added previously, scoped to:
- BuildTargetingString: azure-ai-projects, so only this package runs, not azure-ai-agents or
azure-ai-inference which also live under sdk/ai.
- TestMarkArgument: live_test_only, so only tests decorated with @pytest.mark.live_test_only
run -- currently just tests/agents/test_voice_agent_realtime_livetest.py and its async
counterpart. This intentionally excludes the rest of azure-ai-projects' test suite, which
needs resources/connections test-resources.json does not provision (telephony phone numbers,
fine-tuning jobs, hosted-agent images, Bing/SharePoint/GitHub connections, ...) and in some
cases (telephony) places real phone calls, so it must not run unattended here.
FOUNDRY_VOICE_MODEL_NAME is left as a pipeline variable reference ($(FoundryVoiceModelName))
since test-resources.json provisions the Foundry account/project but not a model deployment;
that still needs to be created once and wired into a pipeline variable group.
Also documents this in tests/README.md, including the note that an actual Azure DevOps pipeline
definition pointing at this tests.yml still needs to be created by the engineering-systems team
before it runs -- adding tests.yml here does not by itself create a running pipeline.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The pipeline lacks a model deployment, includes async tests known to require an uncommitted workaround, and documents an invalid resource-script path.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
sdk/ai/azure-ai-projects/tests/README.md:34
This command is run from sdk\ai\azure-ai-projects per line 8, so ../../eng resolves under sdk and does not exist. Go up three levels to reach the repository-root eng directory.
../../eng/common/TestResources/New-TestResources.ps1 -ServiceDirectory ai
FOUNDRY_VOICE_MODEL (samples) and FOUNDRY_VOICE_MODEL_NAME (tests) both held
the realtime/chat model deployment name (e.g. "gpt-realtime"), not the actual
TTS voice -- "voice model" was misleading. Consolidate both into a single
FOUNDRY_VOICE_AGENT_MODEL name, shared by samples and tests, mirroring how
FOUNDRY_MODEL_NAME is already shared between non-voice samples and tests.
- Update all 10 voice samples (code + docstrings).
- Rename the foundry_voice_model_name EnvironmentVariableLoader kwarg to
foundry_voice_agent_model in tests/test_base.py and all 10 test files that
reference it.
- Consolidate the two separate .env.template entries into one.
- Update tests.yml and tests/README.md references.
Re-ran the live realtime tests and the recorded CRUD tests to confirm
the rename works end-to-end.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The setup command is invalid from the documented directory, and managed-model requirements are described incorrectly.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
sdk/ai/azure-ai-projects/tests.yml:23
These tests pass the value to VoiceAgentDefinition(model_type=VoiceModelType.MANAGED, ...), so it is not the name of a deployment created in this project. This guidance should request a supported managed model name; otherwise pipeline setup may unnecessarily create a deployment that the tests never use. sdk/ai/azure-ai-projects/tests/README.md:34
This command cannot be run from the package directory specified above: ../../eng resolves under sdk, and -ServiceDirectory ai recursively deploys every test-resource template under sdk/ai, not just this package. Point to the repository-level script and scope discovery to this package.
- Convert test-resources.json to test-resources.bicep (referencing
azure-sdk-for-net's voicelive test-resources.bicep), since New-TestResources.ps1
discovers .json and .bicep templates independently and would otherwise deploy
both.
- Exclude the async realtime live tests from the CI pipeline via a new
live_test_only_async marker (registered in tests/conftest.py) and
TestMarkArgument: 'live_test_only and not live_test_only_async', since they
only pass with an azure-core Brotli decompression workaround that isn't
committed. Documented in tests.yml and tests/README.md.
- Fix tests/README.md's New-TestResources.ps1 path: sdk/ai/azure-ai-projects is
3 levels below the repo root, not 2 (../../eng -> ../../../eng).
- Correct documentation that incorrectly implied the live tests' realtime model
needs to be deployed to the Foundry project. The tests exclusively use
VoiceModelType.MANAGED (service-hosted), which requires no project
deployment at all -- only VoiceModelType.SELF_DEPLOYED does. Updated
tests.yml, tests/README.md, test-resources.bicep, and the same "deployment
name" wording in 9 voice sample docstrings for consistency.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove the FOUNDRY_PROJECT_API_KEY output from test-resources.bicep and the
corresponding README instruction: the live tests authenticate with Azure AD
via DefaultAzureCredential (tests/test_base.py's get_credential), so
exporting an account key via listKeys() was an unnecessary secret.
- Preserve FOUNDRY_VOICE_MODEL as a deprecated fallback in
sample_voice_agent_with_tools.py, so an existing self_deployed (BYOM)
configuration doesn't silently start targeting the "gpt-realtime" default
instead of the user's actual deployment name.
- Give the FoundryVoiceAgentModel pipeline variable a checked-in default
("gpt-realtime", matching the samples) in tests.yml, so the pipeline is
runnable without undocumented Azure DevOps UI configuration.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The PR description still says this change adds sdk/ai/azure-ai-projects/test-resources.json, but the implementation has been converted to test-resources.bicep. Update the description so reviewers and release records point to the file that is actually added.
Remove outdated azure-core version qualifier
sdk/ai/azure-ai-projects/tests.yml:28
The repository is already on azure-core 1.42.0, and its aiohttp streaming decompressor still lacks br handling, so the newly added “as of 1.41.0” qualifier is outdated on arrival. Remove the version qualifier to keep this pipeline rationale accurate until support is added.
Update stale azure-core version qualifier
sdk/ai/azure-ai-projects/tests/README.md:60
This version qualifier is already stale: this repository currently contains azure-core 1.42.0 (sdk/core/azure-core/azure/core/_version.py:12), whose aiohttp streaming decompressor still handles only gzip and deflate (_aiohttp.py:459-469). Avoid pinning the explanation to 1.41.0, or update it to the current version.
… vars
Build 6858694 failed with:
Exception: Preparer failure when creating resource EnvironmentVariableLoader
for test ...: 'PAUSED_FINE_TUNING_JOB_ID'
test_base.py's shared `servicePreparer` bakes in ~40 kwargs and is registered
with an empty `directory=""`, which matches every key -- so it requires ALL of
them to be present as environment variables (blank is fine) whenever
AZURE_TEST_RUN_LIVE=true, not just the 2 the realtime tests actually use.
tests.yml's EnvVars only defined 3.
Rather than listing 39 unrelated blank placeholders (fine-tuning, Bing,
SharePoint, memory store, ...) in tests.yml, add a dedicated
`voiceAgentRealtimeServicePreparer` with only the 2 kwargs these tests need
(foundry_project_endpoint, foundry_voice_agent_model), following the same
pattern as this file's existing `modelsServicePreparer`/
`fineTuningServicePreparer`, and mirroring azure-ai-voicelive's dedicated
`VoiceLivePreparer` for its own live tests. Switch both realtime live-test
files to use it.
tests.yml itself needs no changes -- verified by simulating the exact CI
environment locally (hid the real .env, injected only tests.yml's 3 EnvVars
into a fresh process): all 3 sync realtime tests pass. Also confirmed the CRUD
and conversations tests, which still use the shared servicePreparer, are
unaffected.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scope test resource discovery to the target package
sdk/ai/azure-ai-projects/tests/README.md:34
This command scopes discovery to all of sdk/ai, and New-TestResources.ps1 recursively deploys every test-resources.json/.bicep it finds there (eng/common/TestResources/New-TestResources.ps1:747-749). Point it at this package so local Voice Agents setup does not provision unrelated AI resources or fail on their parameters.
All jobs fail across every OS/Python-version matrix combination for the same package (azure-ai-projects, VoiceAgentsRealtime test group), which points to a single shared root cause rather than platform-specific bugs, but the underlying log/test data needed to confirm that root cause could not be retrieved by this workflow.
Relevant pipeline output
azure_pipeline_analyses: []
github_workflow_analyses: []
failing_pull_request_checks:
- python - azure-ai-projects - tests (FAILURE, buildId=6858755)
- ... 6 platform-specific job variants (FAILURE)
operation_status: Succeeded
No failed_pipeline_tasks or failed_pipeline_tests data was returned for this
build, and pipeline-test-results.txt (full failed-test detail) was empty.
Recommended next steps
This workflow does not have the user's Azure DevOps identity and could not retrieve log or test-result detail for this internal/private pipeline build, so no root cause, file/line, or fixability determination can be made from here.
Sign in to Azure DevOps with your own credentials and inspect build 6858755 directly to see the actual test failure output for the VoiceAgentsRealtime jobs (this PR renames live-only test files/classes in azure-ai-projects, which is a likely area to check first: test_voice_agent_realtime_livetest*.py).
Alternatively, run the following locally while authenticated to Azure DevOps to get a structured analysis:
azsdk ci analyze https://github.com/Azure/azure-sdk-for-python/pull/49084 --output json
Copilot detected the failing pipeline and generated the analysis above. To have it attempt a
fix automatically, reply with @copilot please fix the failing pipeline on this PR.
Build 6858755 provisioned test-resources.bicep's Foundry account in "westus"
(the pipeline environment's default location), where "gpt-realtime" is not
supported in managed mode:
azure.core.exceptions.HttpResponseError: (bad_request) Model 'gpt-realtime'
is not supported in managed mode in this region.
Pin `Location: westus2` in tests.yml (a documented archetype-sdk-tests.yml
parameter) -- westus2 is confirmed, via Azure Resource Graph, to be the region
of the Foundry resource these tests have been verified against live all
session. Also add `-Location westus2` to the local dev instructions in
tests/README.md.
Separately, that same failed agent creation was being masked by a second,
cascading error: each test's `finally` block unconditionally called
`agents.delete(agent_name=...)`, so when `create_version`/`create_basic_agent`
raised, the delete call 404'd on an agent that was never created
(ResourceNotFoundError: Agent not found), and Python's finally semantics
surfaced that 404 instead of the real underlying error. Track
`agent_created = False -> True` right after a successful create, and only
attempt cleanup if it's True, across all 3 sync and all 3 async realtime
tests, so a future creation failure (quota, transient errors, ...) is never
hidden behind an unrelated cleanup error again.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This sample now ignores the previously documented FOUNDRY_VOICE_MODEL setting and silently targets gpt-realtime for existing .env files. Preserve the deprecated fallback here, as sample_voice_agent_with_tools.py already does, so the variable rename does not change users' selected model.
This sample now ignores the previously documented FOUNDRY_VOICE_MODEL setting and silently targets gpt-realtime for existing .env files. Preserve the deprecated fallback here, as sample_voice_agent_with_tools.py already does, so the variable rename does not change users' selected model.
This sample now ignores the previously documented FOUNDRY_VOICE_MODEL setting and silently targets gpt-realtime for existing .env files. Preserve the deprecated fallback here, as sample_voice_agent_with_tools.py already does, so the variable rename does not change users' selected model.
This sample now ignores the previously documented FOUNDRY_VOICE_MODEL setting and silently targets gpt-realtime for existing .env files. Preserve the deprecated fallback here, as sample_voice_agent_with_tools.py already does, so the variable rename does not change users' selected model.
This sample now ignores the previously documented FOUNDRY_VOICE_MODEL setting and silently targets gpt-realtime for existing .env files. Preserve the deprecated fallback here, as sample_voice_agent_with_tools.py already does, so the variable rename does not change users' selected model.
This sample now ignores the previously documented FOUNDRY_VOICE_MODEL setting and silently targets gpt-realtime for existing .env files. Preserve the deprecated fallback here, as sample_voice_agent_with_tools.py already does, so the variable rename does not change users' selected model.
This sample now ignores the previously documented FOUNDRY_VOICE_MODEL setting and silently targets gpt-realtime for existing .env files. Preserve the deprecated fallback here, as sample_voice_agent_with_tools.py already does, so the variable rename does not change users' selected model.
This sample now ignores the previously documented FOUNDRY_VOICE_MODEL setting and silently targets gpt-realtime for existing .env files. Preserve the deprecated fallback here, as sample_voice_agent_with_tools.py already does, so the variable rename does not change users' selected model.
This sample now ignores the previously documented FOUNDRY_VOICE_MODEL setting and silently targets gpt-realtime for existing .env files. Preserve the deprecated fallback here, as sample_voice_agent_with_tools.py already does, so the variable rename does not change users' selected model.
xitzhang (xitzhang)
changed the title
[draft][azure-ai-project] Add live test pipeline and live tests of VoiceAgents
[azure-ai-project] Add live test pipeline and live tests of VoiceAgents
Sep 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
xitzhang (xitzhang)
changed the title
Test file naming: rename the live-only WebSocket realtime tests so 'realtime' names the feature (client.beta.voice_agents.realtime) and 'livetest' -- one token -- marks that the module only runs against the live service, instead of 'live' sitting next to 'realtime'/'voice_agent' where it could be misread as describing a streaming agent (and be confused with the unrelated azure-ai-voicelive package):
test_voice_agent_realtime_live.py -> test_voice_agent_realtime_livetest.py
test_voice_agent_realtime_live_async.py -> test_voice_agent_realtime_livetest_async.py
Renamed TestVoiceAgentRealtimeLive -> TestVoiceAgentRealtimeLivetest and the
async counterpart to match.
Sample file naming: the same 'live' ambiguity existed in four samples, so 'live' is replaced with 'realtime' there too:
sample_voice_agent_live_text_conversation.py -> sample_voice_agent_realtime_text_conversation.py
sample_voice_agent_live_text_conversation_async.py -> sample_voice_agent_realtime_text_conversation_async.py
sample_voice_agent_live_audio_conversation_async.py -> sample_voice_agent_realtime_audio_conversation_async.py
sample_voice_agent_live_function_tool.py -> sample_voice_agent_realtime_function_tool.py
Updated every cross-reference (docstrings, USAGE lines, default agent-name
strings, CHANGELOG.md, tests/README.md) accordingly. Verified black-clean,
all files still parse, and all 6 renamed tests collect under their new
names and pass live against a real Foundry endpoint (function-tool/text/
lifecycle scenarios); the async trio only passes once a local azure-core
Brotli decompression gap is worked around, a pre-existing environment issue
unrelated to this rename (confirmed via A/B testing against the original
file).
Also adds sdk/ai/azure-ai-projects/test-resources.json (this package had no test-resources file for any feature area before), modeled on sdk/voicelive/azure-ai-voicelive/test-resources.json: provisions a Cognitive Services AIServices account with a default Foundry project, plus a role assignment granting the built-in Foundry User role to the test principal (needed since these tests authenticate via AAD, unlike voicelive which uses key auth). Documented in tests/README.md.
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines