feat: Support GPT Realtime GA API on Azure

[dhansmair]

Feature Type

I cannot use LiveKit without it

Feature Description

The Azure OpenAI GPT Realtime API is now Generally Available (GA).
According to the Azure documentation, the GPT Realtime GA API is now available and preview versions will be deprecated.

• Migration Guide: https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/realtime-audio-preview-api-migration-guide
• https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/realtime-audio?view=foundry-classic&tabs=api-key%2Cmacos&pivots=programming-language-python#quickstart

Is it currently possible to use livekit-agents with Azure OpenAI GPT Realtime?

I have tried various options to see if we can switch right now:

# Option 1
realtime_model = openai.realtime.RealtimeModel(
    base_url=AZURE_OPENAI_LLM_ENDPOINT,
    api_key=AZURE_OPENAI_LLM_API_KEY,
    model="gpt-realtime",
)

# Option 2
realtime_model = openai.realtime.RealtimeModel.with_azure(
    azure_deployment=AZURE_OPENAI_LLM_DEPLOYMENT,
    azure_endpoint=AZURE_OPENAI_LLM_ENDPOINT,
    api_key=AZURE_OPENAI_LLM_API_KEY,
)

but both attempts didn't work.

The Azure OpenAI GA API is supported by the offical openai Python SDK, that is why I thought the livekit openai (without the with_azure()) could work as well.

Workarounds / Alternatives

No response

Additional Context

No response

[dhansmair]

I did some more troubleshooting on this issue and managed to get it work with Azure GPT Realtime (GA API) and Azure GPT Realtime 1.5 (GA) using the default RealtimeModel constructor.

Problems I found:

• The api_key parameter needs to be set as a query param. If it is set as an "Authorization " header or api-key header, the Azure endpoint will return a redirect where api-key is a query param in the URL string.
• aiohttp apparently cannot deal with websocket redirects and raises the following exception:

aiohttp.client_exceptions.NonHttpUrlRedirectClientError: wss://{resource}.openai.azure.com/openai/v1/realtime?model=gpt-realtime-1.5&api-key={api_key}

I have created a basic integration test for the GA API connection.
A simplified version where the api-key is appended directly to the URL:

@pytest.mark.asyncio()
async def test_gpt_realtime_with_azure_GA_version_succeeds(caplog):
    caplog.set_level("DEBUG")
    logging.getLogger().setLevel("DEBUG")

    import aiohttp

    # GPT Realtime GA API
    # deployment = "gpt-realtime"
    # base_url = "wss://swedencentral.api.cognitive.microsoft.com/openai/v1/realtime"
    # api_key = "..."

    # GPT Realtime 1.5 GA API
    deployment = "gpt-realtime-1.5"
    base_url = "wss://{our resource}.openai.azure.com/openai/v1/realtime"
    api_key = "..."

    # GIVEN
    llm = RealtimeModel(
        model=deployment,
        base_url=f"{base_url}?api-key={api_key}",
        api_key="value does not matter",
        modalities=["text"],
        input_audio_transcription=realtime.AudioTranscription(
            model="gpt-4o-transcribe",
        ),
        input_audio_noise_reduction="near_field",
        turn_detection=realtime.realtime_audio_input_turn_detection.SemanticVad(
            type="semantic_vad",
            create_response=True,
            eagerness="auto",
            interrupt_response=True,
        ),
    )
    session = AgentSession(llm=llm)
    agent = Agent(instructions="you are a friendly AI assistant")

    # WHEN
    await session.start(agent)

    response = await session.generate_reply(user_input="hello")

    # THEN
    assert len(response.chat_items) > 0
    assert len(response.chat_items[0].content) > 0
    await session.aclose()

[davidzhao]

hi @dhansmair, would you like to open a PR to address the compatibility issues?