[Storage Queue] Migrating to TypeSpec by l0lawrence · Pull Request #45649 · Azure/azure-sdk-for-python

l0lawrence · 2026-03-11T19:25:00Z

samples failing are also failing on main

l0lawrence · 2026-03-18T16:09:43Z

/azp run python - storage - tests

azure-pipelines · 2026-03-18T16:10:06Z

Azure Pipelines successfully started running 1 pipeline(s).

l0lawrence · 2026-03-19T22:16:08Z

/azp run python - storage - tests

azure-pipelines · 2026-03-19T22:16:28Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR migrates azure-storage-queue to TypeSpec-generated code (dpcodegen), replacing the legacy AutoRest generated layer and updating the handwritten clients/tests/packaging to work with the new generated surface.

Changes:

Switch packaging from setup.py to pyproject.toml and add TypeSpec generation metadata files.
Update sync/async Queue clients and paging/model handling to use the new generated QueuesClient + model shapes (e.g., items_property).
Adjust test proxy sanitization/matching configuration and lint suppressions for large test/client files.

Reviewed changes

Copilot reviewed 50 out of 50 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
sdk/storage/azure-storage-queue/tsp-location.yaml	Adds TypeSpec spec location config for queue storage.
sdk/storage/azure-storage-queue/tests/test_queue.py	Adds pylint suppression for file length.
sdk/storage/azure-storage-queue/tests/test_queue_async.py	Adds pylint suppression for file length.
sdk/storage/azure-storage-queue/tests/conftest.py	Updates proxy sanitizers and default matcher behavior.
sdk/storage/azure-storage-queue/setup.py	Removes legacy setuptools `setup.py` packaging.
sdk/storage/azure-storage-queue/pyproject.toml	Adds PEP 621 packaging configuration.
sdk/storage/azure-storage-queue/MANIFEST.in	Adjusts sdist included files.
sdk/storage/azure-storage-queue/azure/storage/queue/aio/_queue_service_client_async.py	Migrates async service client to new generated client/ops.
sdk/storage/azure-storage-queue/azure/storage/queue/aio/_queue_client_async.py	Migrates async queue client to new generated client/ops and model shapes.
sdk/storage/azure-storage-queue/azure/storage/queue/aio/_models.py	Updates async paging extraction for new `items_property` containers.
sdk/storage/azure-storage-queue/azure/storage/queue/_shared/response_handlers.py	Adjusts user delegation key date serialization behavior.
sdk/storage/azure-storage-queue/azure/storage/queue/_queue_service_client.py	Migrates sync service client to new generated client/ops.
sdk/storage/azure-storage-queue/azure/storage/queue/_queue_client.py	Migrates sync queue client to new generated client/ops and model shapes.
sdk/storage/azure-storage-queue/azure/storage/queue/_models.py	Updates message decoding and sync paging extraction for `items_property`.
sdk/storage/azure-storage-queue/azure/storage/queue/_message_encoding.py	Updates decode policy to work with new generated model descriptors.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/operations/_service_operations.py	Removes old AutoRest-generated service ops.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/operations/_queue_operations.py	Removes old AutoRest-generated queue ops.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/operations/_messages_operations.py	Removes old AutoRest-generated messages ops.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/operations/_message_id_operations.py	Removes old AutoRest-generated message-id ops.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/operations/_patch.py	Modernizes typing header/`__all__` typing.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/operations/init.py	Points operations exports to new consolidated `_operations` module.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/models/_models_py3.py	Removes old AutoRest model file.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/models/_enums.py	Updates generated enums to dpcodegen names and docstrings.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/models/init.py	Updates exported generated model set to dpcodegen equivalents.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/models/_patch.py	Adds runtime patches for dpcodegen model behavior/descriptors.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/apiview-properties.json	Adds generated APIView cross-language mapping file.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/aio/operations/_service_operations.py	Removes old AutoRest async service ops.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/aio/operations/_queue_operations.py	Removes old AutoRest async queue ops.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/aio/operations/_messages_operations.py	Removes old AutoRest async messages ops.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/aio/operations/_message_id_operations.py	Removes old AutoRest async message-id ops.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/aio/operations/_patch.py	Modernizes typing header/`__all__` typing for async ops patch.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/aio/operations/init.py	Points async operations exports to new consolidated `_operations` module.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/aio/_patch.py	Adds async patched `QueuesClient` to support pre-built pipeline instantiation.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/aio/_configuration.py	Updates async generated configuration to dpcodegen + bearer auth policy wiring.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/aio/_client.py	Updates async generated client to dpcodegen naming/signature and `send_request`.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/aio/init.py	Updates async generated package exports to `QueuesClient`.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/_version.py	Adds dpcodegen-generated version constant for generated layer.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/_utils/serialization.py	Applies formatting changes and minor logging formatting updates.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/_utils/init.py	Updates generator attribution header.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/_patch.py	Adds patched `QueuesClient` to support pre-built pipeline instantiation.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/_metadata.json	Adds dpcodegen API version metadata.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/_configuration.py	Updates generated configuration to dpcodegen + bearer auth policy wiring.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/_client.py	Updates generated client to dpcodegen naming/signature and `send_request`.
sdk/storage/azure-storage-queue/azure/storage/queue/_generated/init.py	Updates generated package exports to `QueuesClient` and sets `__version__`.
sdk/storage/azure-storage-queue/apiview-properties.json	Adds package-level APIView cross-language mapping file.
sdk/storage/azure-storage-queue/_metadata.json	Adds package-level dpcodegen API version metadata.

Copilot · 2026-04-06T20:17:56Z

+directory: specification/storage/Microsoft.QueueStorage
+commit: <replace with your value>
+repo: <replace with your value>
+additionalDirectories: 


tsp-location.yaml still contains placeholder values for commit and repo ("") and an empty additionalDirectories entry. This will break or mislead TypeSpec generation tooling; please replace these with the actual spec repo URL + commit SHA (and set additionalDirectories to a valid YAML list, e.g. [], if none are needed).

Copilot · 2026-04-06T20:17:58Z

+            # When a pre-built pipeline is provided, skip the generated credential/policy setup.
+            _endpoint = "{url}"
+            self._config = QueuesClientConfiguration.__new__(QueuesClientConfiguration)
+            version = kwargs.pop("version", "2026-06-06")


The pipeline-based fallback defaults version to "2026-06-06", which doesn’t match the API version declared elsewhere in the generated code/metadata (2026-04-06). If callers instantiate QueuesClient(..., pipeline=...) without explicitly passing version, requests may be sent with an invalid/unsupported service version. Please default to the generated client’s actual API version (or derive it from _metadata.json).

Suggested change

version = kwargs.pop("version", "2026-06-06")

version = kwargs.pop("version", "2026-04-06")

fair -- need to look into gen code versions

Copilot · 2026-04-06T20:17:58Z

+            # When a pre-built pipeline is provided, skip the generated credential/policy setup.
+            _endpoint = "{url}"
+            self._config = QueuesClientConfiguration.__new__(QueuesClientConfiguration)
+            version = kwargs.pop("version", "2026-06-06")


The async pipeline-based fallback defaults version to "2026-06-06", which doesn’t match the API version declared elsewhere in the generated code/metadata (2026-04-06). If callers instantiate QueuesClient(..., pipeline=...) without explicitly passing version, requests may be sent with an invalid/unsupported service version. Please default to the generated client’s actual API version (or derive it from _metadata.json).

Suggested change

version = kwargs.pop("version", "2026-06-06")

version = kwargs.pop("version", "2026-04-06")

l0lawrence · 2026-04-07T17:08:37Z

/azp run python - storage - tests

azure-pipelines · 2026-04-07T17:08:56Z

Azure Pipelines successfully started running 1 pipeline(s).

annatisch · 2026-04-15T04:48:02Z

        self.version = version
-        kwargs.setdefault("sdk_moniker", "azurequeuestorage/{}".format(VERSION))
+        self.credential_scopes = kwargs.pop("credential_scopes", ["https://storage.azure.com/.default"])
+        kwargs.setdefault("sdk_moniker", "storage-queue/{}".format(VERSION))


Were we always overriding the sdk_moniker? Because if not - this change could break customers (and our) telemetry.

I believe we do here:

azure-sdk-for-python/sdk/storage/azure-storage-queue/azure/storage/queue/_shared/base_client.py

Line 184 in 36c1bc0

self._sdk_moniker = f"storage-{service}/{VERSION}"

annatisch · 2026-04-15T04:49:55Z

+        if _pipeline is not None:
+            # When a pre-built pipeline is provided, skip the generated credential/policy setup.
+            _endpoint = "{url}"
+            self._config = QueuesClientConfiguration.__new__(QueuesClientConfiguration)


this seems..... awkward?

annatisch · 2026-04-15T04:55:55Z

I'm assuming all changes to the _shared directory have been duplicated across the other storage sdks?

annatisch · 2026-04-15T05:14:22Z

+from .._utils.model_base import Model as _Model, rest_field, _MyMutableMapping, _RestField
+
+
+def _patched_getattr(self, name):


I know we had to do this for blobs because of the datalake dependency - though I'm not sure it's the best approach for queues/files?

yes I can try just editing the _models.py file for this and files

annatisch · 2026-04-15T05:15:46Z

+
+# The original ``Model.__new__`` does ``rf._module = cls.__module__`` which
+# lets an external subclass (e.g. from azure-storage-file-datalake) overwrite
+# ``_module`` on the *shared* descriptor, corrupting type resolution for


This also feels like it's specific to the Blobs/Datalake issue, and is added complexity that's not needed for queues?

annatisch · 2026-04-15T05:58:31Z

+    object.__setattr__(self, name, value)
+
+
+def _patched_getattribute(self, name):


Is patching getattribute going to assist with the missing methods on the baseclass?
It doesn't appear to be specifically matching for the lost method names....
Or has the decision been made to take a breaking change there? That seems risky for Storage...

…ue _shared/models.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…k-for-python into storagequeue

l0lawrence · 2026-04-22T19:46:36Z

/azp run python - storage - tests

azure-pipelines · 2026-04-22T19:46:58Z

Azure Pipelines successfully started running 1 pipeline(s).

l0lawrence added 4 commits March 10, 2026 12:28

storage queue

e9affdb

basic changes

bffece2

filter conftest

c8ce399

fixing test issues

703f7c1

github-actions Bot added the Storage Storage Service (Queues, Blobs, Files) label Mar 11, 2026

l0lawrence added 7 commits March 11, 2026 15:10

missing init

550b8a9

update response handler to handle stirng

14b57c5

userdelegationkey

3208396

update gen location

61169cb

pyproject

8687c28

update imports

b1381ee

regen + updates

52380c1

l0lawrence added 10 commits March 18, 2026 09:12

change conftest

ac18061

mypy + blakc

b1a40be

regen

38e05ae

sphinx

7cf6d01

Merge branch 'main' into storagequeue

47df4bb

regen+fixing apistub

1df6198

version file

a049612

q

6124cf3

edit

ced0162

pylint fix

21a96ce

l0lawrence added 4 commits March 23, 2026 13:49

regen

730959c

encoding??

5576280

regen keyinfo as string

0392876

pin azure-core

ca32d82

l0lawrence changed the title ~~[dnm] storage queue~~ [Storage Queue] Migrating to TypeSpec Apr 3, 2026

regen

a20698a

l0lawrence commented Apr 6, 2026

View reviewed changes

Comment thread sdk/storage/azure-storage-queue/azure/storage/queue/_generated/aio/_patch.py Outdated

ai version

23236f7

l0lawrence marked this pull request as ready for review April 6, 2026 20:09

l0lawrence requested a review from annatisch as a code owner April 6, 2026 20:09

Copilot AI review requested due to automatic review settings April 6, 2026 20:09

l0lawrence requested review from jalauzon-msft, vincenttran-msft and weirongw23-msft as code owners April 6, 2026 20:09

Copilot started reviewing on behalf of l0lawrence April 6, 2026 20:10 View session

pyproject updates

63c6f4f

Copilot AI reviewed Apr 6, 2026

View reviewed changes

fix []

7473c90

l0lawrence added 4 commits April 13, 2026 10:27

Merge branch 'main' into storagequeue

e00856d

remove

9e70eb6

regen with new config

3541133

version

eaa2215

annatisch reviewed Apr 15, 2026

View reviewed changes

l0lawrence and others added 7 commits April 16, 2026 10:03

update models patch

3a18e32

updating client patches

8afa3a5

blob shared folder spread

9bfdf74

Fix azure.storage.blob rtype references to azure.storage.queue in que…

19e13ba

…ue _shared/models.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

model update

b16013d

Merge branch 'storagequeue' of https://github.com/l0lawrence/azure-sd…

6c52d14

…k-for-python into storagequeue

black

238c30f

	version = kwargs.pop("version", "2026-06-06")
	version = kwargs.pop("version", "2026-04-06")

		from .._utils.model_base import Model as _Model, rest_field, _MyMutableMapping, _RestField


		def _patched_getattr(self, name):

		object.__setattr__(self, name, value)


		def _patched_getattribute(self, name):

Conversation

l0lawrence commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

l0lawrence commented Mar 18, 2026

Uh oh!

azure-pipelines Bot commented Mar 18, 2026

Uh oh!

l0lawrence commented Mar 19, 2026

Uh oh!

azure-pipelines Bot commented Mar 19, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

l0lawrence commented Apr 7, 2026

Uh oh!

azure-pipelines Bot commented Apr 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

l0lawrence commented Apr 22, 2026

Uh oh!

azure-pipelines Bot commented Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

l0lawrence commented Mar 11, 2026 •

edited

Loading