Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gen AI monitoring is broken due to Marshmallow update #39075

Closed
prise6 opened this issue Jan 8, 2025 · 4 comments · Fixed by #39053
Closed

Gen AI monitoring is broken due to Marshmallow update #39075

prise6 opened this issue Jan 8, 2025 · 4 comments · Fixed by #39053
Assignees
Labels
AI customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.

Comments

@prise6
Copy link

prise6 commented Jan 8, 2025

Hello,
Related to #39039

When configuring monitoring, like this example in documentation.

Job failed with cannot import name '_T' from 'marshmallow.fields' (/home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/marshmallow/fields.py)

When digging in job log

# in logs/azureml/driver/librarymanager/*.log

# [...]
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.10
  - pip:
      - azure-cli-core~=2.45.0
      - azure-identity~=1.12.0
      - azure-ai-ml~=1.10.0
      - azure-keyvault-secrets==4.7.0
      - tiktoken==0.3.3
      - json5==0.9.11
      - scipy~=1.10.0
      - numpy~=1.21.0
      - pandas~=1.4.3
      - azureml-mlflow~=1.49.0
      - mltable~=1.3.0

# [...]

Collecting marshmallow<4.0.0,>=3.5 (from azure-ai-ml~=1.10.0->-r /usr/lib/library-manager/bin/condaenv.s2dyrgt_.requirements.txt (line 3))
  Downloading marshmallow-3.24.1-py3-none-any.whl.metadata (7.1 kB

Correct me if 'im wrong but i can't force the azure-ai-ml version in this component, so

  • is there a workaround ?
  • is it possible to update dependencies for this component ?

Thank you,

@kingernupur

@github-actions github-actions bot added customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jan 8, 2025
@xiangyan99 xiangyan99 added Service Attention Workflow: This issue is responsible by Azure service team. AI Machine Learning and removed needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. labels Jan 8, 2025
@github-actions github-actions bot added the needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team label Jan 8, 2025
Copy link

github-actions bot commented Jan 8, 2025

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @achauhan-scc @kingernupur @luigiw @needuv @paulshealy1 @singankit.

@prise6
Copy link
Author

prise6 commented Jan 14, 2025

Hello again,

Error still persists. Do you have planned actions about this error ?

Image

Thank you,

@achauhan-scc
Copy link
Member

issue is fixed 1.23.1

@prise6
Copy link
Author

prise6 commented Jan 20, 2025

I understand that you fixed the marsmallow issue in the python sdk, but AzureML Monitoring is not fixed because the component version is not updated.

So i asked to your team in another repo: Azure/azureml-assets#3775

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AI customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants