Skip to content

Performance issue regarding Sagemaker Estimator.deploy() #1006

Closed
Closed
Performance issue regarding Sagemaker Estimator.deploy()#1006
Labels
Pending informationtype: question

Description

@UniSabina
UniSabina
opened on Aug 28, 2019

System Information

  • Python Version: 3.6
  • CPU or GPU: CPU
  • Python SDK Version: Anaconda

Describe the problem

I'm quite new to the SageMaker algorithms and estimators so please bear with me.
I'm running a script very similar to this example script for DeepAR
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/deepar_electricity/DeepAR-Electricity.ipynb
And want to start more than hundred of such training + prediction jobs.
The cell

predictor = estimator.deploy(
    initial_instance_count=1,
    instance_type='ml.m4.xlarge',
    predictor_cls=DeepARPredictor)

takes up 70% (~8.5min) of the time of the overall training and predicting job (~12min). Is there a possibility to reduce that time? What is the reason for this deploy job taking so long?

Thanks!

Activity

  1. ChoiByungWook commented on Aug 30, 2019

    @ChoiByungWook
    ChoiByungWook
    on Aug 30, 2019
    Contributor

    Hello @UniSabina,

    Apologies on the slow experience.

    I'll reach out to the team that handles the hosting platform and let them comment.

  2. ChoiByungWook
    added
    type: feature request
    type: question
    on Aug 30, 2019

  3. vedal commented on Dec 16, 2019

    @vedal
    vedal
    on Dec 16, 2019

    I'm also interested in this

  4. laurenyu
    added
    Pending information
    and removed
    type: feature request
    on Feb 19, 2020
  5. laurenyu
    added
    NON-PY-SDK
    on May 8, 2020

  6. lexxx233 commented on Sep 9, 2020

    @lexxx233
    lexxx233
    on Sep 9, 2020

    Any update on this?

  7. rishabh1212 commented on Nov 12, 2020

    @rishabh1212
    rishabh1212
    on Nov 12, 2020

    Hii @ChoiByungWook,
    Any update on this?

  8. nrapanos commented on Jan 5, 2021

    @nrapanos
    nrapanos
    on Jan 5, 2021

    Is there any update on this? I'm experiencing a similar issue.

  9. dsow86 commented on Feb 23, 2021

    @dsow86
    dsow86
    on Feb 23, 2021

    Any update about this issue? I am still having the same problem, the model deployment take too long.

  10. zlufy90 commented on Apr 22, 2021

    @zlufy90
    zlufy90
    on Apr 22, 2021

    It is too slow for training and deploying model via SageMaker Studio. I just tested with Iris dataset.
    Especially when debugging. It makes us give up easily.

  11. bveeramani
    closed this as completedon May 19, 2021
  12. aws
    locked and limited conversation to collaborators on May 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Pending informationtype: question

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions