feat: jumpstart model id suggestions by evakravi · Pull Request #2899 · aws/sagemaker-python-sdk

evakravi · Feb 4, 2022

Issue #, if available:

Description of changes:
This PR improves the error message when a bad model id is inputted for JumpStart utilities. If the model id does not exist for any versions, the error message indicates a closest match model id which can be used.

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov-commenter · Feb 4, 2022

Codecov Report

Merging #2899 (b0a761b) into dev (21ac6fd) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##              dev    #2899   +/-   ##
=======================================
  Coverage   89.47%   89.48%           
=======================================
  Files         196      196           
  Lines       16584    16592    +8     
=======================================
+ Hits        14839    14847    +8     
  Misses       1745     1745

Impacted Files	Coverage Δ
src/sagemaker/jumpstart/validators.py	`91.86% <ø> (ø)`
src/sagemaker/jumpstart/cache.py	`96.58% <100.00%> (+0.25%)`	⬆️
src/sagemaker/jumpstart/types.py	`94.33% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 21ac6fd...b0a761b. Read the comment docs.

sagemaker-bot · Feb 4, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 049a57c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 4, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 049a57c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 4, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 049a57c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

JGuinegagne

Minor comments/doc changes.

Also a question on how many model IDs suggestion do we surface.

JGuinegagne · Feb 7, 2022

src/sagemaker/jumpstart/cache.py

+        )
+        if other_model_id_version is not None:
+            error_msg += (
+                f"Consider using model id {model_id} with version "


"Consider using model ID '{model_id}' with version '{other_model_id_version}'."

JGuinegagne · Feb 7, 2022

src/sagemaker/jumpstart/cache.py

+
+        else:
+            possible_model_ids = [header.model_id for header in manifest.values()]
+            closest_model_id = get_close_matches(model_id, possible_model_ids, n=1, cutoff=0)[0]


question: should you show only one? or say for example: at least 1, up to 3 when score > xx ?

Let's just do 1 and make it simple. If we want to make this really good, we're better off just having a separate utility for searching model ids.

another question, can you get an IndexError here if get_close_matches returns an empty list?

If that's possible, could you handle this edge case and add a unit test for it?

No, because the cutoff is 0, there will always be a match.

Thanks. I know this is super edge case, but could there be a case where possible_model_ids is an empty list?

Only if the manifest is empty

JGuinegagne · Feb 7, 2022

src/sagemaker/jumpstart/cache.py

+        error_msg = f"Unable to find model manifest for {model_id} with version {version}. "
+
+        other_model_id_version = self._select_version(
+            "*", versions_incompatible_with_sagemaker


can you please add a comment as to why you use this variable.

reiterate: please explain why we use a variable called versions_incompatitble_with_sagemaker here.

JGuinegagne · Feb 7, 2022

src/sagemaker/jumpstart/cache.py

 """This module defines the JumpStartModelsCache class."""
 from __future__ import absolute_import
 import datetime
+from difflib import get_close_matches


sagemaker-bot · Feb 7, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 4aa1670
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 7, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 4aa1670
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 7, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 4aa1670
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 8, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: d66432e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 8, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: d66432e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 8, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: d66432e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 9, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c51d9bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 9, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c51d9bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 9, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c51d9bd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 15, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2e017c4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 15, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2e017c4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 15, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2e017c4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 16, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: dded640
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 16, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5150498
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 16, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dded640
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 16, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5150498
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 16, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5150498
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 16, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: dded640
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 16, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: dded640
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 16, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5370f76
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 16, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5370f76
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 17, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5370f76
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 18, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5370f76
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 18, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5370f76
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 18, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b0a761b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 18, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b0a761b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 18, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b0a761b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · Feb 18, 2022

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b0a761b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Navin Soni <navinsoni89@gmail.com> Co-authored-by: Mufaddal Rohawala <89424143+mufaddal-rohawala@users.noreply.github.com> Co-authored-by: qidewenwhen <32910701+qidewenwhen@users.noreply.github.com> Co-authored-by: HappyAmazonian <91216626+HappyAmazonian@users.noreply.github.com>

mufaddal-rohawala force-pushed the dev branch 2 times, most recently from 96c01d7 to c437191 Compare February 5, 2022 00:08

JGuinegagne requested changes Feb 7, 2022

View reviewed changes

evakravi force-pushed the feat/jumpstart-model-id-suggestions branch from 049a57c to 4aa1670 Compare February 7, 2022 21:25

jeniyat force-pushed the dev branch from 9d612a7 to bb7563f Compare February 7, 2022 23:40

evakravi force-pushed the feat/jumpstart-model-id-suggestions branch from 4aa1670 to d66432e Compare February 8, 2022 17:21

jeniyat force-pushed the dev branch from 7ea4b80 to bb7563f Compare February 9, 2022 00:11

evakravi added 2 commits February 9, 2022 13:33

feat: jumpstart model id suggestions

598562b

chore: resolve git comments

c51d9bd

evakravi force-pushed the feat/jumpstart-model-id-suggestions branch from d66432e to c51d9bd Compare February 9, 2022 13:34

navinsoni and others added 2 commits February 15, 2022 00:30

fix: Update Static Endpoint (aws#2931)

b591959

fix: hyperparameter validation error messages

2e017c4

navinsoni and others added 2 commits February 15, 2022 18:27

Add exception in test_action (aws#2938)

3c5ea3a

change: pin test dependencies (aws#2929)

9d84e2e

qidewenwhen and others added 4 commits February 15, 2022 21:30

feature: Add FailStep Support for Sagemaker Pipeline (aws#2872)

cede5fa

change: use recommended inference image uri from Neo API (aws#2923)

a765512

change: add url to model id table for jumpstart

5150498

Merge branch 'dev' into feat/jumpstart-model-id-suggestions

dded640

JGuinegagne approved these changes Feb 16, 2022

View reviewed changes

navinsoni force-pushed the dev branch from 58cf626 to 46391bf Compare February 16, 2022 19:55

Merge branch 'dev' into feat/jumpstart-model-id-suggestions

5370f76

shreyapandit approved these changes Feb 17, 2022

View reviewed changes

navinsoni approved these changes Feb 18, 2022

View reviewed changes

Merge branch 'dev' into feat/jumpstart-model-id-suggestions

b0a761b

navinsoni merged commit 09d3fd1 into aws:dev Feb 18, 2022

Search code, repositories, users, issues, pull requests...

Conversation

evakravi commented Feb 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

codecov-commenter commented Feb 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sagemaker-bot commented Feb 4, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 4, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 4, 2022

AWS CodeBuild CI Report

Uh oh!

JGuinegagne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Feb 7, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 7, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 7, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 8, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 8, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 8, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 9, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 15, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 15, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Feb 15, 2022

AWS CodeBuild CI Report

Uh oh!

evakravi commented Feb 4, 2022 •

edited

Loading

codecov-commenter commented Feb 4, 2022 •

edited

Loading