-
Notifications
You must be signed in to change notification settings - Fork 188
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add dashboard metric support for sam and sam2 benchmarks (#1407)
* Add dashboard metric support for sam and sam2 benchmarks
Summary:
Also makes the `.github/workflows/dashboard_perf_test.yml` more complete by adding runs for both compile and autoquant-all
for llama, sam, sam2
Test Plan:
tested locally:
```
SEGMENT_ANYTHING_FAST_USE_FLASH_4=0 python eval_combo.py --coco_root_dir datasets/coco2017 --coco_slice_name val2017 --sam_checkpoint_base_path checkpoints --sam_model_type vit_h --point_sampling_cache_dir tmp/sam_coco_mask_center_cache --mask_debug_out_dir tmp/sam_eval_masks_out --batch_size 32 --num_workers 32 --use_compile max-autotune --use_half bfloat16 --device cuda --output_json_path output
cat output.csv
{"benchmark": {"name": "TorchAO benchmark", "mode": "inference", "dtype": "torch.bfloat16", "extra_info": {"device": "cuda", "arch": "NVIDIA H100"}}, "model": {"name": "vit_h", "type": "model", "origins": ["pytorch"]}, "metric": {"name": "memory(MiB)", "benchmark_values": [27740], "target_value": null}}
{"benchmark": {"name": "TorchAO benchmark", "mode": "inference", "dtype": "torch.bfloat16", "extra_info": {"device": "cuda", "arch": "NVIDIA H100"}}, "model": {"name": "vit_h", "type": "model", "origins": ["pytorch"]}, "metric": {"name": "img_s(avg)", "benchmark_values": [42.65887304183299], "target_value": null}}
python server.py ~/checkpoints/sam2 large --port 4000 --host localhost --use_autoquant --benchmark --output_json_path output
{"benchmark": {"name": "TorchAO benchmark", "mode": "inference", "dtype": "autoquant", "extra_info": {"device": "cuda", "arch": "NVIDIA H100"}}, "model": {"name": "sam2-large", "type": "model", "origins": ["pytorch"]}, "metric": {"name": "memory(MiB)", "benchmark_values": [4294], "target_value": null}}
{"benchmark": {"name": "TorchAO benchmark", "mode": "inference", "dtype": "autoquant", "extra_info": {"device": "cuda", "arch": "NVIDIA H100"}}, "model": {"name": "sam2-large", "type": "model", "origins": ["pytorch"]}, "metric": {"name": "memory(%)", "benchmark_values": [4], "target_value": null}}
{"benchmark": {"name": "TorchAO benchmark", "mode": "inference", "dtype": "autoquant", "extra_info": {"device": "cuda", "arch": "NVIDIA H100"}}, "model": {"name": "sam2-large", "type": "model", "origins": ["pytorch"]}, "metric": {"name": "time_s(avg)", "benchmark_values": [0.4720584869384766], "target_value": null}}
```
Also will check CI and click house
Reviewers:
Subscribers:
Tasks:
Tags:
* ruff
* add util file
* add scripts
* tiktoken
* dep
* lm_eval
* use default autoquant for now
* path for sam eval
* add conda_run
* more deps
* invalid requirement
* git dep
* include subgraph_utils folder in installation
* reduce num_workers to 12
* try sam again with 8 workers
* skipping sam
* fix sam2 path
* checking path
* add sam2 config for ci
* skip sam2
* remove config- Loading branch information
1 parent
ace7219
commit cbd7c29
Showing
14 changed files
with
227 additions
and
85 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Copyright (c) Meta Platforms, Inc. and affiliates. | ||
| # All rights reserved. | ||
|
|
||
| # This source code is licensed under the license found in the | ||
| # LICENSE file in the root directory of this source tree. | ||
|
|
||
| # Use either wget or curl to download the checkpoints | ||
| if command -v wget &> /dev/null; then | ||
| CMD="wget -P" | ||
| elif command -v curl &> /dev/null; then | ||
| CMD="curl -L -O" | ||
| else | ||
| echo "Please install wget or curl to download the checkpoints." | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Define the URLs for SAM 2 checkpoints | ||
| # SAM2_BASE_URL="https://dl.fbaipublicfiles.com/segment_anything_2/072824" | ||
| # sam2_hiera_t_url="${SAM2_BASE_URL}/sam2_hiera_tiny.pt" | ||
| # sam2_hiera_s_url="${SAM2_BASE_URL}/sam2_hiera_small.pt" | ||
| # sam2_hiera_b_plus_url="${SAM2_BASE_URL}/sam2_hiera_base_plus.pt" | ||
| # sam2_hiera_l_url="${SAM2_BASE_URL}/sam2_hiera_large.pt" | ||
|
|
||
| # Download each of the four checkpoints using wget | ||
| # echo "Downloading sam2_hiera_tiny.pt checkpoint..." | ||
| # $CMD $sam2_hiera_t_url || { echo "Failed to download checkpoint from $sam2_hiera_t_url"; exit 1; } | ||
|
|
||
| # echo "Downloading sam2_hiera_small.pt checkpoint..." | ||
| # $CMD $sam2_hiera_s_url || { echo "Failed to download checkpoint from $sam2_hiera_s_url"; exit 1; } | ||
|
|
||
| # echo "Downloading sam2_hiera_base_plus.pt checkpoint..." | ||
| # $CMD $sam2_hiera_b_plus_url || { echo "Failed to download checkpoint from $sam2_hiera_b_plus_url"; exit 1; } | ||
|
|
||
| # echo "Downloading sam2_hiera_large.pt checkpoint..." | ||
| # $CMD $sam2_hiera_l_url || { echo "Failed to download checkpoint from $sam2_hiera_l_url"; exit 1; } | ||
|
|
||
| # Define the URLs for SAM 2.1 checkpoints | ||
| SAM2p1_BASE_URL="https://dl.fbaipublicfiles.com/segment_anything_2/092824" | ||
| sam2p1_hiera_t_url="${SAM2p1_BASE_URL}/sam2.1_hiera_tiny.pt" | ||
| sam2p1_hiera_s_url="${SAM2p1_BASE_URL}/sam2.1_hiera_small.pt" | ||
| sam2p1_hiera_b_plus_url="${SAM2p1_BASE_URL}/sam2.1_hiera_base_plus.pt" | ||
| sam2p1_hiera_l_url="${SAM2p1_BASE_URL}/sam2.1_hiera_large.pt" | ||
|
|
||
| # $1 is the directory to store the checkpoint | ||
| DEFAULT_DIR=test | ||
| if [ -z "$1" ]; then | ||
| DIR_NAME=$DEFAULT_DIR | ||
| else | ||
| # Use provided directory name | ||
| DIR_NAME=$1 | ||
| fi | ||
|
|
||
| # SAM 2.1 checkpoints | ||
| echo "Downloading sam2.1_hiera_tiny.pt checkpoint..." | ||
| $CMD $DIR_NAME $sam2p1_hiera_t_url || { echo "Failed to download checkpoint from $sam2p1_hiera_t_url"; exit 1; } | ||
|
|
||
| echo "Downloading sam2.1_hiera_small.pt checkpoint..." | ||
| $CMD $DIR_NAME $sam2p1_hiera_s_url || { echo "Failed to download checkpoint from $sam2p1_hiera_s_url"; exit 1; } | ||
|
|
||
| echo "Downloading sam2.1_hiera_base_plus.pt checkpoint..." | ||
| $CMD $DIR_NAME $sam2p1_hiera_b_plus_url || { echo "Failed to download checkpoint from $sam2p1_hiera_b_plus_url"; exit 1; } | ||
|
|
||
| echo "Downloading sam2.1_hiera_large.pt checkpoint..." | ||
| $CMD $DIR_NAME $sam2p1_hiera_l_url || { echo "Failed to download checkpoint from $sam2p1_hiera_l_url"; exit 1; } | ||
|
|
||
| echo "All checkpoints are downloaded successfully." |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| ruff check . --fix | ||
| # --isolated is used to skip the allowlist at all so this applies to all files | ||
| # please be careful when using this large changes means everyone needs to rebase | ||
| ruff check --isolated --select F821,F823,W191 --fix | ||
| ruff check --select F,I --fix | ||
| ruff format . |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.