Semi structured v2 #32

ilmarkov · 2024-11-13T11:55:35Z

Add cusparseLt semi structured matmul wrappers.

Supports fp16, bf16, int8 and fp8 data types.

Benchmark results

Results single layer matmul benchmarks for layers sizes of Llama-2-7b-hf-TP1:

size	cutlass_i8_i8_matmul_scaled	cusparseLt_i8_i8_2_4	cusparseLt_i8_i8_2_4_scaled	cutlass_fp8_fp8_matmul_scaled	cusparseLt_fp8_fp8_2_4	cusparseLt_fp8_fp8_2_4_scaled
MKN=(1024x4096x12288)	135.8	53.9	60.4	121.9	57.2	108.9
MKN=(1024x4096x4096)	62.6	21.1	26.1	58.8	21.5	44.4
MKN=(1024x4096x22016)	227.9	89.8	96.3	195.9	100.3	186.4
MKN=(1024x11008x4096)	104.5	45.2	50.8	99.7	49.1	68.1

WIP, end-to-end fp8 model returns gibberish.

PR Checklist (Click to Expand)

Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

[Bugfix] for bug fixes.
[CI/Build] for build or continuous integration improvements.
[Doc] for documentation fixes and improvements.
[Model] for adding a new model or improving an existing model. Model name should appear in the title.
[Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
[Kernel] for changes affecting CUDA kernels or other compute kernels.
[Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
[Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
[Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

We adhere to Google Python style guide and Google C++ style guide.
Pass all linter checks. Please use format.sh to format your code.
The code need to be well-documented to ensure future contributors can easily understand the code.
Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Adding or changing kernels

Each custom kernel needs a schema and one or more implementations to be registered with PyTorch.

Make sure custom ops are registered following PyTorch guidelines: Custom C++ and CUDA Operators and The Custom Operators Manual
Custom operations that return Tensors require meta-functions. Meta-functions should be implemented and registered in python so that dynamic dims can be handled automatically. See above documents for a description of meta-functions.
Use torch.libary.opcheck() to test the function registration and meta-function for any registered ops. See tests/kernels for examples.
When changing the C++ signature of an existing op, the schema must be updated to reflect the changes.
If a new custom type is needed, see the following document: Custom Class Support in PT2.

Notes for Large Changes

Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.

What to Expect for the Reviews

The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
After the review, the reviewer will put an action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You

Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

Signed-off-by: youkaichao <[email protected]>

Signed-off-by: Woosuk Kwon <[email protected]>

Signed-off-by: youkaichao <[email protected]>

Signed-off-by: Woosuk Kwon <[email protected]> Co-authored-by: Roger Wang <[email protected]>

Signed-off-by: Dipika <[email protected]>

Signed-off-by: Xin Yang <[email protected]>

…age embeddings input with varied resolutions (vllm-project#10221) Signed-off-by: imkero <[email protected]>

…ht/dse-qwen2-2b-mrl-v1 (vllm-project#9944) Signed-off-by: FurtherAI <[email protected]> Co-authored-by: FurtherAI <[email protected]>

…t#10282)

Signed-off-by: B-201 <[email protected]>

) Signed-off-by: Roger Wang <[email protected]>

Removed cmake check for cusparseLt, needs to be reverted when the cmake issue is resolved.

github-actions · 2024-11-13T11:55:49Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

bnellnm · 2024-12-02T20:08:09Z

vllm/_custom_ops.py

+def semi_structured_fp8_compress(input: torch.Tensor) -> torch.Tensor:
+    assert input.dtype == torch.float8_e4m3fn
+    return torch.ops._C.cslt_compress_fp8_semi_structured(input)
+
+
+def semi_structured_mm(
+        A_compressed: torch.Tensor,
+        B_dense: torch.Tensor,
+        scale: Optional[float] = None,
+        bias: Optional[torch.Tensor] = None,
+        out_dtype: Optional[torch.dtype] = None) -> torch.Tensor:
+    return torch.ops._C.cslt_mm_semi_structured(A_compressed, B_dense, scale,
+                                                bias, out_dtype)
+
+
+def semi_structured_mm2(
+        A_compressed: torch.Tensor,
+        B_dense: torch.Tensor,
+        scale: Optional[float] = None,
+        bias: Optional[torch.Tensor] = None,
+        out_dtype: Optional[torch.dtype] = None) -> torch.Tensor:
+    return torch.ops._C.cslt_mm_semi_structured2(A_compressed, B_dense, scale,
+                                                 bias, out_dtype)


Please make sure to add fake/meta functions for these ops (see @register_fake in this file for examples). Some opcheck tests would be good as well. See tests/kernels/test_marlin_gemm.py for an example of an opcheck test.

Let me know if you need any help with either of these.

bnellnm

It looks like some of these changes are from upstream and not related to your PR? I think you need to change your upstream to vllm/main (not neuralmagic/vllm/main) so that it is more obvious what changes are coming from this PR.

bnellnm · 2024-12-02T20:09:39Z

csrc/torch_bindings.cpp

+  ops.def("cslt_compress_fp8_semi_structured(Tensor! input) -> Tensor");
+  ops.impl("cslt_compress_fp8_semi_structured", torch::kCUDA,
+           &cslt_compress_fp8_semi_structured);
+
+  ops.def(
+      "cslt_mm_semi_structured(Tensor! compressed_A, Tensor! denseB,"
+      "Tensor!? scale, Tensor!? bias, ScalarType!? output_dtype) -> Tensor");
+  ops.impl("cslt_mm_semi_structured", torch::kCUDA, &cslt_mm_semi_structured);
+
+  ops.def(
+      "cslt_mm_semi_structured2(Tensor! compressed_A, Tensor! denseB,"
+      "Tensor!? scale, Tensor!? bias, ScalarType!? output_dtype) -> Tensor");
+  ops.impl("cslt_mm_semi_structured2", torch::kCUDA, &cslt_mm_semi_structured2);


do these ops all modify their all inputs?

bnellnm · 2024-12-02T20:11:07Z

csrc/torch_bindings.cpp

+  ops.def("cslt_clear_cache() -> ()");
+  ops.impl("cslt_clear_cache", &cslt_clear_cache);


Since this op takes no arguments and doesn't return anything you can just do: ops.def("cslt_clear_cache", &cslt_clear_cache);

bnellnm · 2024-12-02T20:29:14Z

csrc/quantization/fp8_semi_structured/cusparseLt.cpp

+using cacheID = std::tuple<int64_t, int64_t, int64_t, at::ScalarType,
+                           at::ScalarType, int, int, int>;
+
+std::map<cacheID, cusparseLtEntry> cusparseLt_cache;


Ideally the cache entries could be turned into pytorch custom objects but those are not currently supported by the inductor.

We might also be able to subclass torch.Tensor (see https://pytorch.org/docs/main/notes/extending.html#subclassing-torch-tensor) and add an extra opaque field that contains the cache entry. Or use a Tensor for storage of all the cache entry fields. This might be tricky to get right with torch.compile so it might be worth investigating as a follow up.

youkaichao and others added 30 commits November 12, 2024 14:34

[doc] fix location of runllm widget (vllm-project#10266)

96ae0ea

Signed-off-by: youkaichao <[email protected]>

[doc] improve debugging doc (vllm-project#10270)

1808145

Signed-off-by: youkaichao <[email protected]>

Revert "[ci][build] limit cmake version" (vllm-project#10271)

377b74f

[V1] Fix CI tests on V1 engine (vllm-project#10272)

112fa0b

Signed-off-by: Woosuk Kwon <[email protected]>

[core][distributed] use tcp store directly (vllm-project#10275)

0d4ea3f

Signed-off-by: youkaichao <[email protected]>

[V1] Support VLMs with fine-grained scheduling (vllm-project#9871)

bbd3e86

Signed-off-by: Woosuk Kwon <[email protected]> Co-authored-by: Roger Wang <[email protected]>

Bump to compressed-tensors v0.8.0 (vllm-project#10279)

56a955e

Signed-off-by: Dipika <[email protected]>

[Doc] Fix typo in arg_utils.py (vllm-project#10264)

032fcf1

Signed-off-by: Xin Yang <[email protected]>

[Model] Add support for Qwen2-VL video embeddings input & multiple im…

3945c82

…age embeddings input with varied resolutions (vllm-project#10221) Signed-off-by: imkero <[email protected]>

[Model] Adding Support for Qwen2VL as an Embedding Model. Using MrLig…

1b886aa

…ht/dse-qwen2-2b-mrl-v1 (vllm-project#9944) Signed-off-by: FurtherAI <[email protected]> Co-authored-by: FurtherAI <[email protected]>

[Core] Flashinfer - Remove advance step size restriction (vllm-projec…

b6dde33

…t#10282)

[Model][LoRA]LoRA support added for idefics3 (vllm-project#10281)

d909acf

Signed-off-by: B-201 <[email protected]>

[V1] Add missing tokenizer options for Detokenizer (vllm-project#10288

bb7991a

) Signed-off-by: Roger Wang <[email protected]>

semi_structured for fp16 and bf16 and int8

78eea7b

Fix A100 int8 tests

331e9c5

Add fp8 cusparseLt

381a6b4

wip

b146a79

Fix signatures

0ac01cc

Fix compilation and tests

7472af2

Update for older platforms

3fe8bd4

Add benchmarks

e736027

Fix typo

ae66f77

Added scaled_mm for fp8.

59ee24d

Removed cmake check for cusparseLt, needs to be reverted when the cmake issue is resolved.

Add docstrings

3367704

Update for torch 2.5

368beec

Add handling contiguous dense input for int8 and fp8

5be53f3

Add fp8 cusparseLt

f9546a8

Fix compilation and tests

2187236

Add caching of cusparseLT meta

f45a83b

Cached cusparseLt

b1aaea5

ilmarkov added 8 commits November 13, 2024 11:30

Fix destroy function

c36401c

Prepare for reproduce

9f6a469

Fix cusparseLt caching

2e56de9

Make cached version default function

9ad83cb

Fixes and polishing after rebase

1f6a05b

Add output_dtype option, fix non-padded inputs case

3a2c258

Fix and polish

31cf482

Formatting

68512d4

ilmarkov requested a review from bnellnm November 13, 2024 11:55

ilmarkov requested a review from tlrmchlsmth as a code owner November 13, 2024 11:55

Minor test and benchmarks updates

72d6cd3

bnellnm reviewed Dec 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Semi structured v2 #32

Semi structured v2 #32

ilmarkov commented Nov 13, 2024

github-actions bot commented Nov 13, 2024

bnellnm Dec 2, 2024

bnellnm left a comment

bnellnm Dec 2, 2024

bnellnm Dec 2, 2024

bnellnm Dec 2, 2024 •

edited

Loading

		ops.def("cslt_clear_cache() -> ()");
		ops.impl("cslt_clear_cache", &cslt_clear_cache);

Semi structured v2 #32

Are you sure you want to change the base?

Semi structured v2 #32

Conversation

ilmarkov commented Nov 13, 2024

PR Title and Classification

Code Quality

Adding or changing kernels

Notes for Large Changes

What to Expect for the Reviews

Thank You

github-actions bot commented Nov 13, 2024

bnellnm Dec 2, 2024

Choose a reason for hiding this comment

bnellnm left a comment

Choose a reason for hiding this comment

bnellnm Dec 2, 2024

Choose a reason for hiding this comment

bnellnm Dec 2, 2024

Choose a reason for hiding this comment

bnellnm Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

bnellnm Dec 2, 2024 •

edited

Loading