h2o for kv cache compression #1468

n1ck-guo · 2024-04-10T07:37:20Z

Type of Change

feature

Description

H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
paper

NTD

Expected Behavior & Potential Risk

None

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

Signed-off-by: n1ck-guo <[email protected]>

github-actions · 2024-04-10T07:37:44Z

⛈️ Required checks status: Has failure 🔴

Warning
If you do not have the access to re-run the CI-Summary bot, please contact VincyZhang for help. If you push a new commit, all of the workflow will be re-triggered.

Groups summary

🟢 Format Scan Tests workflow

Check ID	Status
format-scan (pylint)	success	✅
format-scan (bandit)	success	✅
format-scan (cloc)	success	✅
format-scan (cpplint)	success	✅

These checks are required after the changes to intel_extension_for_transformers/transformers/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_gaudi_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/base.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/h2o.py.

🔴 Optimize Unit Test workflow

Check ID	Status	Error details
optimize-unit-test-baseline	success		✅
optimize-unit-test-PR-test	success		✅
Genreate-OptimizeUT-Report	failure	download	❌

These checks are required after the changes to intel_extension_for_transformers/transformers/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_gaudi_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/base.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/h2o.py.

🔴 NeuralChat Unit Test

Check ID	Status	Error details
neuralchat-unit-test-baseline	success		✅
neuralchat-unit-test-PR-test	success		✅
Generate-NeuralChat-Report	failure	download	❌

These checks are required after the changes to intel_extension_for_transformers/transformers/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_gaudi_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/base.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/h2o.py.

🟢 Engine Unit Test workflow

Check ID	Status
engine-unit-test-baseline	success	✅
engine-unit-test-PR-test	success	✅
Genreate-Engine-Report	success	✅

These checks are required after the changes to intel_extension_for_transformers/transformers/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_gaudi_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/base.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/h2o.py.

🟡 Chat Bot Test workflow

Check ID	Status	Error details
call-inference-llama-2-7b-chat-hf / inference test	no_status		❓
call-inference-mpt-7b-chat / inference test	no_status		❓

These checks are required after the changes to intel_extension_for_transformers/transformers/kv_cache_compression/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_gaudi_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/models/modeling_llama.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/__init__.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/base.py, intel_extension_for_transformers/transformers/kv_cache_compression/prune/h2o.py.

Thank you for your contribution! 💜

Note
This comment is automatically generated and will be updates every 180 seconds within the next 6 hours. If you have any other questions, contact VincyZhang or XuehaoSun for help.

for more information, see https://pre-commit.ci

...ion_for_transformers/transformers/modeling/kv_cahe_compression/h2o_real_drop/modify_llama.py

...on_for_transformers/transformers/modeling/kv_cahe_compression/h2o_sim_drop/modify_gptneox.py

Signed-off-by: biao.fang <[email protected]>

for more information, see https://pre-commit.ci

intel_extension_for_transformers/transformers/modeling/kv_cache_compression/h2o.py

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

...nsion_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mistral.py

examples/huggingface/pytorch/text-generation/h2o/run_lm_eval_harness.py

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: biao.fang <[email protected]>

Signed-off-by: n1ck-guo <[email protected]>

…n-for-transformers into hengguo/h2o

for more information, see https://pre-commit.ci

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

...tension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_bloom.py

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

intel_extension_for_transformers/transformers/modeling/kv_cache_compression/prune/__init__.py

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: n1ck-guo <[email protected]>

…n-for-transformers into hengguo/h2o

for more information, see https://pre-commit.ci

Signed-off-by: n1ck-guo <[email protected]>

for more information, see https://pre-commit.ci

PenghuiCheng · 2024-07-02T05:27:11Z

Could we add a document introducing what h2o is?

changwangss · 2024-07-02T09:18:00Z

format scan improved by #1647. merged.

h2o for kv cache compression

41d8647

Signed-off-by: n1ck-guo <[email protected]>

n1ck-guo requested review from PenghuiCheng, VincyZhang and wenhuach21 April 10, 2024 07:37

[pre-commit.ci] auto fixes from pre-commit.com hooks

eb7f564

for more information, see https://pre-commit.ci

wenhuach21 reviewed Apr 10, 2024

View reviewed changes

...ion_for_transformers/transformers/modeling/kv_cahe_compression/h2o_real_drop/modify_llama.py Outdated Show resolved Hide resolved

wenhuach21 reviewed Apr 10, 2024

View reviewed changes

...ion_for_transformers/transformers/modeling/kv_cahe_compression/h2o_real_drop/modify_llama.py Outdated Show resolved Hide resolved

wenhuach21 reviewed Apr 10, 2024

View reviewed changes

...on_for_transformers/transformers/modeling/kv_cahe_compression/h2o_sim_drop/modify_gptneox.py Outdated Show resolved Hide resolved

BiaoFangAIA and others added 3 commits April 23, 2024 16:36

rebuild

c46ea7d

Signed-off-by: biao.fang <[email protected]>

merge

95ff9ae

Signed-off-by: biao.fang <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

9d27733

for more information, see https://pre-commit.ci

wenhuach21 reviewed Apr 25, 2024

View reviewed changes

intel_extension_for_transformers/transformers/modeling/kv_cache_compression/h2o.py Outdated Show resolved Hide resolved

n1ck-guo and others added 4 commits April 25, 2024 03:01

update

444490d

Signed-off-by: n1ck-guo <[email protected]>

update

4309089

Signed-off-by: n1ck-guo <[email protected]>

merge

8c5272e

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

1b83e52

for more information, see https://pre-commit.ci

wenhuach21 reviewed Apr 29, 2024

View reviewed changes

...nsion_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_mistral.py Outdated Show resolved Hide resolved

wenhuach21 reviewed Apr 29, 2024

View reviewed changes

examples/huggingface/pytorch/text-generation/h2o/run_lm_eval_harness.py Outdated Show resolved Hide resolved

n1ck-guo and others added 3 commits May 7, 2024 04:26

update

3fd73cb

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

a2d3ae0

for more information, see https://pre-commit.ci

Merge branch 'main' into hengguo/h2o

ddf5445

VincyZhang added the WIP label May 13, 2024

n1ck-guo and others added 7 commits May 14, 2024 01:43

Merge branch 'main' into hengguo/h2o

91d4394

real drop

a83e6d6

Signed-off-by: n1ck-guo <[email protected]>

modify real drop code

92c8a62

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

70a1cf3

for more information, see https://pre-commit.ci

fix

bc9eade

Signed-off-by: biao.fang <[email protected]>

update for real drop and sim mode, using the same api

9aa25f6

Signed-off-by: n1ck-guo <[email protected]>

Merge branch 'main' into hengguo/h2o

03cdc8d

n1ck-guo and others added 9 commits June 6, 2024 03:14

fix acc bug

30bed25

Signed-off-by: n1ck-guo <[email protected]>

fix

f8a64fc

Signed-off-by: n1ck-guo <[email protected]>

Merge branch 'hengguo/h2o' of https://github.com/intel/intel-extensio…

d892e74

…n-for-transformers into hengguo/h2o

[pre-commit.ci] auto fixes from pre-commit.com hooks

7a12ec6

for more information, see https://pre-commit.ci

refactor code

124bb72

Signed-off-by: n1ck-guo <[email protected]>

fix

5181afc

Signed-off-by: n1ck-guo <[email protected]>

merge

93ad39b

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

58bfcd0

for more information, see https://pre-commit.ci

Merge branch 'main' into hengguo/h2o

76656c9

wenhuach21 reviewed Jun 18, 2024

View reviewed changes

...tension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_bloom.py Outdated Show resolved Hide resolved

wenhuach21 reviewed Jun 18, 2024

View reviewed changes

...tension_for_transformers/transformers/modeling/kv_cache_compression/models/modeling_bloom.py Outdated Show resolved Hide resolved

n1ck-guo and others added 2 commits June 20, 2024 03:36

new api

91c5f3c

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

4884b3a

for more information, see https://pre-commit.ci

wenhuach21 reviewed Jun 20, 2024

View reviewed changes

intel_extension_for_transformers/transformers/modeling/kv_cache_compression/prune/__init__.py Outdated Show resolved Hide resolved

n1ck-guo and others added 11 commits June 24, 2024 02:26

support for gaudi

812a838

Signed-off-by: n1ck-guo <[email protected]>

merge

2cc6a8f

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

2d82bb5

for more information, see https://pre-commit.ci

update

a9488b4

Signed-off-by: n1ck-guo <[email protected]>

Merge branch 'hengguo/h2o' of https://github.com/intel/intel-extensio…

3c185ad

…n-for-transformers into hengguo/h2o

[pre-commit.ci] auto fixes from pre-commit.com hooks

a6d3fc7

for more information, see https://pre-commit.ci

pylint

9698230

Signed-off-by: n1ck-guo <[email protected]>

pylint

14f5a6d

Signed-off-by: n1ck-guo <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

dd6ee3c

for more information, see https://pre-commit.ci

Merge branch 'main' into hengguo/h2o

523ca76

Merge branch 'main' into hengguo/h2o

2618e6f

wenhuach21 approved these changes Jul 2, 2024

View reviewed changes

n1ck-guo requested a review from changwangss July 2, 2024 02:49

Merge branch 'main' into hengguo/h2o

cedcd43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

h2o for kv cache compression #1468

h2o for kv cache compression #1468

n1ck-guo commented Apr 10, 2024 •

edited

Loading

github-actions bot commented Apr 10, 2024 •

edited

Loading

PenghuiCheng commented Jul 2, 2024

changwangss commented Jul 2, 2024 •

edited

Loading

h2o for kv cache compression #1468

Are you sure you want to change the base?

h2o for kv cache compression #1468

Conversation

n1ck-guo commented Apr 10, 2024 • edited Loading

Type of Change

Description

Expected Behavior & Potential Risk

How has this PR been tested?

Dependency Change?

github-actions bot commented Apr 10, 2024 • edited Loading

⛈️ Required checks status: Has failure 🔴

Groups summary

PenghuiCheng commented Jul 2, 2024

changwangss commented Jul 2, 2024 • edited Loading

n1ck-guo commented Apr 10, 2024 •

edited

Loading

github-actions bot commented Apr 10, 2024 •

edited

Loading

changwangss commented Jul 2, 2024 •

edited

Loading