Discussions

• 3 You must be logged in to vote

  💡

  Future plan?

  Katehuuh started Apr 22, 2024 in Ideas

  

  

  

  

  9
• 1 You must be logged in to vote

  🙏

  Question on understanding how model.forward factors in temperature?

  efawe asked Oct 31, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  Priority for MllamaForConditionalGeneration Support?

  gros87 started Sep 30, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  Question about dequantization

  bug Something isn't working

  HaoWeiWang asked Sep 25, 2024 in Q&A · Unanswered

  

  

  1
• 2 You must be logged in to vote

  💬

  Phi 3.5 MOE and supporting MOE in general

  niceblue88 started Sep 24, 2024 in General

  

  

  1
• 1 You must be logged in to vote

  💡

  Injecting noise in hidden state inputs, query, key/value or attention head outputs

  xonfour started Jun 10, 2024 in Ideas

  

  

  

  

  31
• 1 You must be logged in to vote

  💬

  Optimal Quantizing

  GHBigD started Aug 16, 2024 in General

  

  

  2
• 1 You must be logged in to vote

  💡

  Liger kernels and exllamav2?

  jim-plus started Aug 24, 2024 in Ideas

  

  

  0
• 1 You must be logged in to vote

  💬

  Addressing a skill issue getting CUDA 12.1 to support the latest MSVC under Windows when building current exllamav2

  jim-plus started Aug 13, 2024 in General

  

  

  2
• 0 You must be logged in to vote

  💬

  Why Doesn't exllamav2 Produce Consistent Output with a Set Seed?

  GrennKren started Aug 7, 2024 in General · Closed

  

  

  0
• 1 You must be logged in to vote

  💬

  Measurement/inference error (3): hidden_states

  Katehuuh started Jul 30, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  💡

  Model Pruning options during quantization

  gros87 started Jul 29, 2024 in Ideas

  

  

  2
• 1 You must be logged in to vote

  💬

  Python libraries & slow inference with large context windows for v.0.1.8 vs 0.1.7

  arthurmnev started Jul 25, 2024 in General

  

  

  4
• 1 You must be logged in to vote

  💡

  🤗 Space to Quants EXL2?

  Katehuuh started Jul 24, 2024 in Ideas · Closed

  

  

  1
• 1 You must be logged in to vote

  🙏

  Performance on AMD (gfx1030)?

  jeroen-mostert asked Jul 22, 2024 in Q&A · Unanswered

  

  

  1
• 3 You must be logged in to vote

  💬

  Gemma-2 architecture support

  arbi-dev started Jun 30, 2024 in General

  

  

  

  

  4
• 1 You must be logged in to vote

  💬

  Please update the M40 graphics card mirroring kernel please, really need, I do not know how to use github, if you see the trouble to help me contact the developer, thank you

  lugangqi started Jun 29, 2024 in General

  

  

  

  

  8
• 3 You must be logged in to vote

  💬

  Barriers to Wider exllamav2 adoption

  gittb started Jun 18, 2024 in General

  

  

  

  

  8
• 2 You must be logged in to vote

  🙏

  Qwen 2 inference problem

  Sadeghi85 asked Jun 6, 2024 in Q&A · Unanswered

  

  

  1
• 5 You must be logged in to vote

  🙌

  Here's my batching/caching API I made over the weekend. 200+ tk/s with Mistral 5.0bpw.exl2 on an RTX 3090 with concurrent requests. It was for a personal project, and it's not complete, but it's very fast.

  epolewski started Dec 26, 2023 in Show and tell

  

  

  1
• 1 You must be logged in to vote

  💬

  IBM's Granite 20B Code Instruct goes off the rails when using either: FP8, Q4 cache or speculative decoding (n-gram).

  LlamaEnjoyer started May 30, 2024 in General

  

  

  4
• 1 You must be logged in to vote

  💬

  Inference speed exl2 vs gguf - are my results typical?

  LlamaEnjoyer started May 27, 2024 in General · Closed

  

  

  1
• 2 You must be logged in to vote

  💬

  Q4 Cache

  Vhallo started May 29, 2024 in General

  

  

  2
• 1 You must be logged in to vote

  💬

  Flash attention 2 on Turing / Triton?

  dvianisoho started May 27, 2024 in General

  

  

  2
• 1 You must be logged in to vote

  💬

  exllamav2 and IBM Granite models

  RaRasputinRGLM started May 20, 2024 in General · Closed

  

  

  7