Discussions

• 1 You must be logged in to vote

  🙏

  is there a way to run vllm without torch.compiled model?

  carlesoctav announced Dec 10, 2024 in Q&A

  

  

  1
• 1 You must be logged in to vote

  💬

  Structured Generation with Reasoning Parser in offline mode.

  psych0v0yager announced May 4, 2025 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  ......lib/python3.12/site-packages/vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb

  RexLyu announced May 1, 2025 in Q&A

  

  

  0
• 1 You must be logged in to vote

  💬

  vLLM cannot connect to existing Ray cluster

  as-bain announced May 1, 2025 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  Determining Overall Speed for One Long Prompt

  chigkim announced Apr 30, 2025 in Q&A

  

  

  0
• 1 You must be logged in to vote

  🙏

  WARNING 02-09 10:24:45 cpu.py:99] ray is not supported on CPU, fallback to mp distributed executor backend.

  jdmedeiros announced Feb 9, 2025 in Q&A

  

  

  2
• 1 You must be logged in to vote

  🙏

  v0.8.2 "No available memory for the cache blocks." when launching models on the same GPU

  CaoChensy announced Apr 1, 2025 in Q&A

  

  

  

  

  3
• 1 You must be logged in to vote

  💬

  Why gfx908 (mi100) was deleted from the arch list?

  hnhyzz announced Apr 28, 2025 in General

  

  

  0
• 2 You must be logged in to vote

  🙏

  Any known integration with n8n ?

  wugutech announced Apr 28, 2025 in Q&A

  

  

  0
• 8 You must be logged in to vote

  📣

  GitHub discussion is not used anymore, please use the forum for discussion.

  youkaichao announced Mar 20, 2025 in Announcements

  

  

  

  

  4
• 3 You must be logged in to vote

  🙏

  How do let vllm v1 to use a customized stat logger extended from StatLoggerBase?

  liuzijing2014 announced Mar 5, 2025 in Q&A

  

  

  1
• 1 You must be logged in to vote

  🙏

  Why temperature=0,top_p=1,seed=42 is still not enough to fix the llm's output!?

  beyondguo announced Apr 25, 2025 in Q&A

  

  

  0
• 2 You must be logged in to vote

  🙏

  LoRA Adapter enabling with vLLM is not working

  SSPRATAP17 announced Apr 14, 2025 in Q&A

  

  

  

  

  3
• 2 You must be logged in to vote

  🙏

  KV cache in VLLM serve

  farouk09 announced Apr 23, 2025 in Q&A

  

  

  0
• 1 You must be logged in to vote

  🙏

  VLLM Inference Optimizations

  wlm64 announced Apr 21, 2025 in Q&A

  

  

  0
• 1 You must be logged in to vote

  🙏

  Doing Batched Inference Using TokensPrompt

  wlm64 announced Apr 10, 2025 in Q&A

  

  

  1
• 1 You must be logged in to vote

  🙏

  Does VLLM support DP attention for deepseek?

  llc-kc announced Apr 18, 2025 in Q&A

  

  

  0
• 1 You must be logged in to vote

  🙏

  How to configure vllm to gracefully mark the too-long inputs without throwing?

  vadimkantorov announced Apr 16, 2025 in Q&A

  

  

  0
• 1 You must be logged in to vote

  🙏

  The num of dynamically serving LoRA adapters can not limited by param max-loras ?

  estuday announced Apr 15, 2025 in Q&A

  

  

  0
• 1 You must be logged in to vote

  🙏

  Why is the GPU KV cache usage very low?

  usage How to use vllm

  tammypi announced Jun 18, 2024 in Q&A

  

  

  

  

  2
• 1 You must be logged in to vote

  💬

  I use VLLM to deploy DeepSeek, and when there are high concurrency requests, many of them are pending reqs. How can I set a timeout for these waiting requests?

  nvliajia announced Apr 10, 2025 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  Running Custom Model with isinstance(model, VllmModelForTextGeneration) problem

  yesilcagri announced Apr 1, 2025 in Q&A

  

  

  

  

  2
• 1 You must be logged in to vote

  💬

  q_lora_rank in vLLM with DeepSeek MLA Model

  erictanjn announced Apr 7, 2025 in General

  

  

  0
• 2 You must be logged in to vote

  🙏

  Docker Image Build

  artist-genai announced Feb 28, 2025 in Q&A

  

  

  2
• 1 You must be logged in to vote

  🙌

  Compile error

  qux-bbb announced Apr 6, 2025 in Show and tell

  

  

  0