Use proper backend for CLIP #1175
Conversation
|
@iamlemec that's great, thank you! |
|
version llama-cpp-python-0.2.56, how to enable CLIP offload to GPU? my 3090 can do 50 token/s but total time would be tooo slow(92s), much slower than my Macbook M3 max(6s), |
|
You should try to see whether it is in fact using the GPU. Assuming you're running something like the LLaVa example in the README, you can load the CLIP model in verbose mode with: chat_handler = Llava15ChatHandler(clip_model_path="path/to/llava/mmproj.bin", verbose=True)If you're using the CUDA backend, it should have the line: "clip_model_load: CLIP using CUDA backend". But yeah, it would be surprising if a 3090 was that much slower than an M3. |
|
@iamlemec thanks! |
|
i found the issue! when setting the context_size to 4096 the generation never ends!! it will try to decode until the context is full which is why it took so long to finish, but this won't happen on M3 max, strange..... |
|
Hmm, that's curious. Does the output from CUDA look reasonable? Also, what are you setting |
|
i did not set max_tokens, i just set n_ctx=4096 on Macbook is works ok, but on cuda it will keep generatting never ends |
When using LLaVa, CLIP does not load on the default backend (i.e. CUDA/Metal when they are available). This arises because the code in
clip.cppconditions onGGML_USE_XXXrather thanLLAMA_USE_XXX. The mainllama.cppCMake file enables the former when the latter is present, but we need to do it manually for LLaVA since we're bypassing the top level.