model not loading on GPU #212

New issue

Open

opened 2024-07-28 05:35:39 -04:00 by kot197 · 0 comments

kot197 commented

2024-07-28 05:35:39 -04:00

(Migrated from github.com)

Hi, I just want to run this simple code on GPU...

from langchain_community.llms import CTransformers

llm = CTransformers(model="./airoboros-mistral2.2-7b.Q4_K_S.gguf", model_type="mistral", gpu_layers=32, verbose=True)

print(llm.invoke('AI is going to'))

As you can see...GPU usage is at 0% and it took about 1 minute which is very long for a quick prompt

Is there nothing I can do to visualize what's going on, like printing something on the terminal? How do I even know if it's running or not? I cannot fully trust task manager

I'm a newcomer, please help me

Hi, I just want to run this simple code on GPU... ``` from langchain_community.llms import CTransformers llm = CTransformers(model="./airoboros-mistral2.2-7b.Q4_K_S.gguf", model_type="mistral", gpu_layers=32, verbose=True) print(llm.invoke('AI is going to')) ``` ![image](https://github.com/user-attachments/assets/97b8d8df-06b7-4ba0-9ac8-16368f7cb398) As you can see...GPU usage is at 0% and it took about 1 minute which is very long for a quick prompt Is there nothing I can do to visualize what's going on, like printing something on the terminal? How do I even know if it's running or not? I cannot fully trust task manager I'm a newcomer, please help me

👍 1

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

marella/ctransformers#212

No description provided.

Rows
Columns