I'm having trouble running gguf models

fatihsazan commented

2025-01-12 18:31:32 -05:00

(Migrated from github.com)

Hi,

Is there something wrong with using it this way? I can't run the gguf models I want to try.

from ctransformers import AutoModelForCausalLM

model_name = "SanctumAI/Llama-3.2-3B-Instruct-GGUF"
gguf_file = "llama-3.2-3b-instruct.Q2_K.gguf"

llm = AutoModelForCausalLM.from_pretrained(
    model_name,
    model_file=gguf_file,
    model_type="gguf")

Error output:

, line 8, in <module>
    llm = AutoModelForCausalLM.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/ctransformers/hub.py", line 175, in from_pretrained
    llm = LLM(
  File "/usr/local/lib/python3.10/dist-packages/ctransformers/llm.py", line 253, in __init__
    raise RuntimeError(
RuntimeError: Failed to create LLM 'gguf' from '/root/.cache/huggingface/hub/models--SanctumAI--Llama-3.2-3B-Instruct-GGUF/blobs/c77eb142ab869944f388ff093fc7276ea15c4e1f810ceb76554fc5ae77694c19'.

Hi, Is there something wrong with using it this way? I can't run the gguf models I want to try. ``` from ctransformers import AutoModelForCausalLM model_name = "SanctumAI/Llama-3.2-3B-Instruct-GGUF" gguf_file = "llama-3.2-3b-instruct.Q2_K.gguf" llm = AutoModelForCausalLM.from_pretrained( model_name, model_file=gguf_file, model_type="gguf") ``` Error output: ``` , line 8, in <module> llm = AutoModelForCausalLM.from_pretrained( File "/usr/local/lib/python3.10/dist-packages/ctransformers/hub.py", line 175, in from_pretrained llm = LLM( File "/usr/local/lib/python3.10/dist-packages/ctransformers/llm.py", line 253, in __init__ raise RuntimeError( RuntimeError: Failed to create LLM 'gguf' from '/root/.cache/huggingface/hub/models--SanctumAI--Llama-3.2-3B-Instruct-GGUF/blobs/c77eb142ab869944f388ff093fc7276ea15c4e1f810ceb76554fc5ae77694c19'. ```

werruww commented

2025-06-23 21:08:58 -04:00

(Migrated from github.com)

yes

shama2025 commented

2025-11-25 08:45:28 -05:00

(Migrated from github.com)

Hello @fatihsazan I haven't done much with ctransformers but from my understanding, the model type "gguf" isn't a valid type. GGUF is a file format for storing models. You would need to change it to llama since that is the type of the model you are using. Here is an example with The Blokes Mistral models.

MISTRAL_REPO = "TheBloke/Mistral-7B-Instruct-v0.2-GGUF"
model = AutoModelForCausalLM.from_pretrained(
            MISTRAL_REPO, hf=False, model_type="mistral"
        )

If you already download the gguf file locally, that requires the huggingface_hub snapshot_download function which will load the models in a folder of your choice and then you can call using the AutoModelForCasualLm model_path_or_repo_path parameter to point to the folder.

MODEL_DIR = "path/to/mistral/model"
model = AutoModelForCausalLM.from_pretrained(
            MODEL_DIR, hf=False, model_type="mistral"
        )

Hello @fatihsazan I haven't done much with ctransformers but from my understanding, the model type "gguf" isn't a valid type. GGUF is a file format for storing models. You would need to change it to llama since that is the type of the model you are using. Here is an example with The Blokes Mistral models. ```python MISTRAL_REPO = "TheBloke/Mistral-7B-Instruct-v0.2-GGUF" model = AutoModelForCausalLM.from_pretrained( MISTRAL_REPO, hf=False, model_type="mistral" ) ``` If you already download the gguf file locally, that requires the huggingface_hub snapshot_download function which will load the models in a folder of your choice and then you can call using the AutoModelForCasualLm model_path_or_repo_path parameter to point to the folder. ```python MODEL_DIR = "path/to/mistral/model" model = AutoModelForCausalLM.from_pretrained( MODEL_DIR, hf=False, model_type="mistral" ) ```

Rows
Columns

I'm having trouble running gguf models #215