How to load the finetuned model in safetensors format(not in gguf) #201

New issue

Open

opened 2024-02-15 00:28:47 -05:00 by pradeepdev-1995 · 0 comments

pradeepdev-1995 commented

2024-02-15 00:28:47 -05:00

(Migrated from github.com)

I have finetuned the mistral base model with my data using LORA PEFT
Base model tried: mistralai/Mistral-7B-Instruct-v0.2
Finetuned merged model folder structure:

All model files are in safetensors format. not in gguf.
Now i tried to load the finetuned model with ctransformes

import ctransformers
from ctransformers import AutoModelForCausalLM
llm = AutoModelForCausalLM.from_pretrained("Directory Path",
                                           model_file="Model folder name",
                                           model_type="mistral",
                                           gpu_layers=1)

But it shows the given error

ValueError: Model file '<Model folder name>' not found in '<Directory Path>'

I have finetuned the mistral base model with my data using LORA PEFT Base model tried: **mistralai/Mistral-7B-Instruct-v0.2** Finetuned merged model folder structure: ![Screenshot from 2024-02-15 10-49-03](https://github.com/marella/ctransformers/assets/41164884/2e265320-42be-4497-a6cb-94709c31d244) All model files are in safetensors format. not in gguf. Now i tried to load the finetuned model with ctransformes ``` import ctransformers from ctransformers import AutoModelForCausalLM llm = AutoModelForCausalLM.from_pretrained("Directory Path", model_file="Model folder name", model_type="mistral", gpu_layers=1) ``` But it shows the given error ```ValueError: Model file '<Model folder name>' not found in '<Directory Path>'```

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

marella/ctransformers#201

No description provided.

Rows
Columns