Inputting embeddings directly #205

New issue

Open

opened 2024-02-24 11:46:09 -05:00 by liechtym · 0 comments

liechtym commented

2024-02-24 11:46:09 -05:00

(Migrated from github.com)

I have a use case where I need to modify input embeddings before they

from ctransformers import AutoModelForCausalLM

llm = AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7b-Chat-GGUF", model_file="llama-2-7b-chat.Q4_K_M.gguf", model_type="llama", gpu_layers=50)

embeddings = llm.embed('Some text')

# Code to modify embeddings
modified_embeddings = some_function(embeddings)

res = llm.generate(inputs_embeds=modified_embeddings)

Is there any way to do this?

I have a use case where I need to modify input embeddings before they ``` from ctransformers import AutoModelForCausalLM llm = AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7b-Chat-GGUF", model_file="llama-2-7b-chat.Q4_K_M.gguf", model_type="llama", gpu_layers=50) embeddings = llm.embed('Some text') # Code to modify embeddings modified_embeddings = some_function(embeddings) res = llm.generate(inputs_embeds=modified_embeddings) ``` Is there any way to do this?

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

marella/ctransformers#205

No description provided.

Rows
Columns