Cannot generate text on GPU #199

New issue

Open

opened 2024-01-02 05:46:33 -05:00 by congson1293 · 3 comments

congson1293 commented

2024-01-02 05:46:33 -05:00

(Migrated from github.com)

I load the model to GPU like this:

llm = AutoModelForCausalLM.from_pretrained("LLM-model", 
                                            model_file="vinallama-7b-chat_q5_0.gguf",
                                            config=config, torch_dtype=torch.float16, hf=True,
                                            gpu_layers = 100, device_map='cuda')

and generate code like this:

generated_ids = llm.generate(**model_inputs,
                              max_new_tokens=4096,
                              # early_stopping=True,
                              repetition_penalty=1.1,
                              # no_repeat_ngram_size=2,
                              temperature=0.6,
                              do_sample=True,
                              # top_k=5,
                              top_p=0.9,
                              eos_token_id=tokenizer.eos_token_id,
                              pad_token_id=tokenizer.pad_token_id,
                              use_cache=True)

when I ran this code, the model got 6.6Gb on GPU. But I've got the exception: You are calling .generate() with the input_ids being on a device type different than your model's device. input_ids is on cuda, whereas the model is on cpu. when ran generate method.
Does anyone know the way to fix it?

I load the model to GPU like this: ```python llm = AutoModelForCausalLM.from_pretrained("LLM-model", model_file="vinallama-7b-chat_q5_0.gguf", config=config, torch_dtype=torch.float16, hf=True, gpu_layers = 100, device_map='cuda') ``` and generate code like this: ```python generated_ids = llm.generate(**model_inputs, max_new_tokens=4096, # early_stopping=True, repetition_penalty=1.1, # no_repeat_ngram_size=2, temperature=0.6, do_sample=True, # top_k=5, top_p=0.9, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id, use_cache=True) ``` when I ran this code, the model got 6.6Gb on GPU. But I've got the exception: **You are calling .generate() with the `input_ids` being on a device type different than your model's device. `input_ids` is on cuda, whereas the model is on cpu.** when ran generate method. Does anyone know the way to fix it?

Michielo1 commented

2024-01-02 10:23:51 -05:00

(Migrated from github.com)

What device is model_inputs on?

congson1293 commented

2024-01-02 10:26:16 -05:00

(Migrated from github.com)

I use T4 on Google Colab.

davidearlyoung commented

2024-01-05 20:18:17 -05:00

(Migrated from github.com)

Double check your code parts against the gold standard examples at: https://github.com/marella/ctransformers?tab=readme-ov-file#classmethod-automodelforcausallmfrom_pretrained

Do the same for the generate method of your llm class. Here is the gold standard reference for that as well: https://github.com/marella/ctransformers?tab=readme-ov-file#classmethod-automodelforcausallmfrom_pretrained

It looks like you have a lot of unnecessary arguments that you are mimicking from other libraries. Especially for the from_pretrained method call.

Hope that this helps.

Double check your code parts against the gold standard examples at: https://github.com/marella/ctransformers?tab=readme-ov-file#classmethod-automodelforcausallmfrom_pretrained Do the same for the generate method of your llm class. Here is the gold standard reference for that as well: https://github.com/marella/ctransformers?tab=readme-ov-file#classmethod-automodelforcausallmfrom_pretrained It looks like you have a lot of unnecessary arguments that you are mimicking from other libraries. Especially for the from_pretrained method call. Hope that this helps.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

marella/ctransformers#199

No description provided.

Rows
Columns