CUDA error 35

curname commented

2023-08-15 03:50:04 -04:00

(Migrated from github.com)

When i run ctransformers[cuda], i get the error:
CUDA error 35 at /home/runner/work/ctransformers/ctransformers/models/ggml/ggml-cuda.cu:4236: CUDA driver version is insufficient for CUDA runtime version

However, the path "/home/runner/work/ctransformers/ctransformers/models/ggml/ggml-cuda.cu" does not exit. And my CUDA info:

And my package info:

how to fix it?

When i run ctransformers[cuda], i get the error: CUDA error 35 at /home/runner/work/ctransformers/ctransformers/models/ggml/ggml-cuda.cu:4236: CUDA driver version is insufficient for CUDA runtime version However, the path "/home/runner/work/ctransformers/ctransformers/models/ggml/ggml-cuda.cu" does not exit. And my CUDA info: ![gpu_info](https://github.com/marella/ctransformers/assets/38759604/3a85f1bf-13b5-4356-a120-0d906e21fd69) And my package info: ![package](https://github.com/marella/ctransformers/assets/38759604/ff65c231-65f6-4cce-81a6-9f4a9302898f) how to fix it?

👀 1

curname commented

2023-08-15 04:23:45 -04:00

(Migrated from github.com)

Here is my code:
llm = AutoModelForCausalLM.from_pretrained( "starcoder.ggmlv3.q8_0.bin", model_type="gpt_bigcode", top_p=0.95, temperature=0.2, max_new_tokens=512, threads=8, gpu_layers=50 )

Here is my code: `llm = AutoModelForCausalLM.from_pretrained( "starcoder.ggmlv3.q8_0.bin", model_type="gpt_bigcode", top_p=0.95, temperature=0.2, max_new_tokens=512, threads=8, gpu_layers=50 )`

marella commented

2023-08-15 07:46:29 -04:00

(Migrated from github.com)

Please update your NVIDIA Drivers and try again.

Please update your [NVIDIA Drivers](https://www.nvidia.com/download/index.aspx) and try again.

sujeendran commented

2023-09-25 12:25:10 -04:00

(Migrated from github.com)

Hi @marella - I'm facing a similar issue in the servers that I am testing on. Upgrading the drivers might not be an option for me as it is a shared system several people use. Is it possible to manually build this library to run on Cuda 11.8 by making few tweaks to setup/cmake files?

sujeendran commented

2023-09-25 12:52:31 -04:00

(Migrated from github.com)

Just an update, managed to get it running on CUDA 11.8😄 ! I knew it should work as I was able to run GGUF model using llama-cpp with the same CUDA versions and drivers.
Here is the fix if anyone want to try it:

Clone the library git clone https://github.com/marella/ctransformers.git
Edit this line to use older cuda version: https://github.com/marella/ctransformers/blob/main/models/ggml/ggml-cuda.cu#L136 to:

#if CUDART_VERSION >= 11000

In the root folder, execute:

CT_CUBLAS=1 pip install .

Remember to install cuda libraries if you don't have them yet:

pip install nvidia-cuda-runtime-cu11 nvidia-cublas-cu11

@marella - Do you think I can start a PR to include the step 2 fix so this library is compatible with older versions too?

Just an update, managed to get it running on CUDA 11.8😄 ! I knew it should work as I was able to run GGUF model using llama-cpp with the same CUDA versions and drivers. Here is the fix if anyone want to try it: 1. Clone the library `git clone https://github.com/marella/ctransformers.git` 2. Edit this line to use older cuda version: https://github.com/marella/ctransformers/blob/main/models/ggml/ggml-cuda.cu#L136 to: ``` #if CUDART_VERSION >= 11000 ``` 3. In the root folder, execute: ``` CT_CUBLAS=1 pip install . ``` 4. Remember to install cuda libraries if you don't have them yet: ``` pip install nvidia-cuda-runtime-cu11 nvidia-cublas-cu11 ``` @marella - Do you think I can start a PR to include the step 2 fix so this library is compatible with older versions too?

👍 5

gorkemgoknar commented

2023-10-27 03:36:36 -04:00

(Migrated from github.com)

this should be integrated cuda 11.8 is working fine (and 11.8 should be compatible with 11.x) , even latest pytorch version (as of today 2.1) still supports it.
And for updating Nvidia-drivers, it will not be easy on a cloud provider node (or like a HF space) , also from my experience updating nvidia-drivers on older cards (2070 Turing for example), just makes them slower so I stick with the best performing version.

this should be integrated cuda 11.8 is working fine (and 11.8 should be compatible with 11.x) , even latest pytorch version (as of today 2.1) still supports it. And for updating Nvidia-drivers, it will not be easy on a cloud provider node (or like a HF space) , also from my experience updating nvidia-drivers on older cards (2070 Turing for example), just makes them slower so I stick with the best performing version.

👍 1

sujeendran commented

2023-10-27 03:55:30 -04:00

(Migrated from github.com)

@gorkemgoknar - I have created a pull request to get this included in the main repo. In the meantime until it is merged, if someone doesn't want to make manual changes(it's a simple one anyway), they can clone and build directly from my fork: https://github.com/sujeendran/ctransformers

gorkemgoknar commented

2023-10-27 04:05:29 -04:00

(Migrated from github.com)

@gorkemgoknar - I have created a pull request to get this included in the main repo. In the meantime until it is merged, if someone doesn't want to make manual changes(it's a simple one anyway), they can clone and build directly from my fork: https://github.com/sujeendran/ctransformers

Thank you @sujeendran . I actually builded with the fix.
I can confirm with the fix it runs GGUF zephyr or mistral with nvidia-cuda-runtime-cu11==11.7.99
Just a side note for GGUF: generation performance nearly same as llama-cpp-python

> @gorkemgoknar - I have created a pull request to get this included in the main repo. In the meantime until it is merged, if someone doesn't want to make manual changes(it's a simple one anyway), they can clone and build directly from my fork: https://github.com/sujeendran/ctransformers Thank you @sujeendran . I actually builded with the fix. I can confirm with the fix it runs GGUF zephyr or mistral with nvidia-cuda-runtime-cu11==11.7.99 Just a side note for GGUF: generation performance nearly same as llama-cpp-python

👍 1

AlexBlack2202 commented

2023-11-04 11:31:15 -04:00

(Migrated from github.com)

@gorkemgoknar - I have created a pull request to get this included in the main repo. In the meantime until it is merged, if someone doesn't want to make manual changes(it's a simple one anyway), they can clone and build directly from my fork: https://github.com/sujeendran/ctransformers

Thank you @sujeendran . I actually builded with the fix. I can confirm with the fix it runs GGUF zephyr or mistral with nvidia-cuda-runtime-cu11==11.7.99 Just a side note for GGUF: generation performance nearly same as llama-cpp-python

can you run GGUF format with GPU ?

> > @gorkemgoknar - I have created a pull request to get this included in the main repo. In the meantime until it is merged, if someone doesn't want to make manual changes(it's a simple one anyway), they can clone and build directly from my fork: https://github.com/sujeendran/ctransformers > > Thank you @sujeendran . I actually builded with the fix. I can confirm with the fix it runs GGUF zephyr or mistral with nvidia-cuda-runtime-cu11==11.7.99 Just a side note for GGUF: generation performance nearly same as llama-cpp-python can you run GGUF format with GPU ?

gorkemgoknar commented

2023-11-04 11:34:58 -04:00

(Migrated from github.com)

@gorkemgoknar - I have created a pull request to get this included in the main repo. In the meantime until it is merged, if someone doesn't want to make manual changes(it's a simple one anyway), they can clone and build directly from my fork: https://github.com/sujeendran/ctransformers

Thank you @sujeendran . I actually builded with the fix. I can confirm with the fix it runs GGUF zephyr or mistral with nvidia-cuda-runtime-cu11==11.7.99 Just a side note for GGUF: generation performance nearly same as llama-cpp-python

can you run GGUF format with GPU ?

Yes, check the app.py here, GGUF is for both CPU and GPU and with changing layer on GPU you can run some of the ops on GPU if your GPU does not have enougn VRAM

https://huggingface.co/spaces/coqui/voice-chat-with-mistral

> > > @gorkemgoknar - I have created a pull request to get this included in the main repo. In the meantime until it is merged, if someone doesn't want to make manual changes(it's a simple one anyway), they can clone and build directly from my fork: https://github.com/sujeendran/ctransformers > > > > > > Thank you @sujeendran . I actually builded with the fix. I can confirm with the fix it runs GGUF zephyr or mistral with nvidia-cuda-runtime-cu11==11.7.99 Just a side note for GGUF: generation performance nearly same as llama-cpp-python > > > > can you run GGUF format with GPU ? Yes, check the app.py here, GGUF is for both CPU and GPU and with changing layer on GPU you can run some of the ops on GPU if your GPU does not have enougn VRAM https://huggingface.co/spaces/coqui/voice-chat-with-mistral

Rows
Columns

CUDA error 35 #90