[AMD] Fix compilation issue with ROCm

bhargav commented

2023-09-17 14:43:02 -04:00

(Migrated from github.com)

Problem:
Unable to install the package on a Linux machine with an AMD 6800XT GPU using ROCm.

docker run -it --device=/dev/kfd --device=/dev/dri --group-add video docker.io/rocm/dev-ubuntu-22.04:5.6-complete bash

CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers

# Compiling after setting CC and CXX env variables also failed with a similar error.
CC=/opt/rocm/llvm/bin/clang CXX=/opt/rocm/llvm/bin/clang++ CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers

Error logs: https://gist.github.com/bhargav/7f8c2984ba32ff99ce8e93433d9059a6

Solution:
Failures are due to references to CUDA library imports instead of using the HIP versions when compiled for AMD.

Verified that the project can build with the fixes.

apt-get update && apt-get install -y git

git clone https://github.com/bhargav/ctransformers.git
cd ctransformers

git checkout bhargav/fix_rocm_compile

CC=/opt/rocm/llvm/bin/clang CXX=/opt/rocm/llvm/bin/clang++ CT_HIPBLAS=1 pip install .

Build log: https://gist.github.com/bhargav/65bbbd039bda6f39504448656e88ab6b

Package installs successfully and I was able to run a model inference on GPU.

**Problem:** Unable to install the package on a Linux machine with an AMD 6800XT GPU using ROCm. ``` docker run -it --device=/dev/kfd --device=/dev/dri --group-add video docker.io/rocm/dev-ubuntu-22.04:5.6-complete bash CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers # Compiling after setting CC and CXX env variables also failed with a similar error. CC=/opt/rocm/llvm/bin/clang CXX=/opt/rocm/llvm/bin/clang++ CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers ``` Error logs: https://gist.github.com/bhargav/7f8c2984ba32ff99ce8e93433d9059a6 **Solution:** Failures are due to references to CUDA library imports instead of using the HIP versions when compiled for AMD. Verified that the project can build with the fixes. ``` apt-get update && apt-get install -y git git clone https://github.com/bhargav/ctransformers.git cd ctransformers git checkout bhargav/fix_rocm_compile CC=/opt/rocm/llvm/bin/clang CXX=/opt/rocm/llvm/bin/clang++ CT_HIPBLAS=1 pip install . ``` Build log: https://gist.github.com/bhargav/65bbbd039bda6f39504448656e88ab6b Package installs successfully and I was able to run a model inference on GPU.

👍 3

mega-ice commented

2023-09-20 07:28:26 -04:00

(Migrated from github.com)

i can confirm. Compile without a issue now. (rocm nightly and old vega64 :)

👍 2

ahmedashraf093 commented

2023-10-10 20:21:18 -04:00

(Migrated from github.com)

Hello,

Installing done without error but when trying to prompt the model using

llm = AutoModelForCausalLM.from_pretrained("/models/llama-7b.Q3_K_M.gguf", model_type="llama" , local_files_only=True, gpu_layers=100)

print(llm("AI is going to"))

exits with error

CUDA error 98 at ~/ctransformers/models/ggml/ggml-cuda.cu:6045: invalid device function

Hello, Installing done without error but when trying to prompt the model using ```python llm = AutoModelForCausalLM.from_pretrained("/models/llama-7b.Q3_K_M.gguf", model_type="llama" , local_files_only=True, gpu_layers=100) print(llm("AI is going to")) ``` exits with error ```bash CUDA error 98 at ~/ctransformers/models/ggml/ggml-cuda.cu:6045: invalid device function ```

mega-ice commented

2023-10-14 04:29:59 -04:00

(Migrated from github.com)

When using ROCm, "CUDA error 98 ....... invalid device function" (as far as I know) usually means version implementation problems in the HIP stack. Most likely it's solvable with

export HSA_OVERRIDE_GFX_VERSION=11.0.0 for rdna3 gpu
export HSA_OVERRIDE_GFX_VERSION=10.3.0 for older one

When using ROCm, "CUDA error 98 ....... invalid device function" (as far as I know) usually means version implementation problems in the HIP stack. Most likely it's solvable with `export HSA_OVERRIDE_GFX_VERSION=11.0.0` for rdna3 gpu `export HSA_OVERRIDE_GFX_VERSION=10.3.0` for older one

👍 1

afazekas commented

2023-11-26 10:04:42 -05:00

(Migrated from github.com)

Hi,
In the referenced container version it can work with a gfx1030 system.
docker.io/rocm/dev-ubuntu-22.04:5.6-complete

apt show rocm-libs -a

Package: rocm-libs
Version: 5.6.0.50600-67~22.04
Priority: optional
Section: devel

But in "rocm/pytorch:latest-release"

apt show rocm-libs -a
Package: rocm-libs
Version: 5.7.0.50700-63~20.04
Priority: optional

Raises the" invalid device function" Error.

NOTE:
If you import the cloned directory by accident instead of the installed package:
OSError: libcudart.so.12: cannot open shared object file: No such file or directory

NOTE:
I disabled the integrated gfx1036 GPU from the CPU, so I have only the gfx1030 system.

I assume the "invalid device function" depends on the environment's library version(s).

/lgtm

Hi, In the referenced container version it can work with a gfx1030 system. docker.io/rocm/dev-ubuntu-22.04:5.6-complete apt show rocm-libs -a ``` Package: rocm-libs Version: 5.6.0.50600-67~22.04 Priority: optional Section: devel ``` But in "rocm/pytorch:latest-release" ``` apt show rocm-libs -a Package: rocm-libs Version: 5.7.0.50700-63~20.04 Priority: optional ``` Raises the" invalid device function" Error. NOTE: If you import the cloned directory by accident instead of the installed package: `OSError: libcudart.so.12: cannot open shared object file: No such file or directory` NOTE: I disabled the integrated gfx1036 GPU from the CPU, so I have only the gfx1030 system. I assume the "invalid device function" depends on the environment's library version(s). /lgtm

0xGingi commented

2023-12-14 16:17:47 -05:00

(Migrated from github.com)

CUDA error 98 at /home/gingi/github/ctransformers/models/ggml/ggml-cuda.cu:6045: invalid device function

I have export HSA_OVERRIDE_GFX_VERSION=11.0.0
and running HSA_OVERRIDE_GFX_VERSION=11.0.0 python index.py

CUDA error 98 at /home/gingi/github/ctransformers/models/ggml/ggml-cuda.cu:6045: invalid device function I have export HSA_OVERRIDE_GFX_VERSION=11.0.0 and running HSA_OVERRIDE_GFX_VERSION=11.0.0 python index.py

huotianyu commented

2024-03-19 23:39:07 -04:00

(Migrated from github.com)

the below command worked well！
CMAKE_ARGS="-DLLAMA_HIPBLAS=on -DLLAMA_CLBLAST=on -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ -DCMAKE_PREFIX_PATH=/opt/rocm" FORCE_CMAKE=1 pip install ctransformers --no-binary ctransformers

the below command worked well！ CMAKE_ARGS="-DLLAMA_HIPBLAS=on -DLLAMA_CLBLAST=on -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ -DCMAKE_PREFIX_PATH=/opt/rocm" FORCE_CMAKE=1 pip install ctransformers --no-binary ctransformers

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin bhargav/bhargav/fix_rocm_compile:bhargav/bhargav/fix_rocm_compile

git switch bhargav/bhargav/fix_rocm_compile

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff bhargav/bhargav/fix_rocm_compile

git switch bhargav/bhargav/fix_rocm_compile

git rebase main

git switch main

git merge --ff-only bhargav/bhargav/fix_rocm_compile

git switch bhargav/bhargav/fix_rocm_compile

git rebase main

git switch main

git merge --no-ff bhargav/bhargav/fix_rocm_compile

git switch main

git merge --squash bhargav/bhargav/fix_rocm_compile

git switch main

git merge --ff-only bhargav/bhargav/fix_rocm_compile

git switch main

git merge bhargav/bhargav/fix_rocm_compile

git push origin main

Rows
Columns

[AMD] Fix compilation issue with ROCm #137

Checkout

Merge