transformers 4.34 caused NotImplementedError when calling CTransformersTokenizer(PreTrainedTokenizer)

victorlee0505 commented

2023-10-04 12:43:38 -04:00

(Migrated from github.com)

transformers version: pip install transformers==4.34.0
ctransformersversion: pip install ctransformers==0.2.27

I encounter the following error

File ".venv\lib\site-packages\ctransformers\transformers.py", line 84, in __init__kages\ctransformers\transformers.py", line 84, in __init__
    super().__init__(**kwargs)

File ".venv\lib\site-packages\transformers\tokenization_utils.py", line 366, in __init__
    self._add_tokens(self.all_special_tokens_extended, special_tokens=True)

File ".venv\lib\site-packages\transformers\tokenization_utils.py", line 462, in _add_tokens
    current_vocab = self.get_vocab().copy()

File ".venv\lib\site-packages\transformers\tokenization_utils_base.py", line 1715, in ``get_vocab
    raise NotImplementedError()``
NotImplementedError

transformers has PreTrainedTokenizer in tokenization_utils.py code change (2da8853) where _add_tokens on line 454 current_vocab = self.get_vocab().copy().

PreTrainedTokenizer itself has added_tokens_decoder and __len__ implemented, so only get_vocab would cause NotImplementedError()

transformers version: `pip install transformers==4.34.0` ctransformersversion: `pip install ctransformers==0.2.27` I encounter the following error ``` File ".venv\lib\site-packages\ctransformers\transformers.py", line 84, in __init__kages\ctransformers\transformers.py", line 84, in __init__ super().__init__(**kwargs) File ".venv\lib\site-packages\transformers\tokenization_utils.py", line 366, in __init__ self._add_tokens(self.all_special_tokens_extended, special_tokens=True) File ".venv\lib\site-packages\transformers\tokenization_utils.py", line 462, in _add_tokens current_vocab = self.get_vocab().copy() File ".venv\lib\site-packages\transformers\tokenization_utils_base.py", line 1715, in ``get_vocab raise NotImplementedError()`` NotImplementedError ``` transformers has PreTrainedTokenizer in `tokenization_utils.py` code change (2da8853) where _add_tokens on line 454 `current_vocab = self.get_vocab().copy()`. PreTrainedTokenizer itself has `added_tokens_decoder` and` __len__` implemented, so only `get_vocab` would cause NotImplementedError()

👍 2

CHesketh76 commented

2023-10-04 16:17:03 -04:00

(Migrated from github.com)

Issue can be created with this code from the readme

from ctransformers import AutoModelForCausalLM

llm = AutoModelForCausalLM.from_pretrained("/path/to/ggml-model.bin", model_type="gpt2", hf=True)

print(llm("AI is going to"))

or in https://colab.research.google.com/drive/1GMhYMUAv_TyZkpfvUI1NirM8-9mCXQyL.

Hope this issue is address because finding the correct tokenizer from a different source may not be possible for most model.

Issue can be created with this code from the readme ``` from ctransformers import AutoModelForCausalLM llm = AutoModelForCausalLM.from_pretrained("/path/to/ggml-model.bin", model_type="gpt2", hf=True) print(llm("AI is going to")) ``` or in https://colab.research.google.com/drive/1GMhYMUAv_TyZkpfvUI1NirM8-9mCXQyL. Hope this issue is address because finding the correct tokenizer from a different source may not be possible for most model.

victorlee0505 commented

2023-10-04 16:55:56 -04:00

(Migrated from github.com)

PR submitted and it works for me now, this is my setup.

model = AutoModelForCausalLM.from_pretrained(..., hf=True)
tokenizer = AutoTokenizer.from_pretrained(model)

transformers 4.34.0 now support Mistral, so I really want to use it. 😁

PR submitted and it works for me now, this is my setup. ``` model = AutoModelForCausalLM.from_pretrained(..., hf=True) tokenizer = AutoTokenizer.from_pretrained(model) ``` transformers 4.34.0 now support Mistral, so I really want to use it. 😁

CHesketh76 commented

2023-10-04 18:25:49 -04:00

(Migrated from github.com)

I spent all day trying to get Mistral working with ctranformers, but it is returning garbage text on my end. I believe it may be the tokenizer because tokenizer = AutoTokenizer.from_pretrained(model) will not work for any model.

I spent all day trying to get Mistral working with ctranformers, but it is returning garbage text on my end. I believe it may be the tokenizer because ```tokenizer = AutoTokenizer.from_pretrained(model)``` will not work for any model. ![Capture](https://github.com/marella/ctransformers/assets/38713764/c8163c91-fc9f-465e-b3c0-072d8fcea822)

victorlee0505 commented

2023-10-04 18:46:20 -04:00

(Migrated from github.com)

Yes, they refactored PreTrainedTokenizer which cTransformers tokernizer extended from. I ran open orca Mistral it runs fine with 4.34 but all quantized failed unless I go back to 4.33, so my PR fixed that.
I will try to run quantized Mistral tomorrow to see if it works.

Yes, they refactored PreTrainedTokenizer which cTransformers tokernizer extended from. I ran open orca Mistral it runs fine with 4.34 but all quantized failed unless I go back to 4.33, so my PR fixed that. I will try to run quantized Mistral tomorrow to see if it works.

victorlee0505 commented

2023-10-05 10:23:49 -04:00

(Migrated from github.com)

I just ran TheBloke/Mistral-7B-OpenOrca-GGUF, it works fine for me.

CHesketh76 commented

2023-10-05 10:52:37 -04:00

(Migrated from github.com)

Are you able to use the model.generate(...) I have got everything to run until I start generating text, it will just run indefinitely.

Are you able to use the ```model.generate(...)``` I have got everything to run until I start generating text, it will just run indefinitely.

victorlee0505 commented

2023-10-05 12:38:22 -04:00

(Migrated from github.com)

ok, I quickly write this up and it works fine (you will need transformers==4.34.0 then build ctransformers from #155 and install)


import os
from ctransformers import (
    AutoModelForCausalLM as cAutoModelForCausalLM,
    AutoTokenizer as cAutoTokenizer,
)

model = cAutoModelForCausalLM.from_pretrained(
            model_path_or_repo_id="TheBloke/Mistral-7B-OpenOrca-GGUF", 
            model_file="mistral-7b-openorca.Q5_K_M.gguf", 
            model_type="mistral",
            hf=True,
            temperature=0.7,
            top_p=0.7,
            top_k=50,
            repetition_penalty=1.2,
            context_length=8096,
            max_new_tokens=2048,
            threads=os.cpu_count(),
            stream=True,
            gpu_layers=0
            )
tokenizer = cAutoTokenizer.from_pretrained(model)

mistral_no_mem_prompt_template = """
<|im_start|>system
The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.
<|im_end|>
{placeholder}
"""

mistral_openorca_prompt = """
<|im_start|>user
{input}<|im_end|>
<|im_start|>assistant
"""

mistral_no_mem_template = mistral_no_mem_prompt_template.replace("{placeholder}", mistral_openorca_prompt)
question = "The cafeteria had 23 apples. If they used 20 for lunch and bought 6 more, how many apples do they have?"
prompt = mistral_no_mem_template.replace("{input}", question)

input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cpu")
generated_ids = model.generate(input_ids, max_new_tokens=2048, temperature=0.7, do_sample=True)
response = tokenizer.decode(generated_ids[0][input_ids.shape[-1]:], skip_special_tokens=True, clean_up_tokenization_space=True)
print(f"Response: {response}")

ok, I quickly write this up and it works fine (you will need transformers==4.34.0 then build ctransformers from #155 and install) ``` import os from ctransformers import ( AutoModelForCausalLM as cAutoModelForCausalLM, AutoTokenizer as cAutoTokenizer, ) model = cAutoModelForCausalLM.from_pretrained( model_path_or_repo_id="TheBloke/Mistral-7B-OpenOrca-GGUF", model_file="mistral-7b-openorca.Q5_K_M.gguf", model_type="mistral", hf=True, temperature=0.7, top_p=0.7, top_k=50, repetition_penalty=1.2, context_length=8096, max_new_tokens=2048, threads=os.cpu_count(), stream=True, gpu_layers=0 ) tokenizer = cAutoTokenizer.from_pretrained(model) mistral_no_mem_prompt_template = """ <|im_start|>system The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. <|im_end|> {placeholder} """ mistral_openorca_prompt = """ <|im_start|>user {input}<|im_end|> <|im_start|>assistant """ mistral_no_mem_template = mistral_no_mem_prompt_template.replace("{placeholder}", mistral_openorca_prompt) question = "The cafeteria had 23 apples. If they used 20 for lunch and bought 6 more, how many apples do they have?" prompt = mistral_no_mem_template.replace("{input}", question) input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cpu") generated_ids = model.generate(input_ids, max_new_tokens=2048, temperature=0.7, do_sample=True) response = tokenizer.decode(generated_ids[0][input_ids.shape[-1]:], skip_special_tokens=True, clean_up_tokenization_space=True) print(f"Response: {response}") ```

❤️ 3 🚀 1

CHesketh76 commented

2023-10-05 15:20:36 -04:00

(Migrated from github.com)

Still having issues with tokenizer = cAutoTokenizer.from_pretrained(model) but using Open-Orca/Mistral-7B-OpenOrca for the tokenizer appears to resolved it. I am not too happy about the speed though. When using lllm =cAutoModelForCausalLM.from_pretrained(...) and then llm('Tell me a story about a knight') it will generate a full story in 10-24 seconds (200-800 tokens). But when using the generate function it takes about 15 minutes to generate 200 tokens. I am using a 3070ti, for reference

Still having issues with ```tokenizer = cAutoTokenizer.from_pretrained(model)``` but using ```Open-Orca/Mistral-7B-OpenOrca``` for the tokenizer appears to resolved it. I am not too happy about the speed though. When using ```lllm =cAutoModelForCausalLM.from_pretrained(...)``` and then ```llm('Tell me a story about a knight')``` it will generate a full story in 10-24 seconds (200-800 tokens). But when using the generate function it takes about 15 minutes to generate 200 tokens. I am using a 3070ti, for reference

CHesketh76 commented

2023-10-05 15:47:47 -04:00

(Migrated from github.com)

So i get x15 faster token output by having no gpu layers.... I think something is wrong

👍 1

victorlee0505 commented

2023-10-06 10:33:18 -04:00

(Migrated from github.com)

yes, something is wrong, for me, gpu_layer has no effect.😅

I found if I build it myself, gpu_layer does not work, no idea why.

yes, something is wrong, for me, gpu_layer has no effect.😅 I found if I build it myself, gpu_layer does not work, no idea why.

victorlee0505 commented

2023-10-06 17:46:56 -04:00

(Migrated from github.com)

I think my lib was a bit messy yesterday. I copy get_vocab from transformers and pushed to the PR #155 . I test with open ocra mistral code from above (type mistral) and exact same code but switch the model to vicuna 1.5 gguf (type llama) and also works.
@CHesketh76 can you rebuild and give it a try?

model = cAutoModelForCausalLM.from_pretrained(
            model_path_or_repo_id="TheBloke/vicuna-13B-v1.5-16K-GGUF", 
            model_file="vicuna-13b-v1.5-16k.Q6_K.gguf", 
            model_type="llama",
            hf=True,
            temperature=0.7,
            top_p=0.7,
            top_k=50,
            repetition_penalty=1.2,
            context_length=8096,
            max_new_tokens=2048,
            threads=os.cpu_count(),
            stream=True,
            gpu_layers=0
            )

I think my lib was a bit messy yesterday. I copy get_vocab from transformers and pushed to the PR #155 . I test with open ocra mistral code from above (type mistral) and exact same code but switch the model to vicuna 1.5 gguf (type llama) and also works. @CHesketh76 can you rebuild and give it a try? ``` model = cAutoModelForCausalLM.from_pretrained( model_path_or_repo_id="TheBloke/vicuna-13B-v1.5-16K-GGUF", model_file="vicuna-13b-v1.5-16k.Q6_K.gguf", model_type="llama", hf=True, temperature=0.7, top_p=0.7, top_k=50, repetition_penalty=1.2, context_length=8096, max_new_tokens=2048, threads=os.cpu_count(), stream=True, gpu_layers=0 ) ```

Girrajjangid commented

2023-10-11 02:31:23 -04:00

(Migrated from github.com)

@victorlee0505 how to rebuilt #155 ?

victorlee0505 commented

2023-10-11 09:34:16 -04:00

(Migrated from github.com)

@victorlee0505 how to rebuilt #155 ?

pip uninstall ctransformers

straight from my fork

pip install --no-cache-dir git+https://github.com/victorlee0505/ctransformers.git@vlee/transformers#egg=ctransformers[cuda]

(even tho i put [cuda], it does not work 😕)

local

git clone https://github.com/victorlee0505/ctransformers.git
cd ctransformers
git checkout vlee/transformers

# I use venv
python -m venv .venv
pip install scikit-build
pip install cmake
python setup.py sdist

under the folder dist, you will have your new package, get the full path and install

pip install --no-cache-dir full\path\to\ctransformers\dist\ctransformers-0.2.27.tar.gz[cuda]

> @victorlee0505 how to rebuilt #155 ? ``` pip uninstall ctransformers ``` straight from my fork ``` pip install --no-cache-dir git+https://github.com/victorlee0505/ctransformers.git@vlee/transformers#egg=ctransformers[cuda] ``` (even tho i put [cuda], it does not work 😕) local ``` git clone https://github.com/victorlee0505/ctransformers.git cd ctransformers git checkout vlee/transformers # I use venv python -m venv .venv pip install scikit-build pip install cmake python setup.py sdist ``` under the folder dist, you will have your new package, get the full path and install ``` pip install --no-cache-dir full\path\to\ctransformers\dist\ctransformers-0.2.27.tar.gz[cuda] ```

kirill578 commented

2023-10-22 23:46:00 -04:00

(Migrated from github.com)

make sure to run export CT_CUBLAS=ON before python setup.py sdist otherwise it won't build the cuda support.

you might also need to setup these two in your bashrc and confirm the nvcc version matches nvidia-smi

export PATH="/usr/local/cuda/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda/lib64"

make sure to run `export CT_CUBLAS=ON` before `python setup.py sdist` otherwise it won't build the cuda support. you might also need to setup these two in your bashrc and confirm the `nvcc` version matches `nvidia-smi` ``` export PATH="/usr/local/cuda/bin:$PATH" export LD_LIBRARY_PATH="/usr/local/cuda/lib64" ```

🚀 1

heiko-braun commented

2023-10-27 05:34:49 -04:00

(Migrated from github.com)

Hi @victorlee0505 . I've rebuilt with PR https://github.com/marella/ctransformers/pull/155 and can confirm, the NotImplementedError is gone. Thanks!

Hi @victorlee0505 . I've rebuilt with PR https://github.com/marella/ctransformers/pull/155 and can confirm, the `NotImplementedError` is gone. Thanks!

👍 1 🚀 1

victorlee0505 commented

2023-10-27 09:30:35 -04:00

(Migrated from github.com)

I won't move forward with this PR, I don't think it is a good fix, but ok to use as is.

I only copy one of the def get_vocab(self): implementation from transformers related to transformers.models.llama.tokenization_llama.LlamaTokenizer.get_vocab. There are different get_vocab implementation for different type. search def get_vocab(self): in transformers you will see what i mean.

Therefore I can not guarantee nor have time to figure out the perfect solution😥

I won't move forward with this PR, I don't think it is a good fix, but ok to use as is. I only copy one of the def get_vocab(self): implementation from transformers related to transformers.models.llama.tokenization_llama.LlamaTokenizer.get_vocab. There are different get_vocab implementation for different type. search `def get_vocab(self):` in transformers you will see what i mean. Therefore I can not guarantee nor have time to figure out the perfect solution😥

pechaut78 commented

2023-11-07 07:09:05 -05:00

(Migrated from github.com)

ok, no I do not get the error on this:

tokenizer = AutoTokenizer.from_pretrained(model)
but now on:

model_inputs = tokenizer([text], return_tensors="pt")

whit this error:

File "/home/pechaut/miniconda3/envs/cairo-llm/lib/python3.9/site-packages/transformers/tokenization_utils.py", line 803, in _batch_encode_plus
first_ids = get_input_ids(ids)
File "/home/pechaut/miniconda3/envs/cairo-llm/lib/python3.9/site-packages/transformers/tokenization_utils.py", line 770, in get_input_ids
tokens = self.tokenize(text, **kwargs)
File "/home/pechaut/miniconda3/envs/cairo-llm/lib/python3.9/site-packages/transformers/tokenization_utils.py", line 617, in tokenize
tokenized_text.extend(self._tokenize(token))
File "/home/pechaut/miniconda3/envs/cairo-llm/lib/python3.9/site-packages/transformers/tokenization_utils.py", line 628, in _tokenize
raise NotImplementedError
NotImplementedError

ok, no I do not get the error on this: `tokenizer = AutoTokenizer.from_pretrained(model) ` but now on: `model_inputs = tokenizer([text], return_tensors="pt")` whit this error: > File "/home/pechaut/miniconda3/envs/cairo-llm/lib/python3.9/site-packages/transformers/tokenization_utils.py", line 803, in _batch_encode_plus first_ids = get_input_ids(ids) File "/home/pechaut/miniconda3/envs/cairo-llm/lib/python3.9/site-packages/transformers/tokenization_utils.py", line 770, in get_input_ids tokens = self.tokenize(text, **kwargs) File "/home/pechaut/miniconda3/envs/cairo-llm/lib/python3.9/site-packages/transformers/tokenization_utils.py", line 617, in tokenize tokenized_text.extend(self._tokenize(token)) File "/home/pechaut/miniconda3/envs/cairo-llm/lib/python3.9/site-packages/transformers/tokenization_utils.py", line 628, in _tokenize raise NotImplementedError NotImplementedError

Rows
Columns

transformers 4.34 caused NotImplementedError when calling CTransformersTokenizer(PreTrainedTokenizer) #154