Tokenizer not invertible #163

New issue

Open

opened 2024-10-14 14:13:42 -04:00 by t-montes · 0 comments

t-montes commented 2024-10-14 14:13:42 -04:00 (Migrated from github.com)

Copy link

When encoding " ..." through the tokenizer, it's encoded as " ..." (one missing space), but tokenizers should be always 100% invertible, aren't they?

When encoding `" ..."` through the tokenizer, it's encoded as `" ..."` (one missing space), but tokenizers should be always 100% invertible, aren't they?

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

loubnabnl-patch-2

lewtun-patch-1

loubnabnl-patch-1

arjunguha-readme

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

bigcode-project/starcoder#163

No description provided.