fix mask_user_labels

DoffeBupt commented

2023-08-27 03:09:43 -04:00

(Migrated from github.com)

Dear authors of Starcoder, while using the framework, I noticed that mask_user_labels sometimes does not function properly. Upon investigation, I found that there might be an issue with the function invocation. Here are my modifications：

In the group_texts() function in train.py, the result["input_ids"] has a structure of lists nested within lists. However, mask_user_labels deals with a single-layered list structure. Therefore, mask_user_labels did not function as expected. I used a loop to process each label separately, allowing the function to be called normally.
In the mask_user_labels function, I added masking for system-related labels. Moreover, the condition current_idx < len(labels) in the while loop should be placed at the beginning; otherwise, this condition becomes meaningless and can lead to out-of-bounds access with labels[current_idx].

Dear authors of Starcoder, while using the framework, I noticed that mask_user_labels sometimes does not function properly. Upon investigation, I found that there might be an issue with the function invocation. Here are my modifications： 1. In the group_texts() function in train.py, the result["input_ids"] has a structure of lists nested within lists. However, mask_user_labels deals with a single-layered list structure. Therefore, mask_user_labels did not function as expected. I used a loop to process each label separately, allowing the function to be called normally. 2. In the mask_user_labels function, I added masking for system-related labels. Moreover, the condition current_idx < len(labels) in the while loop should be placed at the beginning; otherwise, this condition becomes meaningless and can lead to out-of-bounds access with labels[current_idx].

wanglongxingtianxia commented

2023-08-28 05:37:12 -04:00

(Migrated from github.com)

def group_texts(examples):
        # Concatenate all texts.
        # print(type(examples))
        concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}
        total_length = len(concatenated_examples[list(examples.keys())[0]])
        # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
        # customize this part to your needs.
        if total_length >= block_size:
            total_length = (total_length // block_size) * block_size
        # Split by chunks of max_len.
        labels = concatenated_examples["input_ids"].copy()
        mask_user_labels(tokenizer, dialogue_template, labels)
        concatenated_examples['labels'] = labels
        result = {
            k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
            for k, t in concatenated_examples.items()
        }
        return result

Can we change this?

``` def group_texts(examples): # Concatenate all texts. # print(type(examples)) concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()} total_length = len(concatenated_examples[list(examples.keys())[0]]) # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can # customize this part to your needs. if total_length >= block_size: total_length = (total_length // block_size) * block_size # Split by chunks of max_len. labels = concatenated_examples["input_ids"].copy() mask_user_labels(tokenizer, dialogue_template, labels) concatenated_examples['labels'] = labels result = { k: [t[i : i + block_size] for i in range(0, total_length, block_size)] for k, t in concatenated_examples.items() } return result ``` Can we change this?

DoffeBupt commented

2023-08-28 05:48:37 -04:00

(Migrated from github.com)

def group_texts(examples):
        # Concatenate all texts.
        # print(type(examples))
        concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}
        total_length = len(concatenated_examples[list(examples.keys())[0]])
        # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
        # customize this part to your needs.
        if total_length >= block_size:
            total_length = (total_length // block_size) * block_size
        # Split by chunks of max_len.
        labels = concatenated_examples["input_ids"].copy()
        mask_user_labels(tokenizer, dialogue_template, labels)
        concatenated_examples['labels'] = labels
        result = {
            k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
            for k, t in concatenated_examples.items()
        }
        return result

Can we change this?

change
mask_user_labels(tokenizer, dialogue_template, labels)
to
for label in labels:
mask_user_labels(tokenizer, dialogue_template, label)

I think this should make sense

> ``` > def group_texts(examples): > # Concatenate all texts. > # print(type(examples)) > concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()} > total_length = len(concatenated_examples[list(examples.keys())[0]]) > # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can > # customize this part to your needs. > if total_length >= block_size: > total_length = (total_length // block_size) * block_size > # Split by chunks of max_len. > labels = concatenated_examples["input_ids"].copy() > mask_user_labels(tokenizer, dialogue_template, labels) > concatenated_examples['labels'] = labels > result = { > k: [t[i : i + block_size] for i in range(0, total_length, block_size)] > for k, t in concatenated_examples.items() > } > return result > ``` > > Can we change this? change mask_user_labels(tokenizer, dialogue_template, labels) to for label in labels: mask_user_labels(tokenizer, dialogue_template, label) I think this should make sense

wanglongxingtianxia commented

2023-08-28 22:04:31 -04:00

(Migrated from github.com)

concatenated_examples['input_ids'] is one dimensional list

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin DoffeBupt/fix_mask_user_labels:DoffeBupt/fix_mask_user_labels

git switch DoffeBupt/fix_mask_user_labels

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff DoffeBupt/fix_mask_user_labels

git switch DoffeBupt/fix_mask_user_labels

git rebase main

git switch main

git merge --ff-only DoffeBupt/fix_mask_user_labels

git switch DoffeBupt/fix_mask_user_labels

git rebase main

git switch main

git merge --no-ff DoffeBupt/fix_mask_user_labels

git switch main

git merge --squash DoffeBupt/fix_mask_user_labels

git switch main

git merge --ff-only DoffeBupt/fix_mask_user_labels

git switch main

git merge DoffeBupt/fix_mask_user_labels

git push origin main

Rows
Columns

fix mask_user_labels #130

Checkout

Merge