bigcode-project/starcoder
1
0
Fork 0
Code Issues 93 Pull requests 8 Projects Releases Packages Wiki Activity Actions

torch.cuda.OutOfMemoryError on HuhhingFace NVidia 4xA10G Large #151

New issue
Open
opened 2024-02-09 01:09:30 -05:00 by lkthomas · 2 comments
lkthomas commented 2024-02-09 01:09:30 -05:00 (Migrated from github.com)
Copy link

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 288.00 MiB. GPU 0 has a total capacty of 21.99 GiB of which 59.00 MiB is free. Process 42083 has 21.92 GiB memory in use. Of the allocated memory 21.59 GiB is allocated by PyTorch, and 47.14 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I am trying to run StarCoder and still getting this error after using Nvidia 4xA10G Large, it doesn't seems utilize VRAM more than 1 GPU, how could I fix this ?

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 288.00 MiB. GPU 0 has a total capacty of 21.99 GiB of which 59.00 MiB is free. Process 42083 has 21.92 GiB memory in use. Of the allocated memory 21.59 GiB is allocated by PyTorch, and 47.14 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF I am trying to run StarCoder and still getting this error after using Nvidia 4xA10G Large, it doesn't seems utilize VRAM more than 1 GPU, how could I fix this ?
loubnabnl commented 2024-02-10 20:15:14 -05:00 (Migrated from github.com)
Copy link

Can you try using the code in this section https://github.com/bigcode-project/starcoder?tab=readme-ov-file#inference-hardware-requirements this loads the model in ~16GB of RAM. Also the device_map="auto" is needed to dispatch the model on multiple GPUs.

Can you try using the code in this section https://github.com/bigcode-project/starcoder?tab=readme-ov-file#inference-hardware-requirements this loads the model in ~16GB of RAM. Also the `device_map="auto"` is needed to dispatch the model on multiple GPUs.
lkthomas commented 2024-02-10 20:33:35 -05:00 (Migrated from github.com)
Copy link

Any idea how to load this model on hugging face Spaces? On 11 Feb 2024, at 8:15 AM, Loubna Ben Allal @.***> wrote:
Can you try using the code in this section https://github.com/bigcode-project/starcoder?tab=readme-ov-file#inference-hardware-requirements this loads the model in ~16GB of RAM

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

Any idea how to load this model on hugging face Spaces? On 11 Feb 2024, at 8:15 AM, Loubna Ben Allal ***@***.***> wrote: Can you try using the code in this section https://github.com/bigcode-project/starcoder?tab=readme-ov-file#inference-hardware-requirements this loads the model in ~16GB of RAM —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
loubnabnl-patch-2
lewtun-patch-1
loubnabnl-patch-1
arjunguha-readme
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
bigcode-project/starcoder#151
No description provided.