New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] triton kernel， loss 0， grar-norm nan #6902

Open

mdy666 opened this issue Dec 22, 2024 · 1 comment

Assignees

Labels

mdy666 commented Dec 22, 2024

Describe the bug
i replace the naive kernel in transformers with my triton kernel， after a few steps the loss become 0 and grad_norm become nan

version: deepspeed: 15.4,

i use zero2 train qwen2.5 7B, it's ok when i train qwen2.5 0.5B wihout zero

mdy666 added bug training labels

loadams assigned hwchen2017

Contributor

loadams commented Jan 2, 2025

@mdy666 - can you share more information about your setup, sharing a repro script would help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment