New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slowing down after adding draft model #1301

Open

windkwbs opened this issue Jan 6, 2025 · 1 comment

windkwbs commented Jan 6, 2025

The purpose of the draft model is to speed up, but the current test effect is slower. Is the implementation method incorrect?

windkwbs changed the title ~~The problem of adding a draft model slower than a single model~~ Slowing down after adding draft model

Owner

LostRuins commented Jan 18, 2025

The implementation is probably correct, however the draft can be slow if it's unable to predict the correct tokens for your output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment