regex stopping condition

[jancervenka]

Motivation

Hi @merrymercy! I am interested in contributing to the SGLang project so I gave this issue a shot: #2007 Is this a sensible approach? I am a very new to the project so any pointers are welcomed.

I will add tests and update the docs once the change looks ok to you. I am actually having troubles getting the project running on my machine so I haven't been able to test it yet.

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[merrymercy]

1. It is better to implement it here

   sglang/python/sglang/srt/managers/schedule_batch.py

   Line 325 in a10d530

   def check_finished(self):

2. Can you add a new argument for this stop condition, so we do not make the old simple string match slower? You can add a new field at SamplingParameters

   sglang/python/sglang/srt/sampling/sampling_params.py

   Line 23 in a10d530

   class SamplingParams:

[jancervenka]

thank you @merrymercy. In Req.check_finished, is it ok to ignore tail_str and just check decoded_text? Because we don't know how many tokens to decode to get a regex match.

[merrymercy]

yes

[jancervenka]

hi @merrymercy I was testing my change and I don't think using just decoded_text works. When I tried it with the following debug logging, it shows that decoded_text is empty throughout the entire inference.

We could reuse tail_str and instead of overwriting it every time, append each tail_str to a string and regex check the string? And if there is no stop_str_max_len to produce tail_str, we could just decode and append the last token?

        for stop_regex_str in self.sampling_params.stop_regex_strs:
            logger.debug(f"stop_regex='{stop_regex_str}' decoded_text_length={len(self.decoded_text)}")
            if re.search(stop_regex_str, self.decoded_text):
                self.finished_reason = FINISH_MATCHED_STR(matched=stop_regex_str)
                return

[jancervenka]

hi @merrymercy, tried to solve the problem by decoding one token at a time. Thank you for any feedback!

[merrymercy]

You cannot decode text token by token and concatenate the string. This will lead to wrong outputs.

[jancervenka]

Oh right, I didn't know there are tokenizers where this doesn't work. Is it then ok to decode the entire output each time? Or decode a fixed window and accept that it's not 100% reliable?

[merrymercy]

move to #2699