How to resolve the “Too Many Requests” issue encountered when using the OpenAI API?

[Here1sWqW]

The error message is as follows:

(base) wangweiqin@yile-188-188:/mnt/d/wangweiqin/idea$ '/mnt/d/wangweiqin/idea/TasksWithResults/CSQA/openai/gpt-3.5-turbo-0125/gpt-3.5-turbo-0125/4_sc/run.sh'
2024-12-24:09:06:13,575 INFO     [__main__.py:279] Verbosity set to INFO
2024-12-24:09:06:19,264 INFO     [__main__.py:376] Selected Tasks: ['CSQA-fewshot-cot-self-consistency_gpt']
2024-12-24:09:06:19,266 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-12-24:09:06:19,266 INFO     [evaluator.py:201] Initializing openai-chat-completions model, with arguments: {'model': 'gpt-3.5-turbo-0125'}
2024-12-24:09:06:19,266 WARNING  [openai_completions.py:106] chat-completions endpoint requires the `--apply_chat_template` flag.
2024-12-24:09:06:19,266 INFO     [api_models.py:111] Using max length 2048 - 1
2024-12-24:09:06:19,266 INFO     [api_models.py:114] Concurrent requests are disabled. To enable concurrent requests, set `num_concurrent` > 1.
2024-12-24:09:06:19,266 INFO     [api_models.py:124] Using tokenizer None
2024-12-24:09:06:22,805 INFO     [task.py:415] Building contexts for CSQA-fewshot-cot-self-consistency_gpt on rank 0...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1221/1221 [00:13<00:00, 87.64it/s]
2024-12-24:09:06:36,764 INFO     [evaluator.py:489] Running generate_until requests
Requesting API:   0%|▏                                                                                                                                                                                                             | 8/9768 [00:05<1:58:14,  1.38it/s]2024-12-24:09:06:43,182 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409064294484440526590781)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
2024-12-24:09:06:44,723 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409064446988062461614496)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   0%|▏                                                                                                                                                                                                            | 11/9768 [00:11<3:07:42,  1.15s/it]2024-12-24:09:06:48,359 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409064810514835050260300)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   0%|▎                                                                                                                                                                                                            | 13/9768 [00:14<3:24:33,  1.26s/it]2024-12-24:09:06:51,333 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 202412240906519473059433755784)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   0%|▌                                                                                                                                                                                                            | 26/9768 [00:25<2:04:08,  1.31it/s]2024-12-24:09:07:02,604 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409070236450019447358345)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   0%|▋                                                                                                                                                                                                            | 33/9768 [00:32<2:10:26,  1.24it/s]2024-12-24:09:07:09,403 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409070915765366516382149)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   0%|▉                                                                                                                                                                                                            | 46/9768 [00:44<2:09:54,  1.25it/s]2024-12-24:09:07:21,524 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409072128397748121349092)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   1%|█▏                                                                                                                                                                                                           | 58/9768 [00:56<2:23:35,  1.13it/s]2024-12-24:09:07:33,489 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409073324413875827919436)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   1%|█▏                                                                                                                                                                                                           | 59/9768 [00:58<3:35:24,  1.33s/it]2024-12-24:09:07:35,863 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409073560778743599911171)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   1%|█▌                                                                                                                                                                                                           | 76/9768 [01:13<2:18:52,  1.16it/s]2024-12-24:09:07:51,248 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 202412240907511153427022549684)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   1%|█▋                                                                                                                                                                                                           | 78/9768 [01:17<3:21:11,  1.25s/it]2024-12-24:09:07:54,771 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409075453057092444286576)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   1%|█▋                                                                                                                                                                                                           | 79/9768 [01:19<4:12:50,  1.57s/it]2024-12-24:09:07:57,071 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409075683535840035000481)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   1%|█▋                                                                                                                                                                                                           | 82/9768 [01:23<3:30:45,  1.31s/it]2024-12-24:09:08:01,104 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409080086546088483260494)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   1%|█▊                                                                                                                                                                                                           | 84/9768 [01:26<3:38:57,  1.36s/it]2024-12-24:09:08:04,176 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409080393066674277144636)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   1%|██                                                                                                                                                                                                           | 98/9768 [01:38<2:01:57,  1.32it/s]2024-12-24:09:08:16,263 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 202412240908161106394662387258)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   1%|██                                                                                                                                                                                                           | 99/9768 [01:41<3:11:22,  1.19s/it]2024-12-24:09:08:18,453 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409081820810273247665885)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Requesting API:   1%|██▏                                                                                                                                                                                                         | 102/9768 [01:44<2:56:36,  1.10s/it]2024-12-24:09:08:22,096 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409082185194154112454277)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
2024-12-24:09:08:23,603 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409082337066670226020003)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
2024-12-24:09:08:25,130 WARNING  [api_models.py:347] API request failed with error message: {"error":{"message":"TooManyRequests (request id: 2024122409082488658550647599855)","type":"internal_error","param":"","code":"yesai_api_error"}}. Retrying...
Traceback (most recent call last):
  File "/home/wangweiqin/miniconda3/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/mnt/d/wangweiqin/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/wangweiqin/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/wangweiqin/lm-evaluation-harness/lm_eval/evaluator.py", line 301, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/mnt/d/wangweiqin/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/wangweiqin/lm-evaluation-harness/lm_eval/evaluator.py", line 500, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/wangweiqin/lm-evaluation-harness/lm_eval/models/api_models.py", line 569, in generate_until
    outputs = retry(
              ^^^^^^
  File "/home/wangweiqin/miniconda3/lib/python3.12/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/wangweiqin/miniconda3/lib/python3.12/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangweiqin/miniconda3/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/wangweiqin/miniconda3/lib/python3.12/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/home/wangweiqin/miniconda3/lib/python3.12/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangweiqin/miniconda3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/wangweiqin/miniconda3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/wangweiqin/miniconda3/lib/python3.12/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/wangweiqin/lm-evaluation-harness/lm_eval/models/api_models.py", line 350, in model_call
    response.raise_for_status()
  File "/home/wangweiqin/miniconda3/lib/python3.12/site-packages/requests/models.py", line 1067, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: http://15.204.101.64:4000/v1/chat/completions
Requesting API:   1%|██▏                                                                                                                                                                                                         | 102/9768 [01:48<2:51:33,  1.06s/it]

[aalpat1]

What model are you using to run inference on? Looks like a locally hosted model, implementing OpenAI spec, in which case, maybe setting the batch_size in the call to simple_evaluate with the max num of concurrent requests your server can handle should help?