common: set CURLOPT_UNRESTRICTED_AUTH during http requests

[Khady]

Description of the task

Sometimes people can post a link with a wrong scheme by accident. In this case the github api is doing the right thing, returning a redirection. But when curl follows this redirection it will drop the Authorization header.

https://curl.se/libcurl/c/CURLOPT_UNRESTRICTED_AUTH.html

This option should be used with caution: when curl follows redirects it blindly fetches the next URL as instructed by the server. Setting CURLOPT_UNRESTRICTED_AUTH to 1L makes curl trust the server and sends possibly sensitive credentials to any host the server points to, possibly again and again as the following hosts can keep redirecting to new hosts.

How to test

Setup a tiny http server which does a redirection and logs the headers it receives

from http.server import HTTPServer, BaseHTTPRequestHandler
from urllib.parse import urlparse

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        path = urlparse(self.path).path
        if path == '/redir':
            self.send_response(302)
            self.send_header('Location', 'http://127.0.0.1:8000/demo')
            self.end_headers()
        elif path == '/demo':
            print(self.headers)
            self.send_response(200)
            self.end_headers()

HTTPServer(('', 8000), Handler).serve_forever()

Start it with python3 demo.py

From utop (dune utop):

# Lib.Common.http_request ~headers:["x-test: demo"; "Accept: application/vnd.github.v3+json"; "Authorization: token louis" ] `GET "localhost:8000/redir";;
[2024-08-07T05:28:30.4332] 2467156: [lib:info] GET #2 localhost:8000/redir 
[2024-08-07T05:28:30.4356] 2467156: [lib:info] GET #2 2.3ms ⇓0B ⇑0B 127.0.0.1 HTTP 200 http://127.0.0.1:8000/demo after 1 redirects
- : (string, string) result = Ok ""

With the change from this PR you'll see Authorization: token louis. With the code from master the header is missing

$ python3 demo.py 
127.0.0.1 - - [07/Aug/2024 05:28:30] "GET /redir HTTP/1.1" 302 -
Host: 127.0.0.1:8000
User-Agent: monorobot
Accept-Encoding: deflate, gzip, br, zstd
x-test: demo
Accept: application/vnd.github.v3+json
Authorization: token louis


127.0.0.1 - - [07/Aug/2024 05:28:30] "GET /demo HTTP/1.0" 200 -

This behavior can be reproduced with the curl cli

curl --location-trusted -i -v -L -H "Authorization: token placeholder" "localhost:8000/redir"

--location-trusted is the same as CURLOPT_UNRESTRICTED_AUTH

[rr0gi]

8928db1#commitcomment-145139457 this proposal makes this PR not needed?

[Khady]

superseded by 3c18db3