Every other week, there is someone voicing that curl is far too complex and could be rewritten on a weekend. Ok, maybe a long weekend. So far, there is a complete lack of evidence of this and the persons voicing such expert opinions usually never intend to follow up on this.
If you have a shred of empathy, you can probably feel the pain such statements cause to the authors of curl and Daniel Stenberg, the inventor and decade long maintainer, in particular.
Well, this is the society we created for ourselves, and people are people. Maybe we all need to be more like The Dude:
Today, I heard about some peoples efforts to actually do it. And not just a rewrite in a modern language, no. They provide automated rewrites in 15 languages. Wow!
I am speaking of https://curlconverter.com/, a web interface where you paste your curl command and get the implementation in the language you choose from. That is certainly amazing, so I immediately went over and gave it a try!
curl ftp://host.com
--- python --->
import requests
response = requests.get('http://ftp://host.com')
Uhm, ok. Maybe it is only intended for http urls.
curl --http2 https://host.com
--- python --->
import requests
response = requests.get('https://host.com/')
But I said I wanted HTTP/2, which is not supported by Python's requests library. So, it's not exactly converting a curl command but gives you something which does almost the same thing?
curl --proxy http://myproxy.com https://host.com
--- python --->
import requests
response = requests.get('https://host.com/')
This will not work. And no warning that the "translation" is incomplete is given. This becomes troublesome.
Takes a few steps back.
Maybe they tested this only on the curl commands that are generated by the browsers menu?
That must be it. Let's try that! Opening firefox and copying as Curl Address into the webform:
curl 'https://eissing.org/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15;
rv:97.0) Gecko/20100101 Firefox/97.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,
image/avif,image/webp,*/*;q=0.8' -H 'Accept-Language: de,en-US;q=0.7,en;q=0.3' -H 'Accept-Encoding: gzip, deflate, br'
-H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1'
-H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none'
-H 'Sec-Fetch-User: ?1' -H 'DNT: 1' -H 'If-Modified-Since: Thu, 09 Dec 2021 11:39:19 GMT'
-H 'If-None-Match: "82a-5d2b50e9347c0-gzip"' -H 'Cache-Control: max-age=0' -H 'TE: trailers'
--- python --->
import requests
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:97.0) Gecko/20100101 Firefox/97.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
'Accept-Language': 'de,en-US;q=0.7,en;q=0.3',
'Accept-Encoding': 'gzip, deflate, br',
'Connection': 'keep-alive',
'Upgrade-Insecure-Requests': '1',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'none',
'Sec-Fetch-User': '?1',
'DNT': '1',
'If-Modified-Since': 'Thu, 09 Dec 2021 11:39:19 GMT',
'If-None-Match': '"82a-5d2b50e9347c0-gzip"',
'Cache-Control': 'max-age=0',
'TE': 'trailers',
}
response = requests.get('https://eissing.org/', headers=headers)
Ok, so they understand the Headers options. Nice!
But, wait a minute! why did they copy 'TE: trailers'? This header, when set by a client, indicates that it understands the part of the HTTP protocol that is concerned with headers at the end of a response, e.g. the trailers. Which Firefox and curl do but Python's requests library does not implement (see the issue).
Oh my.
https://curlconverter.com/ does not understand anything about the semantics of curl commands - e.g. what they are supposed to do. It is a dumb template engine and not even a good one at that. Its tendency to ignore unknown/unsupported curl options without any error is definitely wrong.
Why am I writing this? Well, as a server developer and dabbler in the HTTP protocol, I feel strongly about misuse of the protocol. Sending a wrong TE: trailers header does hurt everyone doing HTTP. We need to rely on such information from the client to make meaningful decisions. This does damage.
The aspect that gets me worried is that curlconvert provides code. And that code will get copied into products somewhere - you know it will. And soon, we have billions of IoT or other devices that send junk (well, even more junk). And it was all so easy and convenient to setup!
I do not want to speculate what goals the authors of curlconverter want to achieve. If they read this, I hope they can see the dangers they give into the hands of people. I wish they'd add checks on the parts the tool does not understand or is unable to convert correctly.
Then it could really give knowledge to users: "we cannot convert this command to language X, since X does not support Y and Z!"