website: Add robots.txt for preview / past builds

[4d62]

Suggestion: Add a robots.txt similar to the following to prevent indexing of all preview/historical docs. Will prevent users getting outdated information.

User-agent: *
Disallow: /

[4d62]

Could be made with something like this if I'm not mistaken:

[[redirects]]
from = "/robots.txt"
to = "/robots-deny.txt"
status = 200
force = true

[[redirects]]
from = "/robots.txt"
to = "/robots-allow.txt"
status = 200
force = true
conditions = { Host = ["docs.goauthentik.io", ":docs.goauthentik.io"] }