By default, crawlers are not dissuaded from indexing FreeFlarum forums
Small forums (the primary target for FreeFlarum) would usually want to be discarded by search engines altogether, because forums for e.g. a few friends won’t have the need to be listed - let one indexed - at all.
With the recent influx of AI crawlers, people may have additional worries that their forums will be used for AI datasets. While robots.txt isn’t a cure for all of those crawlers, there sure are lots of crawlers that do adhere to this.

I propose the addition of an input field in the FreeFlarum settings to allow the user to set up their own robots.txt. You could also provide a few presets, such as:

Disallow all:

User-agent: *
Disallow: /

Don’t index user details:

User-agent: *
Disallow: /u/

Disallow AI crawlers:

User-agent: CCBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: FacebookBot
Disallow: /
User-Agent: Applebot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: YouBot
Disallow: /

(source: https://www.cyberciti.biz/web-developer/block-openai-bard-bing-ai-crawler-bots-using-robots-txt-file/)

While the addition of a robots.txt file may be possible via the SEO extension, using this extension has multiple disadvantages that actually go against the entire goal here (additional privacy):

The robots.txt file isn’t completely configurable (e.g. it always adds a reference to sitemap.xml)
Additional metadata is always added to the pages (this can’t be disabled)