Add robots.txt field to FreeFlarum settings

Flarumist

By default, crawlers are not dissuaded from indexing FreeFlarum forums
Small forums (the primary target for FreeFlarum) would usually want to be discarded by search engines altogether, because forums for e.g. a few friends won’t have the need to be listed - let one indexed - at all.
With the recent influx of AI crawlers, people may have additional worries that their forums will be used for AI datasets. While robots.txt isn’t a cure for all of those crawlers, there sure are lots of crawlers that do adhere to this.

I propose the addition of an input field in the FreeFlarum settings to allow the user to set up their own robots.txt. You could also provide a few presets, such as:

Disallow all:

User-agent: *
Disallow: /

Don’t index user details:

User-agent: *
Disallow: /u/

Disallow AI crawlers:

User-agent: CCBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: FacebookBot
Disallow: /
User-Agent: Applebot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: YouBot
Disallow: /

(source: https://www.cyberciti.biz/web-developer/block-openai-bard-bing-ai-crawler-bots-using-robots-txt-file/)

While the addition of a robots.txt file may be possible via the SEO extension, using this extension has multiple disadvantages that actually go against the entire goal here (additional privacy):

The robots.txt file isn’t completely configurable (e.g. it always adds a reference to sitemap.xml)
Additional metadata is always added to the pages (this can’t be disabled)

SKevo

Flarumist I understand what you mean and this seems like a great idea, thanks for the feedback!

However, I would like to keep the settings panel of FreeFlarum simple: as in, to not add many side features that require additional maintenance just to satisfy a certain portion of the userbase, or a single forum. An example is the existing CSV table export feature, which was requested by one forum, but barely anyone uses it (yet there is a price of maintaining its functionality).

Honestly, I would be happy to rather just setup the robots.txt file manually for an individual forum rather than designating a settings area for it and ensure that there are no security vulnerabilities, that it works properly, etc…
I currently expect that such feature would not be used by many forums, and from a perspective of a free service that I maintain in my spare time, it is easier to process such occasional requests manually. However, I am not saying that it is out of question - I will leave this thread open, and if there proves to be demand of such feature, I will consider implementing it (so far, you’re the only one who brought this particular topic up)

With that being said, feel free to send me an e-mail at info@freeflarum.com with your forum name and your desired robots.txt, and I’ll write the file in the public root of your forum directory for you. Of course, if that’s something that you want and this is not just a generic feedback? Otherwise, I would be glad to discuss the impacts and possible implementation of this further, also with other FreeFlarum administrators

[deleted]

Use SEO extension can do it

Flarumist

[deleted] I already tried the SEO extension for this and it would not fit the needs, as I said in the first post.

Flarumist

SKevo to not add many side features that require additional maintenance

I’m not sure if this would require any maintenance if added (unless you would add that AI preset, but that was just a little extra).

SKevo An example is the existing CSV table export feature

I must say I find it a nice option, was happy to see it available.

SKevo I currently expect that such feature would not be used by many forums, and from a perspective of a free service that I maintain in my spare time, it is easier to process such occasional requests manually.

I don’t think many people who would benefit from it even realize it’s possible if it’s not added to the settings area.

SKevo With that being said, feel free to send me an e-mail at info@freeflarum.com with your forum name and your desired robots.txt, and I’ll write the file in the public root of your forum directory for you.

I just added a hacky workaround: since a /robots.txt request returns a 404, I just put the contents of my desired robots.txt as a custom error page using the FoF HTML Errors extension. It isn’t pretty, but the output is plain text (not HTML), so it does the job well.

SKevo

Flarumist I see. Very well, I’ll have a look at the codebase and see if I can implement something simple that just does the job. I suppose it doesn’t hurt to have an extra textarea that writes to a file.

SKevo

I have now implemented a working prototype at https://freeflarum.com/settings/forum_settings. Thanks for the suggestion, and please let me know if you spot any bugs or issues

Flarumist

SKevo Wow, that was superfast! Looks like it’s working perfectly, thanks so much! 😄

Flarumist

You mention Cloudflare caching by the way, which is good to know when editing.

Just to make sure I know what’s going on: this caching is only because the main FreeFlarum website is proxied by Cloudflare, but there is nothing Cloudflare handles for forums hosted by FreeFlarum, right?

SKevo

Flarumist there is nothing Cloudflare handles for forums hosted by FreeFlarum, right?

No, all forums are proxied through Cloudflare as well. The robots.txt file will be cached, so the changes won’t be immediately reflected on the forum if you edit it.

I can make a rule at Cloudflare to exempt robots.txt from the cache, but not right now. As a workaround, I think it is possible to bypass the cache if you delete the file (e. g.: leave the input blank) and then re-create it (paste back content before deletion)

Flarumist

SKevo No, all forums are proxied through Cloudflare as well.

Oh, that’s a bit confusing, I don’t think that’s what is said in the privacy policy:

Cloudflare, for proxying requests made to the main FreeFlarum website to guard the service against attacks and other potential internet thread[t]s;
https://docs.freeflarum.com/en/legal/privacy-policy/#third-party-services-software-used-by-freeflarum

So, this description implies that only the main website is proxied, not the forums created by users. I think it should be corrected if it’s not the case.

That said, I don’t see explicit Cloudflare requests on my forum (which I do see on this forum) using a custom domain. I saw the following line in the docs:

Make sure to disable the Cloudflare proxy, so that the icon is grey (not orange).
https://docs.freeflarum.com/en/tutorials/custom-domain

I don’t see such an option in the FreeFlarum settings area, so does this mean that the setting is implied if you set up a custom domain? Or do you mean with this that when the custom domain is offered by Cloudflare, the user should set it to unproxied there?
And does this mean that forums using a custom domain are not proxied through Cloudflare?

~~(I apologize that this went a bit off-topic, perhaps the discussion needs to be split now 🙈)~~