Instead of everyone having to update their robots.txt all the fucking time, is it possible to just link in a trusted common robots.txt that can be kept up to date with the latest AI shit to block for everyone?
@forteller How much faith do people have that the AI crawlers respect that anyway? It’s all on the honors system anyway. Like there’s no technical restriction, and no legal ones thought?
@slembcke Yeah, I know. Which makes it even less likely that too many people will bother updating their robots.txt regularly. I know I haven't even done it once. Which makes the idea of being able to have a common one that you can auto-import into your own even more important, I think.