Flip it around, why would you go to the trouble of creating a web page and content for it, if some AI bot is going to scrape it and save people the trouble of visiting your site? The value of your work has been captured by some AI company (by somewhat nefarious means too).
I don't see the problem. I want AI agents to learn my website and have it available as part of its corpus of knowledge to users. If asked for something it can answer the question, and if user requests make a parallel web search for sources which would bring up my page. The latter is only a bonus not a necessity for me. Getting the information out there by whatever means is my first priority.
