Understandably this is something you should make the correct choice for yourself, you might want to be scraped on readonly articles or public pages for things like thumbnails on reddit from EC2, but deny all `HTTPS POST /register` or something from EC2.