Response to
by
James
I just took a quick look at my server logs. And I see that there are some bots that are constantly crawling my site. If that would be search engines, feed readers etc. that wouldn’t be a problem. But this are crawlers by companies whose websites try to se
I wonder if there isn't a list of these cralwers out there that you could then automatically block at the nginx/apache level. Even handling it at the app server level would discourage them from crawling your site too much.