They’re saying who they are, what they do, and are linking to their website and sometimes sends hundreds of requests in a minute. It might not say "For only €49.99 you can get your very own thing!”, but that does not mean they aren’t throwing their name up in every website owner’s arse whether they like it or not.
That could have been done by just having a single entry called GoogleBot or BingBot, not an entire sentence explaining their product offering let alone hundreds of times a minute.
lmao that’s not an ad, dude.
They’re saying who they are, what they do, and are linking to their website and sometimes sends hundreds of requests in a minute. It might not say "For only €49.99 you can get your very own thing!”, but that does not mean they aren’t throwing their name up in every website owner’s arse whether they like it or not.
It’s so you know who is scanning you.
That could have been done by just having a single entry called GoogleBot or BingBot, not an entire sentence explaining their product offering let alone hundreds of times a minute.