@[email protected] to [email protected]English • 10 months agoX updates its Terms to prohibit crawling/scraping of its datastackdiary.commessage-square50fedilinkarrow-up1361file-textcross-posted to: [email protected]
arrow-up1340external-linkX updates its Terms to prohibit crawling/scraping of its datastackdiary.com@[email protected] to [email protected]English • 10 months agomessage-square50fedilinkfile-textcross-posted to: [email protected]
minus-square@[email protected]linkfedilinkEnglish6•10 months agoTook a look at their robots.txt, it appears to block all bots except Google.
Just update robot.txt coward!
Took a look at their robots.txt, it appears to block all bots except Google.