@[email protected] to [email protected]English • 1 year agoX updates its Terms to prohibit crawling/scraping of its datastackdiary.commessage-square48fedilinkarrow-up1361file-textcross-posted to: [email protected]
arrow-up1340external-linkX updates its Terms to prohibit crawling/scraping of its datastackdiary.com@[email protected] to [email protected]English • 1 year agomessage-square48fedilinkfile-textcross-posted to: [email protected]
minus-square@[email protected]linkfedilinkEnglish6•1 year agoTook a look at their robots.txt, it appears to block all bots except Google.
Just update robot.txt coward!
Took a look at their robots.txt, it appears to block all bots except Google.