@[email protected] to Lemmy [email protected] • 1 year agoChad scrapersh.itjust.worksmessage-square96fedilinkarrow-up11.05K
arrow-up11.02KimageChad scrapersh.itjust.works@[email protected] to Lemmy [email protected] • 1 year agomessage-square96fedilink
minus-square@[email protected]linkfedilink73•1 year agoJust a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.
Just a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.
But, but, norobots.txt!
Chad doesn’t care!
uses curl manually instead