> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
> The anubis author has stated they recognize it's an arms race, but PoW scales.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
Well, we don't use a captcha either. If it were a choice between a captcha and a proof of work system, we'd have to reevaluate things. Luckily, for now, we're able to get away with a much lighter touch.
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
Anubis appears to be a temporarily-useful stopgap that has been cargo culted into prominence and an expectation of permanent usefulness, for reasons I don't fully understand.
The cost of solving the default Anubis PoW is negligible on cloud servers, and it's even lower if you use native code rather than JavaScript to solve it, which Tavis Ormandy helpfully demonstrated last year (https://lock.cmpxchg8b.com/anubis.html). If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.
So Anubis doesn't do much to stop bots, but it makes otherwise lightweight websites (little JavaScript or interactivity) almost unusable on low-resource systems like my old phone or an old Atom-based nettop.
> when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape
This "IP-bound proof-of-work" thing is gonna kill multipath TCP and bring down IPv6 with it. Uffff.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
You can't do that any more. Too many ISPs, especially mobile carriers, don't hand out anything resembling a fixed IP address any more. It's CGNAT and constantly changing IP addresses alllll the time now.
PoW can theoretically scale effectively infinite because it can mine cryptocurrency. Millions of compromised IoT devices hitting your server? Now you have enough money for a faster server.
It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
Ironically in early 2023 a lot of websites went out of their way to block Common Crawl. Unsurprisingly that shifted scraping toward individual actors whereas the previous solution in research was to download CC dumps and process them.
I'm sure there are those who would participate, either because they want their data to be captured by AI labs or as a form of compromise.
That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
The article at the end talks about how is very easy for arbitrary apps from app stores can install a residential proxy on your phone.
10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what. And both ios and android make it hard to deny apps network access.
But really, this finally explains the hordes of really basic boring games that just advertise other boring games. Idle games and the like that really just want you to keep your phone unlocked and open. Millions of downloads on the app stores for entirely offline content (and ads) and no way to block the network access.
> 10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what.
Why does network access need to be a binary, all or nothing?
When you install an app, the app should request permissions to specific DNS names, i.e. pointing to the servers that the app's authors operate. If I install Todoist, the app should only ask for access to Todoist's servers. If I install Netflix, the app should only ask for access to Netflix's servers. The OS can then put a DNS firewall in and block any network access that wasn't granted when the app was installed.
> And both ios and android make it hard to deny apps network access
The list of apps that genuinely need "any" network access (web browsers, VPN apps, stuff like Termux...) is incredibly small compared to the list of apps that need access to a small number of VPN targets (these days, most apps). Apple / Google could even decide, if they really want to make it easy for apps to request network access, to basically allow apps to automatically get network access, so long as the list of domains the app needs access to is no more than a handful. The security value of isolating "all" network access permissions to only the relative handful of apps that actually need to request it, would be huge.
GrapheneOS allows you to deny network access per app pretty trivially. Google Play services make it a bit more difficult because the app might marshall the network request through that; I'm not sure how to verify that behavior when it happens.
The Bright Data “free” VPN they’re talking about requires the user to go through steps to enable it.
These aren’t as simple as downloading a free game and then the phone is compromised as long as it’s installed.
The users who install these things don’t care about permissions prompts. They’ll follow instructions to tap any prompt the instructions ask. They want the free thing and don’t care what they have to do to get it.
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!
I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
As this article points out, it's tremendously unclear who is using residential proxies.
The big AI models claim they're not using them. I'm not inclined to "just believe them", but no incriminating evidence has leaked, and—as pointed out in the article—many of the bots that are running on these residential proxy botnets are coded in incredibly stupid and inefficient ways.
How confident are people who research this stuff that the RP botnets are actually being used for AI training?
That's the only theory we have that doesn't sound like a conspiracy theory. The only other credible ideas are that someone's doing some kind of dataset arbitrage by scraping the fuck out of everything and selling companies data that is technically new (by means of the scrape date being newer).
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
And it's made necessary because another group of people thought that selling IP blocking services would be a good idea. One party sells walls, another party sells ladders.
Well, one party gives away free walls if you agree to fill your castle with surveillance cameras you don't control.
This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
i think the best way to keep your site working for legit users is serving static cached pages to suspected bots. decide based off something like cloudflare bot score.
crawlers get redirected to content that might be stale but its still useful for them and costs you almost nothing to serve. put a warning banner on it so if a user (or smart agent) accidentally ends up on the bot version they can click on it and get anubis checked for the real site.
of course this only works for people like me who want their work to be used for training ai.
mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
I don't run one of these sites that has these issues so I'm really not aware of this problem. How can it be that sites are getting overwhelmed with scrapers that are just looking for training data? You only need to scrape it once to train a model, so shouldn't there be less traffic from this than there is from search engines?
On the other hand if the article is wrong and the traffic is coming from other ai uses (like an agent visiting pages on behalf of a user) then that would make sense.
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
> Tracking which devices still send lots of data when no one else is using the network?
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
Opnsense has a traffic capture feature in the interface diagnostics menu, if you want to spot check what servers the devices are currently talking to.
Should be pretty obvious: client devices and internal services will have no traffic >95% of the time, just NTP for timekeeping, DHCP lease renewal, and associated ARP (running total: two dozen packets if you monitor them for a full 24h), then any system updaters (readily identifiable by the initial DNS requests), and finally of course you'll see the traffic of the service that the device hosts, if any, which can be easily dismissed by not looking at incoming connections (scraping uses outgoing connections)
"The startup, which markets itself as the second-largest data collection firm after Alphabet Inc.s Google, has grown significantly on demand from data-hungry AI companies."
I was involved in both sides of this battle over ten years ago. Things haven't changed all that much.
It's important to note that neither side has moral legitimacy. Not everyone who carries a rifle is a enemy. Not everyone wearing body armor is a saint.
I have given up on the idea that "human vs bot" matters at all when it comes to anything other than voting (which should only be done in person with paper and pen, by the way.)
You could make an argument that "likes" are a form of voting, but you shouldn't. We need to abandon the idea of supposedly democratized algorithms and focus instead on actual democracy.
I think this Anubis project is a terrible solution to the problem posed by aggressive web scrapers. Using a web browser with reasonable privacy settings has become a big loss in quality of life already, but the first time I encountered Anubis I got completely locked out of most web servers that deployed it. The situation has improved a little, but I hate that maintainers of great web services have rationalized themselves into believing that creating massive barriers to access their sites is a fair trade-off. Unsurprisingly, I have nothing but negative associations with their mascot.
The FSF has the right idea about all this:
> Some web developers have started integrating a program called Anubis to decrease the amount of requests that automated systems send and therefore help the website avoid being DDoSed. The problem is that Anubis makes the website send out a free JavaScript program that acts like malware. A website using Anubis will respond to a request for a webpage with a free JavaScript program and not the page that was requested. If you run the JavaScript program sent through Anubis, it will do some useless computations on random numbers and keep one CPU entirely busy. It could take less than a second or over a minute. When it is done, it sends the computation results back to the website. The website will verify that the useless computation was done by looking at the results and only then give access to the originally requested page.
> At the FSF, we do not support this scheme because it conflicts with the principles of software freedom. The Anubis JavaScript program's calculations are the same kind of calculations done by crypto-currency mining programs. A program which does calculations that a user does not want done is a form of malware. Proprietary software is often malware, and people often run it not because they want to, but because they have been pressured into it. If we made our website use Anubis, we would be pressuring users into running malware. Even though it is free software, it is part of a scheme that is far too similar to proprietary software to be acceptable. We want users to control their own computing and to have autonomy, independence, and freedom.
I think all of these mitigations are unfortunate. They hurt one of the things that makes the web cool: it's a stable, stateless, idempotent way to access data.
This makes it a prime target for aggressive scraping by LLM companies, but it also makes it accessible and fast, and a prime target for benign use (like archive.org or "read later" services).
For my own sites, I'll eat the cost of the crawlers (mitigated by making the sites as efficient as possible) and keep them available to everyone.
> There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same address after the initial one?
The main problem becomes managing lists of millions of individual addresses. My (only semi-reliable these days, due to lack of time for maintenance) little project has nearly 2.3 million addresses recorded - although only 590k are from 2026, and only 38 were probes on ports 80 and 443. So maybe more manageable than I thought (but my servers don't host anything beyond personal interest to me, and access is filtered via cloudflare, which is it's own "internet control issue").
> In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.
Overall, my gut feel on residential proxies is that they're an untrustworthy scourge. I'd be interested in any arguments for residential proxies by people who don't (intend to) profit from using it facilitating them.
In regards to Bright Data, one of the companies that attempts to appear legitimate, at minimum these domains should be blocked:
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
Blocking in ipv6 works roughly the same way as in ipv4, just that the scale is different. Instead of blocking something like a company's /24 or an ISP's /16 when they don't respond to abuse messages, you block the company's /48 or the ISP's /32. It'll vary per organisation how large a range they got exactly but you can see that in WHOIS. End users are no longer at a /32 (v4) but at /64 (v6), or some prosumers might have a /29 (v4) and /56 (v6). Same concept, just a different prefix length
> do you want random IoT devices talking to your web server?
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
Sorry, I understand scraping is a problem, but talking about open Internet while simultaneously complaining you can no longer discriminate datacenter IPs like you used to is hypocrisy.
I use a datacenter-based IPv6 address because my local ISPs don't offer v6 connectivity and the Internet is already broken for me. And generally the entire idea of a "residential" IP address smells.
Noone complained we can't discriminate DC IPs, though to be fair some (imo bad) operators did just that. This is not even about preventing bots, which has perfectly legitimate usecases (eg. Internet Archive).
This is about filtering out bad bots/actors who have no respect for your resources and will drain all of it causing bad experience for everyone. But because they know they don't respect robots.txt or even simple rate-limiting, they have to employ so-called residential VPNs. They're residential in that they route through real user connections, and so you can't block the IP/subnet without dropping a certain amount of legitimate human-driven traffic.
Personal example: some time ago, i had to disable a wordpress plugin on a site that was causing 100% CPU usage on the whole box (hosting dozens of wordpress instances). That plugin was a simple calendar, but a bot was repeatedly scraping non-existent (or rather, "no event planned for this day") pages for every date in the calendar that you can represent in the DB timestamp, clearing the cache as it went to try and find new events for 1000 years ago. Whoever operates this IP space doesn't matter to me, i'd just like to block them because they don't respect robot.txt… but i can't because they use a "residential proxy" and will change IP address every hour or so.
This is definitely an issue, where data centre addresses are, by default, second class citizens or even persona non grata. The problem is that this reputation has been, unfortunately, well-earned.
There's no easy answer here. The ephemerality and pseudonymity of VPS address usage screams untrustworthy, and the only way to reign that in is better identification of who is using the VPS/address or significantly more restrictive rules applied to data/port usage. And I'm not sure if I like the general direction that points towards - away from the "open internet".
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
You can just Google this. It isn't illegal, you can find many providers, you can even pay with your credit card. Usually around $0.20/GB - other types of proxies are cheaper.
I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
EDIT: Lwn.net is perhaps not a fair target of my ire.
“There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.”
> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
I don't think people sit around going "Grrrr who can I ban next?". Instead this stuff gets noticed because you see the webserver at 99% CPU utilization for 2 days straight, check the logs, and see you are somehow getting crawled by half the IPs in New York City.
Ever since bots became a problem on the internet 10-20 years ago, it has seemed like the common-sense solution is some kind of micropayment. Pay $0.01 to view the page. When money is on the line, scrapers are likely to be more well-behaved, even if they do pay. The problem is, and has always been, the friction of payment. How do you pay $0.01? The credit card processors will tack on a $6 surcharge. We need a trusted third-party that turn money into "internet article credits" that you can spend in small increments, like a video game. But I suspect that thousands of people have already though of this system, and tried it, but ran into some roadblock. I'm guessing there's some egregious regulation that makes micropayments impossible.
> I'm guessing there's some egregious regulation that makes micropayments impossible.
More likely there isn't any kind of universal standard that's easy to implement for browser makers, has low overhead, and preserves internet users' anonymity as much as possible.
The currently existing friction of using micropayments is the problem here, I suspect.
Probably with something similar to Lightning Network, which reallocates pre-committed funds between two or more parties. But not with Lightning Network itself, for several reasons including how costly it is to pre-commit the funds.
I find the notion that you would use residential proxies to scrape LWN somewhat laughable, I'm reading this article using a VPN.
residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).
I ran a gitweb server which was battered by bots so I eventually had to take it down. Gitweb! You can just connect using the git protocol and download everything vastly more efficiently!
In other words, they don't care at all. For them, residential bandwidth is completely free.
>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
> The moment it does you'll be able to tell because Bitcoin will crash to 0.
The "workaround" for PoW is running the PoW computation on hardware that's better suited for the task. Bitcoin mining has been using ASIC for many years now.
Let's say a legitimate user is willing to wait for one minute on a budget phone. Then your PoW is limited to what that phone can compute in one minute. But on the attacker's specialized hardware this computation only costs fractions of a penny, so they are barely hindered by it.
The SHA256 based PoW scheme has a very heavy ASIC advantage. People have tried to design PoW scheme that minimize the custom hardware advantage, but I'm not sure if they managed to close the gap far enough to make PoW feasible for this application.
This is a predictable consequence of age verification laws and social media bans. Formerly VPNs were a nice to have but now they are a necessity in many countries to navigate the modern internet.
The cheapest way to get a VPN (and if you're a horny and broke teenager perhaps the only way) is to trade your clean but censored IP address for an uncensored IP address in another country. You accept the bot traffic in return, or externalize it to your parents or the owner of the internet connection.
Running a small public JSON API, the traffic breakdown is eye-opening. Roughly half is trust/uptime "scanners" and generic monitors; a solid chunk is well-behaved crawlers that declare themselves with real UAs and honor robots; and then there's a long tail of vuln-scanners blindly probing for /.env, /.git, wp-login and the like. The genuinely evasive residential-proxy scraping is a minority by volume but by far the hardest to separate from real users — it's the one bucket where UA and IP both look residential, so you can't tell bot from human without behavioral signals. What's shifted in the last year: the "polite" bots got politer, while the abusive layer moved almost entirely onto residential proxies. IP reputation alone is basically dead as a filter now.
I wanted to jump in and share a few thoughts. Not all the residential networks mentioned on this page are bad actors.
Transparency & Auditing: Our clients are completely open-source, and we run strict internal audits before every single release [1][2]. You don't need to be a security wizard to verify this, either. You can easily audit the code yourself—just clone the repo, feed it into an AI, and ask the right questions.
Ethical Sourcing: Consent is everything. At Proxybase, we always get explicit consent from our providers before adding them to the pool. This is exactly how ethical sourcing should be done. Historically, this industry has been incredibly shady think malware bundled into iOS/Android apps or second-tier smart TVs secretly installing background scrapers. Fortunately, the sector is finally becoming more ethically aware.
Fair Payouts: A lot of networks hold onto provider funds for months, staking them to earn passive income while making users wait. Between sky-high payout thresholds and endless waiting periods, it’s a broken system. At Proxybase, we have a $1 minimum payout sent directly to your wallet using US stablecoins.
If you have a better idea on how we can make this industry better, just lmk. I'm reading/writing on HN everyday.
There is a large community of people that poison scrapers.
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
Edit: If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
The most recent 60 (!) comments plus every submission of the last 6 months were all about the same thing. That's extreme. The posts didn't all mention that specific project, but there was only one topic and they were extremely repetitive. This is not a close call.
Submissions: 8 most recent on PF, 9+ cover nuclear power, Tor dark web, robotaxis, and other topics.
Again: Not a one-note flute, though fairly focused of late on AI and poisoning.
Again: I think the ban is unwarranted. I'm not sure what's driving your thinking here, but a no-warnings ban seems excessive. And given YC's current preponderance of AI/agentic launches (<https://news.ycombinator.com/launches>), self-serving and contrary to the "we moderate YC stories less" guideline.
(Yes, I'm aware "less" isn't "none", and this is an account/user rather than story, I hope my point stands and is clear.)
The Yann LeCun posts you link are ... a bit OTT. That's also a couple of years ago.
I've said my bit. I'm hoping you and atomic128 can come to an understanding in email.
Jon's been around a while and some of the piss and vinegar of youth may have subsided. He does tend to show up with LWN comes up, whether as a topic of discussion (or more often) from submitted articles. That's his baliwick, and again, I don't fault him at all for it.
Our other friend here is a more recent participant to HN, at least under this handle. (I don't know that there are others, only what I can see from this one.)
I think the reasoning is about having alt accounts for different purposes. He intention is to map one human to one account and have all of their thoughts from that one account, instead of one human having one account to discuss scraping on, and a different account to discuss crypto on.
I'm pretty sure that the specific gripe is posting excessively (not even necessarily exclusively) on a single topic or theme. See <https://news.ycombinator.com/item?id=19392902> for a more detailed comment from dang.
Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.
I'm confused why everyone is pretending that a ban holds any meaning here. he probably already has a new account, it might have made sense to just silently ban him in the hope of imposing a minor cost on what you believe he is doing, but wasting your time addressing it imposed a far greater cost on you than him.
I understand how it can be confusing. The key factors in doing it this way are (1) the community regards older accounts, especially ones that have significant posting history, as more credible; and (2) doing it publicly rather than silently has transparency value.
It's not a sign about that project in any way. I had never heard of it and have no opinion about it one way or the other.
It's just a sign that single-agenda accounts aren't allowed here—no more, no less. That's why I said "We ban such accounts regardless of what the single purpose happens to be".
In theory, proof of work that is used to mine a cryptocurrency could be a solution.
Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads.
The question is which resources typical consumer hardware has that large centralized compute power does not. In-browser POW to pay websites would only be possible if such a resource exists.
I am not familiar with the topic, but maybe CPU power and memory? Both seem significant in a typical consumer device.
Napkin math: If a consumer device can generate $100 per month, that would be 100/30/24/60/60=$0.00004 per second. If the user waits for 5 seconds before the first pageview, that would then make the website provider $0.0002 per visitor. Serving a million visitors per month is nowadays easily possible on a $10/month machine. So the $0.0002x1000000 = $200 would make the website a nice profit.
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
> The anubis author has stated they recognize it's an arms race, but PoW scales.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
At least anubis works for me. (I run umatrix)
Unfortunately whatever HN is using routinely blocks my login with "Sorry."
some websites just always give me 403.
Well, we don't use a captcha either. If it were a choice between a captcha and a proof of work system, we'd have to reevaluate things. Luckily, for now, we're able to get away with a much lighter touch.
> but PoW scales
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
[1]: https://news.ycombinator.com/item?id=30175269
Or you can go full Reddit and just block anything that seems even remotely suspicious.
Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too.
Using VPN? Blocked.
Your iPhone is too old, blocked.
Your screen brightness too low? Believe or not, blocked.
I'm not gonna wait a minute to read an article. Instead, I'll either just leave, or go query it from archive.$tld that bypasses it for me.
Anubis appears to be a temporarily-useful stopgap that has been cargo culted into prominence and an expectation of permanent usefulness, for reasons I don't fully understand.
The cost of solving the default Anubis PoW is negligible on cloud servers, and it's even lower if you use native code rather than JavaScript to solve it, which Tavis Ormandy helpfully demonstrated last year (https://lock.cmpxchg8b.com/anubis.html). If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.
So Anubis doesn't do much to stop bots, but it makes otherwise lightweight websites (little JavaScript or interactivity) almost unusable on low-resource systems like my old phone or an old Atom-based nettop.
> when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape
This "IP-bound proof-of-work" thing is gonna kill multipath TCP and bring down IPv6 with it. Uffff.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
You can't do that any more. Too many ISPs, especially mobile carriers, don't hand out anything resembling a fixed IP address any more. It's CGNAT and constantly changing IP addresses alllll the time now.
[flagged]
PoW can theoretically scale effectively infinite because it can mine cryptocurrency. Millions of compromised IoT devices hitting your server? Now you have enough money for a faster server.
It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
Ironically in early 2023 a lot of websites went out of their way to block Common Crawl. Unsurprisingly that shifted scraping toward individual actors whereas the previous solution in research was to download CC dumps and process them.
I'm sure there are those who would participate, either because they want their data to be captured by AI labs or as a form of compromise.
That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
Feels like it would be a good time for freenet and the like to catch on.
The article at the end talks about how is very easy for arbitrary apps from app stores can install a residential proxy on your phone.
10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what. And both ios and android make it hard to deny apps network access.
But really, this finally explains the hordes of really basic boring games that just advertise other boring games. Idle games and the like that really just want you to keep your phone unlocked and open. Millions of downloads on the app stores for entirely offline content (and ads) and no way to block the network access.
> 10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what.
Why does network access need to be a binary, all or nothing?
When you install an app, the app should request permissions to specific DNS names, i.e. pointing to the servers that the app's authors operate. If I install Todoist, the app should only ask for access to Todoist's servers. If I install Netflix, the app should only ask for access to Netflix's servers. The OS can then put a DNS firewall in and block any network access that wasn't granted when the app was installed.
> And both ios and android make it hard to deny apps network access
The list of apps that genuinely need "any" network access (web browsers, VPN apps, stuff like Termux...) is incredibly small compared to the list of apps that need access to a small number of VPN targets (these days, most apps). Apple / Google could even decide, if they really want to make it easy for apps to request network access, to basically allow apps to automatically get network access, so long as the list of domains the app needs access to is no more than a handful. The security value of isolating "all" network access permissions to only the relative handful of apps that actually need to request it, would be huge.
GrapheneOS allows you to deny network access per app pretty trivially. Google Play services make it a bit more difficult because the app might marshall the network request through that; I'm not sure how to verify that behavior when it happens.
Google should be able to detect this and ban those apps from the Play Store. They have the incentive too.
The Bright Data “free” VPN they’re talking about requires the user to go through steps to enable it.
These aren’t as simple as downloading a free game and then the phone is compromised as long as it’s installed.
The users who install these things don’t care about permissions prompts. They’ll follow instructions to tap any prompt the instructions ask. They want the free thing and don’t care what they have to do to get it.
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
> I think that nobody would care if I use wget or curl for few pages
If only you were the only one doing it...
One article mentioned in the OP was discussed here:
Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)
How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!
I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
As this article points out, it's tremendously unclear who is using residential proxies.
The big AI models claim they're not using them. I'm not inclined to "just believe them", but no incriminating evidence has leaked, and—as pointed out in the article—many of the bots that are running on these residential proxy botnets are coded in incredibly stupid and inefficient ways.
How confident are people who research this stuff that the RP botnets are actually being used for AI training?
That's the only theory we have that doesn't sound like a conspiracy theory. The only other credible ideas are that someone's doing some kind of dataset arbitrage by scraping the fuck out of everything and selling companies data that is technically new (by means of the scrape date being newer).
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
And it's made necessary because another group of people thought that selling IP blocking services would be a good idea. One party sells walls, another party sells ladders.
Well, one party gives away free walls if you agree to fill your castle with surveillance cameras you don't control.
This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.
I think they also have to operate in countries that don't mind shady things like this.
Thank god for residential proxies.
Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet
TIL:
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
WTF. That's just botnets.
Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...
i think the best way to keep your site working for legit users is serving static cached pages to suspected bots. decide based off something like cloudflare bot score.
crawlers get redirected to content that might be stale but its still useful for them and costs you almost nothing to serve. put a warning banner on it so if a user (or smart agent) accidentally ends up on the bot version they can click on it and get anubis checked for the real site.
of course this only works for people like me who want their work to be used for training ai.
mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
"He who has the gold makes the rules" is older than the pyramids.
What exactly should be illegal here? Scraping websites? AI agents? Not following robots.txt?
> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
I don't run one of these sites that has these issues so I'm really not aware of this problem. How can it be that sites are getting overwhelmed with scrapers that are just looking for training data? You only need to scrape it once to train a model, so shouldn't there be less traffic from this than there is from search engines?
On the other hand if the article is wrong and the traffic is coming from other ai uses (like an agent visiting pages on behalf of a user) then that would make sense.
What a pity. Mostly I just want personal archives of things so that I can search them much faster than commercial solutions and the like.
From the article:
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
> Tracking which devices still send lots of data when no one else is using the network?
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
[0]: https://github.com/jow-/nlbwmon
[1]: https://www.maxchernoff.ca/files/grafana-network-bandwidth.p...
[2]: https://docs.mossdef.org/adblock-fast/
Opnsense has a traffic capture feature in the interface diagnostics menu, if you want to spot check what servers the devices are currently talking to.
Should be pretty obvious: client devices and internal services will have no traffic >95% of the time, just NTP for timekeeping, DHCP lease renewal, and associated ARP (running total: two dozen packets if you monitor them for a full 24h), then any system updaters (readily identifiable by the initial DNS requests), and finally of course you'll see the traffic of the service that the device hosts, if any, which can be easily dismissed by not looking at incoming connections (scraping uses outgoing connections)
"The startup, which markets itself as the second-largest data collection firm after Alphabet Inc.s Google, has grown significantly on demand from data-hungry AI companies."
https://www.bloomberg.com/news/articles/2026-07-10/web-scrap...
Seems like Google is going after the competition
https://fzakaria.com/2026/07/09/who-does-anubis-actually-sto...
I wrote about this recently as well.
I was involved in both sides of this battle over ten years ago. Things haven't changed all that much.
It's important to note that neither side has moral legitimacy. Not everyone who carries a rifle is a enemy. Not everyone wearing body armor is a saint.
I have given up on the idea that "human vs bot" matters at all when it comes to anything other than voting (which should only be done in person with paper and pen, by the way.)
You could make an argument that "likes" are a form of voting, but you shouldn't. We need to abandon the idea of supposedly democratized algorithms and focus instead on actual democracy.
I think this Anubis project is a terrible solution to the problem posed by aggressive web scrapers. Using a web browser with reasonable privacy settings has become a big loss in quality of life already, but the first time I encountered Anubis I got completely locked out of most web servers that deployed it. The situation has improved a little, but I hate that maintainers of great web services have rationalized themselves into believing that creating massive barriers to access their sites is a fair trade-off. Unsurprisingly, I have nothing but negative associations with their mascot.
The FSF has the right idea about all this:
> Some web developers have started integrating a program called Anubis to decrease the amount of requests that automated systems send and therefore help the website avoid being DDoSed. The problem is that Anubis makes the website send out a free JavaScript program that acts like malware. A website using Anubis will respond to a request for a webpage with a free JavaScript program and not the page that was requested. If you run the JavaScript program sent through Anubis, it will do some useless computations on random numbers and keep one CPU entirely busy. It could take less than a second or over a minute. When it is done, it sends the computation results back to the website. The website will verify that the useless computation was done by looking at the results and only then give access to the originally requested page.
> At the FSF, we do not support this scheme because it conflicts with the principles of software freedom. The Anubis JavaScript program's calculations are the same kind of calculations done by crypto-currency mining programs. A program which does calculations that a user does not want done is a form of malware. Proprietary software is often malware, and people often run it not because they want to, but because they have been pressured into it. If we made our website use Anubis, we would be pressuring users into running malware. Even though it is free software, it is part of a scheme that is far too similar to proprietary software to be acceptable. We want users to control their own computing and to have autonomy, independence, and freedom.
https://www.fsf.org/blogs/sysadmin/our-small-team-vs-million...
I think all of these mitigations are unfortunate. They hurt one of the things that makes the web cool: it's a stable, stateless, idempotent way to access data.
This makes it a prime target for aggressive scraping by LLM companies, but it also makes it accessible and fast, and a prime target for benign use (like archive.org or "read later" services).
For my own sites, I'll eat the cost of the crawlers (mitigated by making the sites as efficient as possible) and keep them available to everyone.
the options available to small web hosts are:
1. do nothing: everyone gets page loads of 30s if not worse.
2. put the content behind anubis -> load drops by 80% and page loads are back to "a couple of seconds" (including the client-side challenge).
3. take the content offline -> load drops to 0.
which of these would you prefer?
https://archive.fo/PAcF5
> There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same address after the initial one?
The main problem becomes managing lists of millions of individual addresses. My (only semi-reliable these days, due to lack of time for maintenance) little project has nearly 2.3 million addresses recorded - although only 590k are from 2026, and only 38 were probes on ports 80 and 443. So maybe more manageable than I thought (but my servers don't host anything beyond personal interest to me, and access is filtered via cloudflare, which is it's own "internet control issue").
> In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.
Overall, my gut feel on residential proxies is that they're an untrustworthy scourge. I'd be interested in any arguments for residential proxies by people who don't (intend to) profit from using it facilitating them.
In regards to Bright Data, one of the companies that attempts to appear legitimate, at minimum these domains should be blocked:
brdtnet.com
luminatinet.com
bright-sdk.com
luminati.io
As listed in this article, on HN's front page 34 days ago: https://news.ycombinator.com/item?id=48422993 (https://blog.includesecurity.com/2026/06/the-smart-tv-in-you...)
Why would anyone who doesn't have a use for a residential proxy have an argument for residential proxies?
I use them to scrape closed sites to make the information more open. For example YouTube.
Well the argument appears to be, people put them in their apps instead of ads. (Or more likely on top of ads.) The argument is money.
The users presumably don't know about this, or you know, they clicked, "I agree."
Nearly Half of LG Smart TV Apps Contain Residential Proxies
https://news.ycombinator.com/item?id=48635954
Of course the issue with blocking residential IPs is that then they would be prevented from doing normal things on the internet.
At which point, millions of people will be forced to complain to their local representatives and... hey presto? :)
Scanners and scrapers are mostly two different groups.
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
Blocking in ipv6 works roughly the same way as in ipv4, just that the scale is different. Instead of blocking something like a company's /24 or an ISP's /16 when they don't respond to abuse messages, you block the company's /48 or the ISP's /32. It'll vary per organisation how large a range they got exactly but you can see that in WHOIS. End users are no longer at a /32 (v4) but at /64 (v6), or some prosumers might have a /29 (v4) and /56 (v6). Same concept, just a different prefix length
> do you want random IoT devices talking to your web server?
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
Sorry, I understand scraping is a problem, but talking about open Internet while simultaneously complaining you can no longer discriminate datacenter IPs like you used to is hypocrisy.
I use a datacenter-based IPv6 address because my local ISPs don't offer v6 connectivity and the Internet is already broken for me. And generally the entire idea of a "residential" IP address smells.
Noone complained we can't discriminate DC IPs, though to be fair some (imo bad) operators did just that. This is not even about preventing bots, which has perfectly legitimate usecases (eg. Internet Archive).
This is about filtering out bad bots/actors who have no respect for your resources and will drain all of it causing bad experience for everyone. But because they know they don't respect robots.txt or even simple rate-limiting, they have to employ so-called residential VPNs. They're residential in that they route through real user connections, and so you can't block the IP/subnet without dropping a certain amount of legitimate human-driven traffic.
Personal example: some time ago, i had to disable a wordpress plugin on a site that was causing 100% CPU usage on the whole box (hosting dozens of wordpress instances). That plugin was a simple calendar, but a bot was repeatedly scraping non-existent (or rather, "no event planned for this day") pages for every date in the calendar that you can represent in the DB timestamp, clearing the cache as it went to try and find new events for 1000 years ago. Whoever operates this IP space doesn't matter to me, i'd just like to block them because they don't respect robot.txt… but i can't because they use a "residential proxy" and will change IP address every hour or so.
This is definitely an issue, where data centre addresses are, by default, second class citizens or even persona non grata. The problem is that this reputation has been, unfortunately, well-earned.
There's no easy answer here. The ephemerality and pseudonymity of VPS address usage screams untrustworthy, and the only way to reign that in is better identification of who is using the VPS/address or significantly more restrictive rules applied to data/port usage. And I'm not sure if I like the general direction that points towards - away from the "open internet".
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.
How much does routing traffic though residential proxies cost?
You can just Google this. It isn't illegal, you can find many providers, you can even pay with your credit card. Usually around $0.20/GB - other types of proxies are cheaper.
Has no one noticed their miniflux instance failing to fetch feeds because of this?
So far I’ve only encountered one site that blocked automated access to its web feed. I assume that was just an oversight.
i wonder if residential ISPs can play a bigger role here
I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
EDIT: Lwn.net is perhaps not a fair target of my ire.
“There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.”
Is reasonable! Many others are not
> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
If it weren't a real problem, these types of articles and services wouldn't exist.
I don't think people sit around going "Grrrr who can I ban next?". Instead this stuff gets noticed because you see the webserver at 99% CPU utilization for 2 days straight, check the logs, and see you are somehow getting crawled by half the IPs in New York City.
Why the air quotes? Evading a ban and using (potentially ill gotten) residential ips to circumvent that refusal of service, is a bad actor.
[dead]
Ever since bots became a problem on the internet 10-20 years ago, it has seemed like the common-sense solution is some kind of micropayment. Pay $0.01 to view the page. When money is on the line, scrapers are likely to be more well-behaved, even if they do pay. The problem is, and has always been, the friction of payment. How do you pay $0.01? The credit card processors will tack on a $6 surcharge. We need a trusted third-party that turn money into "internet article credits" that you can spend in small increments, like a video game. But I suspect that thousands of people have already though of this system, and tried it, but ran into some roadblock. I'm guessing there's some egregious regulation that makes micropayments impossible.
> I'm guessing there's some egregious regulation that makes micropayments impossible.
More likely there isn't any kind of universal standard that's easy to implement for browser makers, has low overhead, and preserves internet users' anonymity as much as possible.
The currently existing friction of using micropayments is the problem here, I suspect.
Probably with something similar to Lightning Network, which reallocates pre-committed funds between two or more parties. But not with Lightning Network itself, for several reasons including how costly it is to pre-commit the funds.
Google itself is a huge database.Who makes these rules depends on who's leading the market.
> widespread scraping of web sites in search of training data for large language models and related projects
This is a good thing, thanks to this we have powerful open source LLMs.
> This activity overwhelms sites with traffic.
When LLMs get good enough, we won't need those sites anymore :)
[not satire, this is what I think, without self-censorship]
I find the notion that you would use residential proxies to scrape LWN somewhat laughable, I'm reading this article using a VPN.
residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).
I ran a gitweb server which was battered by bots so I eventually had to take it down. Gitweb! You can just connect using the git protocol and download everything vastly more efficiently!
In other words, they don't care at all. For them, residential bandwidth is completely free.
Right. This whole controversy makes no sense as a pure scraping thing. It seems more like someone is trying to take the web offline.
>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
> The moment it does you'll be able to tell because Bitcoin will crash to 0.
The "workaround" for PoW is running the PoW computation on hardware that's better suited for the task. Bitcoin mining has been using ASIC for many years now.
Let's say a legitimate user is willing to wait for one minute on a budget phone. Then your PoW is limited to what that phone can compute in one minute. But on the attacker's specialized hardware this computation only costs fractions of a penny, so they are barely hindered by it.
The SHA256 based PoW scheme has a very heavy ASIC advantage. People have tried to design PoW scheme that minimize the custom hardware advantage, but I'm not sure if they managed to close the gap far enough to make PoW feasible for this application.
Residential proxy users don't have the ability to run compute on their proxies.
This is a predictable consequence of age verification laws and social media bans. Formerly VPNs were a nice to have but now they are a necessity in many countries to navigate the modern internet.
The cheapest way to get a VPN (and if you're a horny and broke teenager perhaps the only way) is to trade your clean but censored IP address for an uncensored IP address in another country. You accept the bot traffic in return, or externalize it to your parents or the owner of the internet connection.
That does not explain why so many residential VPNs operate with so many IPs in countries where there are no social media bans. Here in France:
- i know many people who buy shady IPTV boxes from stores/markets for like 50€/year
- i know some people who use "smart lightbulbs" and other nonsense
- almost everyone i know plays free smartphone games, which as LWN reminded, may contain a shady SDK
Running a small public JSON API, the traffic breakdown is eye-opening. Roughly half is trust/uptime "scanners" and generic monitors; a solid chunk is well-behaved crawlers that declare themselves with real UAs and honor robots; and then there's a long tail of vuln-scanners blindly probing for /.env, /.git, wp-login and the like. The genuinely evasive residential-proxy scraping is a minority by volume but by far the hardest to separate from real users — it's the one bucket where UA and IP both look residential, so you can't tell bot from human without behavioral signals. What's shifted in the last year: the "polite" bots got politer, while the abusive layer moved almost entirely onto residential proxies. IP reputation alone is basically dead as a filter now.
[flagged]
Disclaimer: I own and operate proxybase.xyz [0]
Hi HN,
I wanted to jump in and share a few thoughts. Not all the residential networks mentioned on this page are bad actors.
Transparency & Auditing: Our clients are completely open-source, and we run strict internal audits before every single release [1][2]. You don't need to be a security wizard to verify this, either. You can easily audit the code yourself—just clone the repo, feed it into an AI, and ask the right questions.
Ethical Sourcing: Consent is everything. At Proxybase, we always get explicit consent from our providers before adding them to the pool. This is exactly how ethical sourcing should be done. Historically, this industry has been incredibly shady think malware bundled into iOS/Android apps or second-tier smart TVs secretly installing background scrapers. Fortunately, the sector is finally becoming more ethically aware.
Fair Payouts: A lot of networks hold onto provider funds for months, staking them to earn passive income while making users wait. Between sky-high payout thresholds and endless waiting periods, it’s a broken system. At Proxybase, we have a $1 minimum payout sent directly to your wallet using US stablecoins.
If you have a better idea on how we can make this industry better, just lmk. I'm reading/writing on HN everyday.
[0] https://proxybase.xyz
[1] https://github.com/proxybasehq/proxybase-gui
[2] https://github.com/proxybasehq/proxybase-cli
[dead]
There is a large community of people that poison scrapers.
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...
I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
https://news.ycombinator.com/newsguidelines.html
Edit: If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
Just curious dang, did you warn them before banning?
Im not against the ban perse (single purpose accounts are bad), just curious if they had a chance to change their contribution style.
No, but if they have a change of heart and genuinely want to use HN as intended, they're welcome to let us know. I've added that in an edit now.
Seriously, dang?
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
The most recent 60 (!) comments plus every submission of the last 6 months were all about the same thing. That's extreme. The posts didn't all mention that specific project, but there was only one topic and they were extremely repetitive. This is not a close call.
I made it all the way back to https://news.ycombinator.com/posts?id=atomic128&next=4628060... (6 months ago) before seeing posts about anything else, only to find that there was a different agenda before that. Not cool.
Edit: and before all that, there was this: https://news.ycombinator.com/posts?id=atomic128&next=4164795.... This is obviously not using HN as intended.
Exceptions?
<https://news.ycombinator.com/item?id=47116093> (LLM but not PF).
<https://news.ycombinator.com/item?id=47095664> (a16h)
<https://news.ycombinator.com/item?id=46695693> (vuln exploits) 2026-1-20
<https://news.ycombinator.com/item?id=46280602> (???, but not PF)
<https://news.ycombinator.com/item?id=46195234> (Monero / Dark Web) 2025-12-8
<https://news.ycombinator.com/item?id=45894305> and <https://news.ycombinator.com/item?id=45826273> (Tor hidden service) Nov 2025
That's from the past 20 comments.
Submissions: 8 most recent on PF, 9+ cover nuclear power, Tor dark web, robotaxis, and other topics.
Again: Not a one-note flute, though fairly focused of late on AI and poisoning.
Again: I think the ban is unwarranted. I'm not sure what's driving your thinking here, but a no-warnings ban seems excessive. And given YC's current preponderance of AI/agentic launches (<https://news.ycombinator.com/launches>), self-serving and contrary to the "we moderate YC stories less" guideline.
(Yes, I'm aware "less" isn't "none", and this is an account/user rather than story, I hope my point stands and is clear.)
The Yann LeCun posts you link are ... a bit OTT. That's also a couple of years ago.
I've said my bit. I'm hoping you and atomic128 can come to an understanding in email.
> Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I took a look at the most recent comments from both accounts and they don't look similar to me in this respect.
I think there are two questions here though:
1. Was the violation egregious?
2. Did it deserve an immediate ban, or did they deserve a warning etc.?
Seems to me the answer to (1) is yes, but the answer to (2) I'm less sure about.
Jon's been around a while and some of the piss and vinegar of youth may have subsided. He does tend to show up with LWN comes up, whether as a topic of discussion (or more often) from submitted articles. That's his baliwick, and again, I don't fault him at all for it.
Our other friend here is a more recent participant to HN, at least under this handle. (I don't know that there are others, only what I can see from this one.)
I think the reasoning is about having alt accounts for different purposes. He intention is to map one human to one account and have all of their thoughts from that one account, instead of one human having one account to discuss scraping on, and a different account to discuss crypto on.
I'm pretty confident it's not that.
HN's prime directive is "anything that gratifies one's intellectual curiosity": <https://news.ycombinator.com/newsguidelines.html> and many, many, many dang comments.
I'm pretty sure that the specific gripe is posting excessively (not even necessarily exclusively) on a single topic or theme. See <https://news.ycombinator.com/item?id=19392902> for a more detailed comment from dang.
Occasional alts are explicitly permitted, though not to engage in abuse (e.g., mutual admiration societies, sock-puppetry). See: <https://news.ycombinator.com/item?id=9963551> <https://news.ycombinator.com/item?id=9823379> (both against sock puppetry) and <https://news.ycombinator.com/item?id=9122086> and <https://news.ycombinator.com/item?id=7504621> (on where throwaways are/aren't permitted).
Where HN does favour persistent accounts the stated claim is to foster community, rather than for nefarious tracking purposes: <https://news.ycombinator.com/item?id=18082346> and <https://news.ycombinator.com/newsguidelines.html>. From that last:
Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.
I'm confused why everyone is pretending that a ban holds any meaning here. he probably already has a new account, it might have made sense to just silently ban him in the hope of imposing a minor cost on what you believe he is doing, but wasting your time addressing it imposed a far greater cost on you than him.
I understand how it can be confusing. The key factors in doing it this way are (1) the community regards older accounts, especially ones that have significant posting history, as more credible; and (2) doing it publicly rather than silently has transparency value.
This is a strong positive sign that poison fountain works.
I wasn't aware of this project. Thanks for the heads up.
It's not a sign about that project in any way. I had never heard of it and have no opinion about it one way or the other.
It's just a sign that single-agenda accounts aren't allowed here—no more, no less. That's why I said "We ban such accounts regardless of what the single purpose happens to be".
>types of operator running residential-proxy networks to attack web sites.
This is such a malicious interpretation. Do you think VPN operating are also trying to attack websites? Both offer the same kind of product.
>paid for hijacking their users' network connections
Nothing is being hijacked. Again the author is using wording to try and paint these people as malicious actors.
>Recently, LWN was subjected what was, by far, the heaviest scraper attack yet.
LWN is a static site. To me it seems more expensive to use Anubis than just serve the actual page.
>will now check for NetNut-infected apps
Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.
Again, why do we allow China on the Internet?
Backbone operators should not be allowed to knowingly maintain connections to networks that allow connections from China or Russia.
In theory, proof of work that is used to mine a cryptocurrency could be a solution.
Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads.
The question is which resources typical consumer hardware has that large centralized compute power does not. In-browser POW to pay websites would only be possible if such a resource exists.
I am not familiar with the topic, but maybe CPU power and memory? Both seem significant in a typical consumer device.
Napkin math: If a consumer device can generate $100 per month, that would be 100/30/24/60/60=$0.00004 per second. If the user waits for 5 seconds before the first pageview, that would then make the website provider $0.0002 per visitor. Serving a million visitors per month is nowadays easily possible on a $10/month machine. So the $0.0002x1000000 = $200 would make the website a nice profit.