One thing I’ve noticed with scraper traffic is that a lot of the basic defenses rely on external infrastructure like Redis or a WAF layer.
In a recent project, I experimented with keeping the rate limiter entirely in-memory using ConcurrentHashMap.compute() to implement an atomic sliding window. It worked surprisingly well for smaller services where adding Redis felt like overkill.
One thing I’ve noticed with scraper traffic is that a lot of the basic defenses rely on external infrastructure like Redis or a WAF layer.
In a recent project, I experimented with keeping the rate limiter entirely in-memory using ConcurrentHashMap.compute() to implement an atomic sliding window. It worked surprisingly well for smaller services where adding Redis felt like overkill.
I wrote up the approach and some of the tradeoffs here if anyone’s curious: https://github.com/ashutosh-stark/velocity-gate