{"id":50479949,"title":"Show HN: AI-drop project – drop traffic fast from misbehaving crawlers (AGPL)","points":2,"user":"catab","time":1794557144,"time_ago":"7 hours ago","type":"link","content":"

Hello!

All started with 4 million requests / day on rocketgit.com (AGPL git hosting).\nInstead of thousands. Enough is enough!

Presentation (8 pages):\nhttps://rocketgit.com/user/catalinux/ai-drop/artefacts/downl...

So, I built AI-drop free-software project (AGPL) to block this kind of crawlers.\nrocketgit.com has a very restrictive robots.txt (only around 10 pages are allowed).\nBut the crawlers are not respecting it. So, who is not respecting it will be blocked.

Limiting by IP or user-agent is not working anymore. The crawlers are using millions of IPs\nand it takes days to reuse an IP. And they use fake user-agents.

I am using honeypots to build a list of misbehaving IPs which are streamed to\ndownstream instances. Right now I have around 1.573.000 IPv4 and 6.000 IPv6\nmisbehaving addresses. I keep them in the list only for 7 days to not block somebody legit.\nI am watching the traffic on rocketgit.com and the solution is working nicely.

It uses an eBPF program to fast drop the packets.

You can try it easily on your Linux servers:

# Check https://rocketgit.com/user/catalinux/ai-drop/pkg\n# If your distribution is already there, follow the instructions.\n# It is the best way to keep you up-to-date.

# Look inside 'ai-drop.spec' or 'debian/control' and install the dependencies.\ngit clone https://rocketgit.com/user/catalinux/ai-drop\ncd ai-drop\n./configure\nmake install\n# edit /etc/ai-drop.conf\n# Very important:\n# set 'interface' (non 'eth' are not supported now)\n# set 'safe_mode = 0' if you only want to evaluate the product.

Then:

systemctl enable --now ai-drop\njournalctl -f -u ai-drop

# The list of misbehaving IPs will be streamed from the main server.

# Look at the statistics, especially 'drops':\nai-drop dump-stats\n# Output example:\n# now=1794482762 ipv4_total=0 ipv6_total=0 total=0 \\\n# ipv4_adds=0 ipv6_adds=0 adds=0 \\\n# ipv4_updates=0 ipv6_updates=0 updates=0 \\\n# ipv4_drops=0 ipv6_drops=0 drops=0 \\\n# ipv4_real_drops=0 ipv6_real_drops=0 real_drops=0 \\\n# ipv4_expires=0 ipv6_expires=0 expires=0 \\\n# ipv4_deletes=0 ipv6_deletes=0 deletes=0 \\\n# last_load=0 daemon_start=0

Please note that no AI was used to build this software.

Thank you for your time!

Catalin(ux) C.M.T. BOIE","url":"https://rocketgit.com/user/catalinux/ai-drop","domain":"rocketgit.com","comments":[],"comments_count":0}