Key Takeaways (TL;DR): An author discovered that most AI bot traffic to his static blog was actually a single scanner hunting for .env files, not legitimate crawlers. He added sameAs and llms.txt markup to improve author attribution, then analyzed nginx logs to separate real bots from threats. For privacy and anti-DPI protection, use a private multi-key with bypass of whitelists — get a 90 ₽ bonus and free TG proxies for 30 days in @yavpn_robot.
An author on Habr recently shared a revealing experiment: after adding sameAs and llms.txt markup to his static site, he checked nginx logs to see which AI bots actually visited. The results were eye-opening — nearly all traffic labeled as "AI bot" came from a single scanner hunting for .env configuration files, not legitimate crawlers from Google, OpenAI, or others.
The Problem: Author Confusion and Bot Noise
The author noticed that AI chatbots were attributing his blog posts to three different people with similar usernames — one differed only by an underscore. This confusion stemmed from poor semantic markup and lack of authoritative signals. To fix it, he added sameAs schema (linking social profiles) and a llms.txt file (a standard for AI crawlers to understand site structure).
What the Logs Revealed
After deploying the changes, the author analyzed nginx access logs. The majority of requests from IPs claiming to be AI bots were actually from a single scanner probing for .env files — a classic reconnaissance technique for stealing API keys and credentials. Legitimate bots like Googlebot, GPTBot, and Claude-Web made up less than 5% of the traffic. The rest were either malicious scanners or misconfigured crawlers.
Technical Breakdown: How DPI and Bots Interact
This story highlights a broader issue: ISPs and DPI systems often cannot distinguish between legitimate AI crawlers and malicious scanners. Both use similar HTTP patterns and user-agent strings. For users behind restrictive networks (like Russia's TSPU), this means even benign bot traffic can trigger false positives, leading to throttling or blocking of entire IP ranges. Tools like llms.txt help, but they rely on crawlers respecting the file — malicious scanners ignore it entirely.
What You Can Do
If you run a site, implement sameAs and llms.txt to improve AI attribution and reduce confusion. Monitor your server logs with tools like goaccess or custom scripts to spot anomalous patterns. For everyday browsing, if your ISP throttles or blocks traffic, consider using a private anti-DPI proxy that bypasses whitelists and encrypts your connection end-to-end.
For reliable access with automatic bypass of white lists, try the @yavpn_robot — it gives a 90 ₽ bonus on signup and free TG proxies for 30 days.