How stealth AI crawlers hide, and how we spot them
Well-behaved crawlers announce themselves, so a user-agent rule is enough to stop them. Stealth crawlers don’t: they pretend to be an ordinary browser and rotate through many IP addresses so no single address looks busy.
AgentGate looks past the name. It notices when “browsers” come from hosting and data-centre networks rather than homes and phones, counts requests per IP block instead of per address so rotating doesn’t reset the count, and watches for crawl patterns no person produces. Search engines and AI crawlers that claim a famous name are verified rather than trusted.