Keeping bots out of search

We added a search log to the admin panel today so community owners can see what people are actually looking for.

Within hours, it filled up with random queries in a dozen languages. Insurance, anime, AI gateways. Nothing to do with the community. Bots were hitting our search pages directly.

They couldn’t get those pages into Google, because search pages were already set to noindex. But they were wasting server resources and burying the real searches.

So we fixed it.

User-agent: *
Allow: /
Disallow: /search$
Disallow: /search?
Disallow: /search/

Sitemap: https://community.jatra.club/sitemap.xml

Guest searches now run only when they start on your community. If someone opens a search link from elsewhere, they click “Show results” once. Known crawlers don’t get search results at all, and members won’t notice any change.

We also updated robots.txt to tell well-behaved crawlers to skip search pages. Your posts and channels stay open to Google, exactly as before.

Frankly speaking, robots.txt alone doesn’t stop anyone. Bad bots ignore it. That’s why the real block happens on the server.

Your search log should now show what your community is really looking for.