Track AI Crawlers Server-Side: See GPTBot & ClaudeBot Hits

Script-based analytics — ours included — works by executing JavaScript in the visitor’s browser. That is the right architecture for humans, and it even catches Googlebot, because Google renders pages with a real browser engine before indexing them (Google documents this rendering step).

Most AI crawlers do not render. GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Amazonbot and the rest fetch your raw HTML, parse it, and leave — no JavaScript ever runs, so no analytics beacon ever fires. A script-based tracker is not undercounting them; it is structurally incapable of seeing them.

A CDN like Cloudflare sees everything because it terminates every request before your server does. Your analytics vendor is not in that path — which is exactly why the fix has to live on your server.

AI crawler hits are not vanity data. They are the top of the **AI visibility** funnel: an assistant can only cite or recommend pages its crawlers have fetched. If you are investing in generative engine optimization, crawl activity is the first measurable signal that it is working.

• Coverage — which of your pages AI crawlers actually fetch (and which they ignore). Crawlers hammer `/robots.txt`, `/llms.txt` and your sitemap; if they never reach your money pages, your llms.txt and internal linking need work.
• Trend — whether AI attention is growing after content or GEO changes.
• Correlation — crawl activity → citations in AI answers → referral visits from assistants (the AI channel in your analytics). Seeing all three in one place tells you where the chain breaks.
• Policy — whether your robots.txt or CDN bot settings are silently blocking crawlers you actually want (check with an AI crawler checker).

The pattern (used by every tool in this space) is a fire-and-forget hook in your request path: when the user agent looks like a bot, POST the user agent + path to an ingestion endpoint and move on. Never await the call in the request path — a slow or failing analytics endpoint must never slow a real response.

The important design decision: your middleware only does a cheap pre-filter; the ingestion server owns the real classification (which bot, which company, which category — answer engine, training crawler, search engine). Unknown user agents are dropped server-side, never stored as visitors. Because classification lives server-side, your snippet never needs updating when new crawlers appear — the maintained crawler lists move fast.

Bot hits are stored separately from human analytics: they never inflate visitor counts, sessions or billing meters. This matters — mixing crawler hits into visitor metrics is how analytics gets 30% bot pollution.

Your tracker ID (the `trk_…` value) is in your install snippet. Each example forwards suspected bots to the BusinessMCP bot-hit endpoint; adapt the regex freely — over-sending is fine because the server drops non-bots.

Next.js (middleware.ts — or add the same block to your existing middleware):

import { NextResponse, type NextRequest, type NextFetchEvent } from 'next/server'

const BOT = /bot|crawl|spider|slurp|gptbot|claude|anthropic|perplexity|amazonbot|bytespider|ccbot|meta-external|applebot|headless/i

export function middleware(req: NextRequest, event: NextFetchEvent) {
  const ua = req.headers.get('user-agent') || ''
  if (BOT.test(ua)) {
    event.waitUntil(
      fetch('https://businessmcp.com/api/tracker/trk_XXXXXXXX/bot-hit', {
        method: 'POST',
        headers: { 'content-type': 'application/json' },
        body: JSON.stringify({ userAgent: ua, path: req.nextUrl.pathname }),
      }).catch(() => {}),
    )
  }
  return NextResponse.next()
}

export const config = { matcher: ['/((?!api/|_next/).*)'] }

Express / Node:

const BOT = /bot|crawl|spider|slurp|gptbot|claude|anthropic|perplexity|amazonbot|bytespider|ccbot|meta-external|applebot|headless/i

app.use((req, res, next) => {
  const ua = req.headers['user-agent'] || ''
  if (BOT.test(ua)) {
    fetch('https://businessmcp.com/api/tracker/trk_XXXXXXXX/bot-hit', {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ userAgent: ua, path: req.path }),
    }).catch(() => {})
  }
  next()
})

Cloudflare Worker (if your DNS is proxied through Cloudflare):

const BOT = /bot|crawl|spider|slurp|gptbot|claude|anthropic|perplexity|amazonbot|bytespider|ccbot|meta-external|applebot|headless/i

export default {
  async fetch(request, env, ctx) {
    const ua = request.headers.get('user-agent') || ''
    if (BOT.test(ua)) {
      ctx.waitUntil(
        fetch('https://businessmcp.com/api/tracker/trk_XXXXXXXX/bot-hit', {
          method: 'POST',
          headers: { 'content-type': 'application/json' },
          body: JSON.stringify({ userAgent: ua, path: new URL(request.url).pathname }),
        }).catch(() => {}),
      )
    }
    return fetch(request)
  },
}

Measuring crawlers and blocking them are different decisions. Some CDNs now block AI crawlers by default — Cloudflare, for instance, lets sites block or charge AI crawlers with managed robots.txt rules. That is a legitimate content-licensing posture, but it has a visibility cost: crawlers that cannot fetch you cannot cite you, and the assistant traffic those citations drive goes to whoever they can fetch.

Our recommendation for most B2B sites: measure first, then decide per crawler class. Training-only crawlers (CCBot, Bytespider) are a fair robots.txt target if training use bothers you; answer-engine crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot) are the ones that turn into cited answers and referral visits — blocking those is usually self-harm. Publish crawler policy in robots.txt (crawlers self-identify their user agents and IP ranges) and keep server-side tracking of what actually hits you.