The practical answer is to measure AI crawler traffic at the first layer you control. Eligible Webflow customers can start with Webflow's native AEO analytics. If Cloudflare sits in front of the Webflow site through Orange-to-Orange, you can add Peec's Cloudflare Worker for continuous Crawl Insights. If another CDN, proxy or log shipper already records requests, send those logs to Peec's generic webhook. Use CSV or Common Log Format upload for a historical check, not continuous tracking.
None of those methods proves that an AI answer retrieved the page, cited it, sent a person to the site or produced a lead. They prove that a named crawler made a request and show how the server responded. That is useful evidence, but it is one rung in a longer measurement ladder.
What does an AI crawler visit actually prove?
An AI crawler visit proves that a request with a recognised bot user agent reached the measurement point. The response status tells you whether the request succeeded, redirected or failed. It does not expose the model's later decision process.
Keep these five states separate:
| State | Evidence you need | What it proves |
|---|---|---|
| Allowed to crawl | Published robots.txt rules and a crawlability test | The bot is not blocked by those rules |
| Requested by a crawler | Server, CDN or platform request log | A named bot requested a URL and received a status |
| Retrieved as a source | Answer-engine or visibility-platform source record | A system consulted or surfaced the URL for an answer |
| Cited in an answer | The answer transcript and its inline citation | The URL was presented as supporting evidence |
| Followed by a person who converts | Referral, analytics event and CRM record | A human session reached the site and completed a useful action |
A clean robots.txt file does not prove a visit. A visit does not prove retrieval. Retrieval does not always become an inline citation. A citation does not guarantee a click. A click without a key event or qualified CRM record is not a lead.
This distinction matters when someone asks whether "AI traffic is growing." The answer can mean bot requests, human referrals from AI products or revenue influenced by an AI answer. Report each separately instead of folding them into one headline.
Which Webflow tracking setup should you use?
Choose the lowest-complexity route that produces the evidence you actually need.
| Your current setup | Best first option | Continuous? | Main limitation |
|---|---|---|---|
| Webflow account with eligible AEO analytics | Webflow LLM bot insights | Yes, within the product's retention and plan rules | Plan eligibility and available dimensions must be confirmed with Webflow |
| Cloudflare in front of Webflow through O2O | Peec Cloudflare Worker | Yes | Worker limits and route safety need active management |
| Another CDN, proxy or log shipper captures requests | Peec generic webhook | Yes | Your system must transform and send valid request logs |
| You only have an access-log export | Peec CSV or CLF upload | No | A point-in-time import does not create ongoing monitoring |
Peec documents native connections for Cloudflare and several other infrastructure providers, plus a generic webhook and file upload. It does not document Webflow hosting itself as a native connector. Webflow's own AEO analytics is therefore a separate route, not a Peec integration.
Webflow's current AEO analytics overview uses different plan language from its detailed LLM bot page. Confirm eligibility in the customer's Workspace and with Webflow before promising access. Product access can change, so the account is the final check.
How does Webflow's native AEO analytics work?
Webflow now documents AEO analytics and a dedicated LLM bot insights view. For an eligible account, this is the cleanest first check because no proxy or custom log pipeline is required.
Use it to answer operational questions such as:
- Which recognised AI bots requested the site?
- Which pages received bot requests?
- Did those requests return successful, redirected or failed responses?
- Did bot activity change after a robots.txt or publishing change?
Before using the data in a report, confirm the date range, bot definitions, export fields and account eligibility in the live product. Also inspect a sample export before assuming it can be imported elsewhere.
Do not promise that Webflow's AEO CSV can feed another platform until a real export has been checked field by field. It can still provide an independent operational view of bot requests inside Webflow.
How do Cloudflare Orange-to-Orange and Peec fit together?
Orange-to-Orange, often shortened to O2O, is Cloudflare's supported way for a customer Cloudflare zone to sit in front of a SaaS provider that also uses Cloudflare. Webflow publishes a specific O2O setup guide. Follow that guide rather than applying a generic proxy recipe to a production Webflow hostname.
Once the Webflow domain is correctly routed through the customer's Cloudflare account, Peec can deploy a Worker to the selected zone. Peec's documentation says the Worker captures AI crawler requests and forwards the relevant records into Crawl Insights.
A controlled setup looks like this:
- Confirm the production hostname, Webflow project owner, DNS owner and Cloudflare account owner.
- Confirm that the Webflow O2O route is supported for the site and complete Webflow's documented setup.
- Record the current DNS and rollback state before changing the traffic route.
- In Cloudflare, create a scoped API token with Workers Scripts Edit, Zone Read and Workers Routes Edit for the relevant zone.
- Connect Cloudflare inside Peec and deploy the Worker only to the intended hostname and route.
- Set the Worker route to Fail Open, then test ordinary visitor requests and known crawler records.
- Verify that Crawl Insights receives a request with the correct URL, status and bot identity.
- Watch request volume and errors after launch, then retain a tested rollback step.
Cloudflare's Workers Free plan currently allows 100,000 requests per day, resetting at midnight UTC. Cloudflare says that when a Worker exceeds that daily limit, route behaviour depends on its mode. Fail Open bypasses the Worker, while Fail Closed returns an error page. Peec also recommends Fail Open as a minimum safeguard. This is not merely a reporting preference: a Worker route on the production hostname sits in the visitor path.
High-traffic sites should estimate all requests through the Worker, not just expected AI bot visits. Upgrade the Workers plan or narrow the route if the free allowance is not appropriate. After any change, test a normal page, a form or checkout path if applicable, the sitemap and a nonexistent URL. Crawler measurement is not worth creating a visitor outage.
When should you use Peec's generic webhook?
Use the generic webhook when a system already captures request-level logs but Peec does not provide a native connector for that system. The sender could be a CDN, reverse proxy, log shipper or application edge.
Peec's current webhook documentation accepts arrays of up to 500 log objects per request. The required fields are:
- timestamp in ISO 8601 format
- request method
- full request URL, including scheme and host
- HTTP response status
- the raw User-Agent header value
Optional fields can add country, client IP, referrer, response time and other context. Minimise personal data. A crawler dashboard rarely needs a visitor's full IP address, and the customer's privacy policy and retention rules still apply.
The webhook has nothing to send if no system records the requests. A JavaScript analytics tag in the Webflow page is not a substitute because many crawlers do not execute client-side analytics, and it will not give you the raw server response record. In a direct-to-Webflow setup without native AEO access or a customer-controlled traffic layer, adding a log shipper may mean introducing new infrastructure. Treat that as a separate architecture decision, not a checkbox inside Peec.
When is CSV or CLF upload enough?
File upload is useful for a baseline, a migration or a one-off investigation. Export a representative period, validate the fields, upload it and compare the bot, URL and response patterns. Common Log Format can work when conventional web-server logs are available, while CSV is better for a deliberately mapped export.
It is not continuous tracking. Every new period needs another export and upload. Keep raw User-Agent strings intact, use consistent time zones and include the full hostname. Use the file to answer "what happened in this sample?" rather than "what is happening now?"
What should the customer's implementation brief include?
Ask for architecture facts, not credentials in chat:
- production domain and important subdomains
- Webflow Workspace and Site plan, plus the person who can publish
- DNS provider and account owner
- whether Cloudflare already proxies the Webflow hostname through O2O
- any existing CDN, reverse proxy, edge function or access-log source
- approximate daily request volume
- a sanitised 24 to 48 hour log sample, if logs exist
- the analytics and CRM owner for human referral and lead measurement
- the security contact who can enter scoped credentials in the approved system
The sample should include timestamp, method, full URL, status and raw User-Agent. Do not ask the customer to email passwords, tokens or unredacted logs. Define acceptance before touching DNS: visitor traffic works, the route is Fail Open, rollback is tested and crawler records arrive.
How should robots.txt be handled?
Webflow lets site owners publish custom robots.txt rules. Use that control deliberately. Decide separately whether to allow search or user-request bots and training crawlers, because providers publish different user agents for different purposes.
OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User as distinct controls. Anthropic and Perplexity also publish crawler documentation. Copy current user-agent names from provider documentation rather than from an old blog post, then test the production robots.txt after publishing.
An Allow rule is access policy, not evidence that the bot visited. A Disallow rule can explain missing crawler requests, but an allowed bot may simply have no reason to fetch the page. Keep robots.txt testing beside the request data and label the two signals correctly.
For a wider explanation of why accessible pages can still be absent from answers, read why your website never shows up in ChatGPT. If you want to understand the measurement and execution layer around a visibility platform, see how to build AI-search agents with the Peec MCP and our AI search visibility tools comparison.
What should appear in the final dashboard?
Note incomplete referrers, spoofable user agents and sample limits. Show where the chain breaks instead of blending every signal into one "AI traffic" number.
The bottom line
Start with Webflow's native bot view if the account is eligible. Use Cloudflare O2O plus the Peec Worker when you need continuous, independent crawler logs and already have a safe Cloudflare path. Use a generic webhook only when a real request-log source exists. Use file upload for historical analysis.
Schmitdy is our own managed AI-search service, so this is a disclosure as well as a recommendation. We set up the measurement ladder, fix the technical and content gaps, ship the work and remeasure it, with human review on production changes. We do not treat a bot request as a citation or a citation as revenue. If you want the first read before choosing infrastructure, start with the Schmitdy GEO audit or read how visibility data becomes citation work.
Frequently asked questions
Can Webflow track AI crawler traffic without Cloudflare?
Yes, if the customer's Webflow account is eligible for Webflow's native AEO analytics and LLM bot insights. Confirm access in the live account because Webflow's current documentation uses different plan wording across its pages. Without native access, you need another request-log source or a customer-controlled traffic layer.
Is Peec a native Webflow integration?
No native Webflow connector is documented. Peec documents Cloudflare and other infrastructure connections, a generic webhook, and CSV or CLF upload. A Webflow site can still feed Peec through Cloudflare O2O or another system that captures and sends request logs.
Does an AI crawler visit mean my page was cited?
No. It means a recognised bot user agent requested the URL at the measurement point. Retrieval and inline citation need separate answer-level evidence. Human referral and conversion need web analytics and CRM evidence.
Can I send a Webflow AEO CSV straight to Peec?
Do not assume so. Inspect the export first. Peec's raw request schema needs timestamp, method, full URL, response status and raw User-Agent. The Webflow export must contain fields that can be mapped accurately to those requirements.
What happens when the Cloudflare Worker free limit is reached?
Cloudflare Workers Free currently allows 100,000 requests per day. With a Fail Open route, Cloudflare bypasses the Worker when the limit is exceeded, so the site can remain available while collection pauses. With Fail Closed, visitors can receive an error page. Verify the live route mode and rollback before launch.
