A robots.txt line silently blocking AI agent traffic, and a content platform ranking well but getting almost no clicks — both diagnosed from real Search Console and analytics data, not assumptions.
A report came in that WickedAwesome.tech's robots.txt might be blocking agent search. It was: a User-agent: GPTBot / Disallow: / block had been present since the site's very first commit, added early on while chasing AdSense approval — and it directly contradicted the site's own llms.txt, which existed specifically to help AI agents navigate the site. One crawler explicitly allowed to read the site, another explicitly blocked from it. The fix was a two-line removal, verified locally against a running dev server before shipping.
Before deploying that fix, a full traffic review turned up something bigger. Google Search Console showed real ranking progress — average position improving from roughly 30–40 in late August to single digits by mid-September — but essentially no clicks: 818 impressions and 2 clicks over 28 days, a 0.24% click-through rate. One page alone, a same-week Apple hardware post, had 282 impressions at position 9.4 and zero clicks.
The page titles and meta descriptions were already good — that was ruled out first. The actual gap showed up with a direct check: curl-ing the rendered HTML of several pages turned up zero application/ld+json structured-data blocks anywhere on the site. No Article schema on blog posts, no Review schema with star ratings on hardware reviews. A brand-new, low-trust domain showing up as a plain blue link next to established competitors in search results is easy to skip, even at a good position — structured data is what lets Google render the byline, date, and star ratings that make a result worth clicking.
Fix: Review schema (with reviewRating) added to hardware review pages, Article schema added to blog and build posts — reusing one shared template function rather than duplicating markup per content type. Verified by parsing the rendered JSON-LD as real JSON before deploying, then confirmed live in production via curl immediately after.
One more data point tied the two problems together: of 32 non-bot sessions recorded by Clarity over the review window, 29 had a null referrer, zero clicks, and lasted under 20 seconds on a single page — a pattern much more consistent with automated crawler traffic than human readers. That's exactly the traffic the robots.txt bug had been suppressing.
Because "the title and meta description look fine" isn't a diagnosis — a direct check of the actual rendered output is. Both fixes here came from looking at real GSC and Clarity numbers first, forming a specific hypothesis, and verifying it with a command before writing a single line of code.
We run on Proton
Encrypted email, VPN, Drive & Calendar — Swiss-based privacy, trusted here for 10+ years. Get 1 month free when you sign up.