The GEO Advice Everyone's Giving
Everyone's suddenly talking about GEO — Generative Engine Optimization, the SEO-shaped discipline of getting your content surfaced and cited by AI answer engines instead of just ranked by Google. Most of the advice reads like SEO advice with new vocabulary: structured data, clear factual statements, an llms.txt file. Good advice, as far as it goes. But none of it answers the question I actually needed answered for my own site: what does an AI crawler get when it fetches my page, given that my page doesn't really exist until a browser runs it?
A Site That Doesn't Exist Until JavaScript Runs
My portfolio, dougrosenbergdev.com, is a Blazor WebAssembly single-page app — a deliberate choice, since I'm a full-stack .NET developer and the site itself is supposed to be proof of that. But it means the actual content — my bio, my experience, my case studies — doesn't live in the HTML. It lives in .razor components that fetch JSON data and render it client-side, after a multi-megabyte WASM runtime finishes booting in the visitor's browser.
Most AI crawlers don't run that runtime. They issue a plain HTTP GET and read whatever HTML comes back. So I did the same thing — fetched my own homepage with nothing but curl — to see what's actually there.
What survives a plain GET: the <head> — meta description, Open Graph tags, three blocks of Schema.org JSON-LD (Person, WebSite, SoftwareApplication), and a static HTML snapshot of the hero section that I'd built for an unrelated reason (avoiding a layout flash while Blazor boots). That snapshot happens to double as crawlable content, which was a nice accident.
What doesn't survive: everything else. My About section, my full work history, the consulting page — all of it sits behind a loading spinner until WASM finishes booting. To a non-JS crawler, my portfolio is a name, a job title, and a spinner.
The Twist: My Own robots.txt Was Blocking the Fix
I went looking for where the real bio and experience content lives, expecting to find it buried inside compiled Blazor component state. Instead it's sitting in plain JSON files — served as ordinary static assets, fetched at runtime via HttpClient.GetFromJsonAsync. Structurally, those files are exactly as crawlable as any other file on the site. A crawler that knew to look could read my actual bio in plain JSON without executing a single line of WASM.
Except my own robots.txt was stopping it: Disallow: /*.json. A blanket rule, presumably written to keep crawlers out of Blazor's framework manifests, was also blocking the one place my real content already existed in crawler-readable form.
What I Shipped
The first fix wasn't architectural — it was two small changes:
- Removed the blanket JSON disallow, keeping
/_framework/blocked on its own (that's where the actual noise is — build manifests, boot config, nothing worth indexing). - Added an
llms.txtat the site root: a plain-text, purpose-built summary of who I am and what's on the site, following the small emerging convention for exactly this problem — a place for AI systems to get a straight answer without needing to parse or render anything.
Both are live now. Neither fixes the underlying issue.
What's Still Broken
Per-page content (/consulting, /webdesign, /blog) is still invisible to anything that won't run WASM, per-page <PageTitle> and meta tags included. That needs actual prerendering: either Blazor's built-in static prerendering support, or a build-time step that snapshots each route to static HTML for crawlers while the WASM app takes over for real visitors. That's a meatier problem, and one I think is more interesting than "just switch frameworks" — because the honest answer for anyone building a Blazor WASM app that cares about being found isn't to abandon WASM, it's to understand precisely which parts of the page a plain GET request can see, and design around that boundary on purpose.
The Takeaway
If you're running a Blazor WASM, Angular, or any other client-rendered SPA and haven't checked what a curl of your own homepage actually returns — do that before you write a single line of GEO advice into a meta tag. You might be surprised what's missing.