Why Your Recruitment Agency Isn't Showing Up in AI Search

A recruitment agency doesn't show up in AI search for one of seven reasons: the AI crawler is blocked from the site, the jobs and copy only appear after JavaScript runs, the facts are hard for an engine to lift, the site never says plainly what the agency recruits and where, no page answers the question the buyer asked, a page answers it without naming the agency, or no other website mentions the agency. Each one is explained below twice: the technical reason, for whoever looks after your website, and the same thing in recruiter terms. Every screenshot is real, taken from live sites and from Found, our AI search tool, running on my own agency.

Full transparency: some of these screenshots show my own agency getting it wrong. AYM Recruitment places people across EV charging in the US, and it runs on Redsun. If we can trip over these, so can anyone.

1. The AI's robot is turned away at the door

The technical reason

Before ChatGPT, Perplexity or Claude can quote your site, their crawler has to fetch it. Your robots.txt file and your firewall decide whether it can. OpenAI runs separate crawlers for separate jobs: GPTBot gathers training data, and OAI-SearchBot decides whether ChatGPT search can show and cite your pages. OpenAI's own documentation says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Since 1 July 2025, Cloudflare has also asked every new domain whether to allow AI crawlers, with blocking selected by default, and that block never shows up in robots.txt.

In recruiter terms

Picture a receptionist told to turn away every headhunter who calls, including the one ringing with a client who wants to hire you. That's what a blocked crawler is. The good news: in our audit of 391 agency websites, 98% of the sites that responded left the door open, so this is the least common cause. If it does apply to you, though, nothing else in this guide matters until it's fixed.

Check it: open yourdomain.co.uk/robots.txt and look for Disallow: / under OAI-SearchBot, PerplexityBot or User-agent: *. Then ask whoever manages your domain whether an AI bot block is switched on in Cloudflare or your hosting.

2. Your jobs only exist on the screen, not in the page

The technical reason

A browser downloads your page, runs its JavaScript and draws the jobs list. The major AI crawlers download the page and stop there: they fetch JavaScript files but don't run them (Vercel and MERJ, December 2024). A jobs board pulled in by a widget or iframe after the page loads doesn't exist as far as they're concerned. Googlebot does run JavaScript, so a page can look fine in Google and be empty to ChatGPT.

In recruiter terms

It's a job board that only shows roles once you've logged in. The AI is a visitor who never logs in, so it sees a page called "Live jobs" with nothing on it. That hurts more than it sounds, because your live roles are the best proof you have of what you actually recruit. A client asking "who places procurement managers in EV charging?" needs your procurement manager role to be readable.

Check it: open your jobs page, right-click and choose "View page source". Press Ctrl+F (Cmd+F on a Mac) and search for the title of a job you know is live. If it's not there, the AI can't see it either.

3. Your facts are hard for an engine to lift

The technical reason

Engines prefer facts they can take with confidence: text on the page rather than inside images or PDFs, questions as headings with the answer directly underneath, structured data (schema markup) that labels the organisation, the FAQ and the author, and an llms.txt file, a plain-text summary some AI tools read to understand who a site belongs to. In our audit of 391 agency websites, 3% had FAQ markup, 7% had article markup, and none had an llms.txt.

In recruiter terms

It's sending a client your terms as a scanned PDF of a fax. The information is there, but nobody can copy it into the approval form, so it gets parked. Structured data and llms.txt are the filled-in form: a cover letter written for the AI, saying who you are and which pages to read. Here's AYM's.

Check it: type yourdomain.co.uk/llms.txt into a browser. Then look for any fee, salary or key fact on your site that only lives in an image or a PDF, and put it in the page text.

4. Your site never says what you recruit, or where

The technical reason

Engines match the words in a question against the words on your pages. A buyer asking for "finance recruiters in Leeds" gets matched to pages that say finance, recruiters and Leeds, close together and near the top. Slogans carry no matchable terms, and they often end up in the page's structured data too: AYM's own organisation description in its markup is the slogan "Powering up the future, one hire at a time."

In recruiter terms

You'd never shortlist a CV that says "hard-working team player" and lists no job titles, sectors or locations. The AI shortlists agencies the same way. In September we ran 100 agency websites through our AI readiness grader; of the 94 where it could read the copy, none passed the check for wording that could only describe that agency. And yes, that includes us. Look at AYM's homepage below: the big line is a slogan. The small line above it, "Recruitment built for the EV sector", is the one doing the work.

Check it: read the first two lines of your homepage as a stranger. Can you tell from those alone what you recruit, where, at what level and on what terms? If not, that's your first rewrite.

5. No page answers the question the client asked

The technical reason

Engines answer a question by quoting pages shaped like that question. A page titled "Charge point operator recruitment" that says who you place, at what level, how fast and what it costs gets quoted. A homepage with a sectors dropdown doesn't. Found reads an agency's site and matches it against the questions being tracked. For AYM, it read 22 pages and found that the site answers 11 of 17 tracked questions directly, 3 in part, and 3 not at all.

In recruiter terms

A client asks you for someone with DC fast-charging experience. You've got a candidate who has it, but it's not on their CV, so they never get put forward. Your website works the same way. The engine is building a longlist, and agencies with a page for the exact question make it. The rest don't, however good they are.

Check it: write down the ten questions your best clients asked before they hired you, then find the page on your site that answers each one in its first paragraph. Most agencies find two or three.

6. The engine answers the question and names someone else

The technical reason

An engine can read your page, use your facts, and still credit a competitor, because it names whoever states their name next to the answer most clearly. We caught ourselves doing this. We track the question "Which recruitment website platforms integrate with JobAdder?". Redsun integrates with JobAdder and has a page about it. Across 114 answers, engines cited that page three times and named Redsun once, while two other platforms were named in 60 and 54. The page explained the integration in detail and never said, in one sentence, that Redsun is a recruitment website platform that integrates with JobAdder. We fixed it on 21 October.

In recruiter terms

It's the market report your team spent a week on, forwarded around the industry with your logo cropped off. You did the work and someone else got the credit. Below is a real answer Gemini gave on 20 October to "Which EV charging recruitment agencies actually deliver?". It named four firms and LinkedIn. AYM, which recruits nothing but EV charging, wasn't one of them.

Check it: on each page you want quoted, find the first sentence with your agency's name and what you do. If it isn't in the first paragraph, move it there, and write it so it would still make sense copied out on its own: "Agency name recruits X for Y in Z."

7. Nobody else vouches for you

The technical reason

Engines don't take your site's word for it. They weigh what the rest of the web says: job boards, LinkedIn, directories, trade press, partner pages. Over the last 90 days, the engines answering client questions in AYM's market drew on 500 different websites. Two of them were AYM's. The most cited source was a niche EV job board, cited 661 times; AYM's own site was second at 428, and LinkedIn third at 383.

In recruiter terms

References. You wouldn't put forward a candidate nobody would vouch for, and the AI won't recommend an agency that only talks about itself. The agencies that get named turn up on the sector job board, have a LinkedIn company page that says the same thing as their website, and get mentioned in the trade press or on a partner's site.

Check it: ask ChatGPT and Perplexity the question your best client asked, and read the sources listed under the answer. Those are the sites the AI trusts in your market. Count how many of them mention you.

How to see which of these is costing you

The technical reason

Engines give different answers to the same question on different days and in different tools, so one test proves nothing. You need the same questions asked repeatedly, across ChatGPT, Gemini, Perplexity and Google's AI answers, with a record of who was named and which pages were read.

In recruiter terms

Every desk tracks its fill rate. Found tracks the same thing for AI answers. From 17 August to 14 September, AYM was named in none of the answers to its tracked client questions, in any week. AYM's new Redsun site went live on 18 September, with its jobs and copy in the page code, structured data on its pages and an llms.txt. It has been named every week since: 15% of 1,141 client answers over 90 days, and 25% of 908 candidate answers.

That's one agency in one niche, and we've added questions as we've gone, so read it as direction rather than a promise. It shows the fixes move the number. How far yours moves depends on your market and where you start.

The ten-minute check

Cause	What to check	Time
Robot turned away	robots.txt for OAI-SearchBot and PerplexityBot; ask about a Cloudflare AI bot block	2 min
Jobs only on screen	View page source on your jobs page and search for a live job title	1 min
Facts hard to lift	Open yourdomain/llms.txt; look for facts stuck in images or PDFs	1 min
Vague specialism	First two lines of your homepage: sector, place, level, terms?	1 min
No page for the question	List ten client questions; find the page that answers each	3 min
Answered but not named	Is your name and what you do in the first paragraph?	1 min
Nobody vouches	Read the sources under an AI answer in your market	1 min

Fix the first two first. They're the only causes that stop an engine reading your site at all, and they're usually a single setting or a single widget.

Fix the infrastructure once, then use data for the rest

Causes 1 to 3 are infrastructure, and Redsun fixes them out of the tin. Every Redsun website lets the AI crawlers in, writes every job and every line of copy into the page's code, adds structured data to every page and publishes an llms.txt for you. Causes 4 to 7 need data: which questions your clients ask, which of your pages the engines read, who gets named instead of you, and which sites they trust. That's where Found gives you the edge.