Common rejections

robots.txt is blocking your ads

This is the single most confusing disapproval there is. The page ranks perfectly well in Google Search, so you are certain nothing is blocked, and yet every Shopping ad for it is refused. The cause is one line most people have never had a reason to read closely.

What Google requires

Three different crawlers, all allowed

Google reaches your shop with more than one crawler, and Shopping depends on three of them:

Googlebot, which reads your landing pages.
Googlebot-image, which fetches your product images.
AdsBot-Google, which checks the landing page behind an ad.

For a product to be advertised, all three have to be allowed to reach the relevant parts of your site in robots.txt. Search and Shopping are answered by different bots, and being welcome to one says nothing about the other.

Why it is so confusing

AdsBot-Google ignores the wildcard group

Most robots.txt files carry a wildcard group, User-agent: *, meaning "these rules apply to every crawler". That intuition is what makes this fault so hard to spot, because AdsBot-Google is a deliberate exception: it does not obey the wildcard group at all. It only follows rules written for it by name.

So a file like this looks harmless and is not:

User-agent: *
Disallow: /checkout/
Disallow: /cart/

Search crawlers read the wildcard group, avoid /checkout/, and index the rest of the site normally, which is why the page ranks. AdsBot-Google ignores that group entirely. If there is no group named AdsBot-Google that allows the landing pages, or worse, a stray Disallow: / under its name, every Shopping ad pointing at those pages is refused, while Search carries on as though nothing is wrong.

How to fix it by hand

Name the bot, and allow it

Fetch https://yourdomain/robots.txt in a browser and read it in full, including any group written for AdsBot-Google specifically.
If AdsBot is disallowed from your landing pages, add a group that allows them, remembering it will not inherit anything from the wildcard group:
```
User-agent: AdsBot-Google
Allow: /
```
Confirm Googlebot can reach your landing pages and Googlebot-image can reach your image paths, so neither Search nor image review is blocked either.
Watch for a security plugin, CDN rule, or firewall that blocks unfamiliar user agents, which can stop AdsBot even when robots.txt is correct.

Which of our checks apply

The robots checks FeedMender runs

The scan makes one request per domain and tests the three user agents separately, because the whole point is that they do not behave the same way. It is a sample rather than exhaustive coverage: at most three domains, and at most five landing-page paths per domain, over HTTPS. A shop serving from a single domain, which is nearly all of them, is fully covered.

robots_blocks_adsbot_googlelink, blocks
robots_blocks_googlebotlink, blocks
robots_blocks_googlebot_imagelink, blocks

These are one family of code, assembled per crawler, so the same check reads each user agent in turn. It appears on the checks list as *robots_blocks_ under the per-attribute families.

The quicker way

Let the scan read it for you

Because this fault hides behind a page that ranks, it is worth having something check the robots rules for all three crawlers explicitly. The free scan reads your robots.txt and reports separately on Googlebot, Googlebot-image and AdsBot-Google.