Check If Certain Content on Your Webpage is SEO Friendly


Know if specific content on your webpage is SEO friendly & will be picked by Googlebot during indexing.

What it means?

You generally expect Googlebot to index all the content on your page. But, the bot may not always index your page’s dynamic content. This is the content that requires user-action (such as scrolling) or JavaScript execution to appear on our page. Google Search Console (or Google Webmaster) provides tools to verify if the content you expect Googlebot to index is being indexed. And in case you don’t have Google Search Console access, you can still access your page as Googlebot to decipher this.

[Google Search Console] Check if certain content appeared in the Crawled Copy of your page:

Login to Google Search Console and enter the URL you want to verify to access Google’s crawled copy of your page:


Viewing Google's crawled copy of your page (from Google Search Console)
Viewing Google’s crawled copy of your page (from Google Search Console)

Once you copy the HTML source of the crawled page, simply search for your desired content within the HTML source:

  • If this is a certain text, it should be present within the HTML source of your crawled page copy.
  • If this is an image, it should be present within the src attribute of an <img> tag or within srcset attribute for a <picture> tag.

If the content doesn’t appear as explained above, Googlebot may not be indexing that content.

[Google Search Console] Check if Googlebot will index certain content in next crawl:

If the crawled copy of your page doesn’t contain the content you intend Googlebot to index, you may change your page to fix the issues. Once fixed, you can test if future Googlebot crawls will pick the content you intend to index. This can be achieved by testing live URL from Google Search Console by clicking ‘Test Live URL’:


Viewing how your live URL will appear to Googlebot during next crawl
Viewing how your live URL will appear to Googlebot during next crawl

For above-the-fold content, you can view how the content renders and for rest of the page, you can copy the HTML source of the page and search for your desired content within the HTML source.

[Without Google Search Console] Check if Googlebot will index certain content in next crawl:

Without access to Google Search Console, there are various ways to fetch your page as Googlebot. With each of these methods, you need to ensure you fetch the page with Googlebot’s User-Agent string:

Here’s the User-Agent string for mobile:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

And here’s one for desktop:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/74.0.0.0 Safari/537.36

  • Via Chrome Devtools: Using the User Agent string above, follow this guide to override User-Agent string within Chrome devtools.
  • Via cURL command: Run the following cURL command on Linux command-line or online here:
  • For Mobile (replace with your URL in the snippet below):
  • curl '<ENTER_YOUR_URL_HERE>' \
      -H 'cache-control: no-cache' \
      -H 'user-agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.0.1 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
      -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
      -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
    

  • For Desktop (replace with your URL in the snippet below):
  • curl '<ENTER_YOUR_URL_HERE>' \
      -H 'cache-control: no-cache' \
      -H 'user-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/74.0.0.1 Safari/537.36' \
      -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
      -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
    

Once you obtain the HTML response from any of the above methods, you can search for the specific content within the HTML. If present as a regular HTML content / tag, it shall be picked by Googlebot during next crawl. If it requires execution of JavaScript, it may or may not be picked by the Googlebot during the next crawl immediately.