Google Indexing Issue With No Follow, No Index

Hi All!

I have have been having an issue with Google displaying my site on Search for about the past 8 months. I had installed the Ultimate SEO plugin about last November on my v 1.5.8 site and that is when my site's Google Search results rapidly started to decline. I removed the module, thinking that this was the problem, and have requested site indexing from Google once again and waited for results to recover as Google states it could take months.


My website (nitwitcollections.com) has been active for nearly 18 years and we previously derived 30% of our sales from organic Google Search....so this has obviously impacted our site in a very negative way.

I have again been searching for reasons why it is not showing. I do not have anything in my robots.txt that prevents it, but in using Google Search Console do a live URL inspection of a recently released product, GSC results show "URL is Not Available to Google" "Page cannot be indexed: Excluded by ‘noindex’ tag" and under indexing allowed it states "No: 'noindex' detected in 'robots' meta tag"

I see in the /templates/common/html_header file that the code is:

<?php if (defined('ROBOTS_PAGES_TO_SKIP') && in_array($current_page_base,explode(",",constant('ROBOTS_PAGES_TO_SKIP'))) || $current_page_base=='down_for_maintenance' || $robotsNoIndex === true) { ?>
<meta name="robots" content="noindex, nofollow">
<?php } ?>

Is this noindex, nofollow code causing my problem? If so, can I change this to ALL or is there a setting in Admin->Config that is overriding?

Any help would be greatly appreciated.

Jeff

The code in the header file looks okay.

Adding an url rewrite is no problem for your SERP because the indexed pages are still available with their dynamic urls and will be rewritten when a page is loaded.
The problem is when Google has re-indexed your urls with the new SEO urls and you then go back to dynamic urls by removing the SEO plugin.

Normally there is only a robots sample file and you normally don't need one at all.
If you have a robots.txt then rename it and try GSC again.

Thanks for your help Shop Suey!

I have tried changing the <meta name="robots" content="noindex, nofollow"> to <meta name="robots" content="index, follow"> and run a live URL test in GSC and it corrected this problem of Google not being able to index the page. I tried this in 3 other product pages from my site (live URL test) and they have now all passed and can and will be indexed.

Thanks for the heads up on the robots.txt I will also make this change and have GSC recrawl my site.

Thanks again!

You're welcome

The header code is intended to give the crawler the advise to noindex stuff like popup images, under construction sites and so on.
It's good that you can solve the problem at the moment but not the code itself is the problem (as far I could compare to mine).
You should continue your analysis.

BTW: your file probably should be under includes/templates/<your-template>/common/

Thanks again for your help Shop Suey....and for letting me know that this likely is not the culprit.

I will continue to monitor and and search for what is causing my issue.

Quote:

---

Originally Posted by NWCE

PHP Code:

---

<?php if (defined('ROBOTS_PAGES_TO_SKIP') && in_array($current_page_base,explode(",",constant('ROBOTS_PAGES_TO_SKIP'))) || $current_page_base=='down_for_maintenance' || $robotsNoIndex === true) { ?>
  <meta name="robots" content="noindex, nofollow">
<?php } ?>

---

Is this noindex, nofollow code causing my problem? If so, can I change this to ALL or is there a setting in Admin->Config that is overriding?

---

If your store is outputting the noindex,nofollow tag, then it means one of the several conditions above it is true:
- the URL of the page is main_page=xyz where xyz is in the ROBOTS_PAGES_TO_SKIP list (most stores never need to touch that list; found in extra_configures/robots_pages_to_skip.php on latest ZC version, was in meta_tags.php language file in older versions)
- the URL of the current page is "down_for_maintenance", because ...er, well, the store is down for maintenance according to the config switch in your Admin
- $robotsNoIndex is set to true

What sets $robotsNoIndex to true:
- invalid category specified in browser URL by customer visiting the page
- product is not found in the database
- product is in the database but status is turned off

Thank you for shedding light on the extra_configures/robots_pages_to_skip.php Dr.Byte....I really appreciate it....as you know this cart inside and out!

I will definitely look into this and insure that it is not causing the problem as I have eliminated your other possible options as to why a product page is not being indexed.

Thanks again for your help!

One other thing to check: If you added the product by any means other than using the Admin, did the product description get saved properly? One way a product might become "not found" is if the "description" portion of the record (which is stored in a separate db table) isn't present. It's kinda unusual for that to happen, especially if a store has only 1 language set up. But I mention it for the sake of thoroughness.

Thanks for this additional info and to check on to insure it is not causing me the problem!

I will double check these....you never know if there was a glitch during insertion of a product....

Thanks again....much appreciated!