.Google.com's John Mueller addressed a question concerning why Google.com marks webpages that are refused from creeping by robots.txt as well as why the it's safe to neglect the related Browse Console documents regarding those crawls.Robot Web Traffic To Question Guideline URLs.The person asking the inquiry recorded that robots were actually creating hyperlinks to non-existent query guideline URLs (? q= xyz) to web pages along with noindex meta tags that are actually also blocked in robots.txt. What caused the question is actually that Google.com is crawling the links to those webpages, obtaining shut out by robots.txt (without seeing a noindex robots meta tag) at that point getting shown up in Google Look Console as "Indexed, though blocked out by robots.txt.".The individual inquired the observing question:." However here's the major inquiry: why will Google index pages when they can't even see the material? What is actually the benefit during that?".Google.com's John Mueller validated that if they can not creep the page they can't observe the noindex meta tag. He additionally creates an interesting acknowledgment of the internet site: hunt driver, advising to dismiss the end results since the "normal" consumers will not find those end results.He wrote:." Yes, you're right: if our experts can not crawl the web page, our team can't find the noindex. That pointed out, if we can not crawl the webpages, at that point there is actually not a whole lot for us to index. Thus while you might view some of those web pages along with a targeted web site:- question, the normal user will not view all of them, so I would not fuss over it. Noindex is likewise great (without robots.txt disallow), it simply indicates the URLs will certainly wind up being crawled (and end up in the Look Console report for crawled/not listed-- neither of these conditions induce problems to the rest of the site). The integral part is that you don't produce them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the limitations being used the Web site: search evolved hunt driver for diagnostic causes. Among those factors is considering that it is actually not attached to the normal search index, it is actually a separate factor entirely.Google.com's John Mueller talked about the site search operator in 2021:." The quick answer is that a web site: concern is actually not indicated to be full, neither utilized for diagnostics objectives.An internet site question is actually a details sort of search that confines the results to a certain site. It's generally simply words website, a bowel, and afterwards the web site's domain name.This query restricts the results to a certain internet site. It's not indicated to become a detailed collection of all the webpages from that web site.".2. Noindex tag without making use of a robots.txt is alright for these kinds of scenarios where a robot is actually linking to non-existent webpages that are obtaining uncovered through Googlebot.3. URLs along with the noindex tag will definitely produce a "crawled/not indexed" item in Explore Console and also those won't have a bad result on the rest of the site.Check out the inquiry and respond to on LinkedIn:.Why would Google.com mark pages when they can't even view the information?Included Photo through Shutterstock/Krakenimages. com.