.Google.com's John Mueller addressed a question regarding why Google.com marks pages that are actually refused coming from creeping through robots.txt and also why the it's secure to neglect the relevant Browse Console records about those creeps.Crawler Website Traffic To Question Criterion URLs.The individual talking to the question documented that crawlers were producing hyperlinks to non-existent inquiry parameter Links (? q= xyz) to web pages with noindex meta tags that are actually additionally shut out in robots.txt. What triggered the concern is actually that Google.com is actually creeping the hyperlinks to those webpages, getting obstructed through robots.txt (without watching a noindex robots meta tag) at that point obtaining reported in Google Look Console as "Indexed, though blocked out through robots.txt.".The individual talked to the adhering to question:." But below's the large inquiry: why would certainly Google index pages when they can't even see the web content? What is actually the benefit because?".Google.com's John Mueller verified that if they can not crawl the page they can not see the noindex meta tag. He likewise produces an intriguing mention of the website: hunt operator, encouraging to ignore the outcomes since the "typical" users won't see those outcomes.He wrote:." Yes, you're appropriate: if our company can not crawl the page, we can't find the noindex. That said, if our experts can't creep the pages, after that there's certainly not a great deal for our company to index. So while you may view a number of those pages along with a targeted site:- query, the typical user will not view them, so I would not bother it. Noindex is actually likewise alright (without robots.txt disallow), it merely means the URLs will definitely end up being crawled (and find yourself in the Search Console record for crawled/not catalogued-- neither of these standings induce issues to the rest of the web site). The vital part is actually that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's response affirms the limits being used the Web site: search progressed search operator for analysis explanations. Some of those main reasons is given that it's not hooked up to the routine hunt mark, it is actually a separate point completely.Google's John Mueller commented on the web site search operator in 2021:." The brief answer is that a site: inquiry is actually certainly not implied to be full, nor utilized for diagnostics objectives.A website concern is a certain type of search that limits the end results to a specific website. It's essentially merely the word site, a digestive tract, and after that the web site's domain.This inquiry confines the end results to a specific website. It's certainly not indicated to become a detailed compilation of all the web pages from that web site.".2. Noindex tag without using a robots.txt is actually fine for these type of situations where a bot is actually connecting to non-existent pages that are actually receiving uncovered by Googlebot.3. URLs with the noindex tag will produce a "crawled/not recorded" item in Explore Console and also those will not have an adverse impact on the rest of the web site.Read the question and address on LinkedIn:.Why would certainly Google.com index pages when they can't also find the material?Featured Photo by Shutterstock/Krakenimages. com.