.Google.com's John Mueller addressed an inquiry about why Google marks pages that are actually prohibited coming from crawling through robots.txt and why the it is actually safe to dismiss the similar Look Console documents concerning those creeps.Robot Web Traffic To Query Guideline URLs.The individual inquiring the inquiry documented that bots were creating web links to non-existent question specification URLs (? q= xyz) to pages with noindex meta tags that are additionally blocked in robots.txt. What triggered the question is that Google.com is crawling the hyperlinks to those web pages, obtaining shut out through robots.txt (without seeing a noindex robots meta tag) at that point receiving shown up in Google.com Explore Console as "Indexed, though blocked by robots.txt.".The person asked the observing question:." Yet listed below's the major concern: why would certainly Google mark webpages when they can't even view the information? What's the perk in that?".Google's John Mueller confirmed that if they can not crawl the page they can not find the noindex meta tag. He additionally makes an interesting mention of the site: hunt operator, suggesting to dismiss the end results considering that the "normal" consumers will not find those outcomes.He composed:." Yes, you're right: if we can't crawl the web page, our company can't see the noindex. That mentioned, if our experts can not crawl the web pages, at that point there is actually certainly not a lot for our team to index. So while you could find some of those web pages with a targeted website:- concern, the average individual won't see them, so I definitely would not bother it. Noindex is likewise fine (without robots.txt disallow), it just indicates the Links are going to find yourself being actually crawled (as well as wind up in the Explore Console document for crawled/not listed-- neither of these standings trigger issues to the remainder of the site). The vital part is actually that you do not create all of them crawlable + indexable.".Takeaways:.1. Mueller's solution validates the limits being used the Website: search progressed hunt driver for diagnostic causes. One of those causes is actually because it is actually certainly not hooked up to the routine hunt index, it's a distinct trait altogether.Google.com's John Mueller discussed the site hunt operator in 2021:." The brief response is that a web site: question is actually certainly not meant to be full, neither utilized for diagnostics reasons.A website question is actually a specific type of hunt that confines the outcomes to a specific website. It is actually generally simply words internet site, a colon, and then the web site's domain.This query limits the outcomes to a particular site. It is actually certainly not suggested to be a comprehensive collection of all the pages coming from that site.".2. Noindex tag without making use of a robots.txt is fine for these type of circumstances where a crawler is linking to non-existent web pages that are actually getting found by Googlebot.3. Links with the noindex tag will definitely create a "crawled/not listed" item in Browse Console and that those won't possess a negative result on the rest of the site.Review the concern and also answer on LinkedIn:.Why would certainly Google.com mark pages when they can't even view the web content?Included Photo by Shutterstock/Krakenimages. com.