Can Google Robot crawl html files within protected folders? In other words, do I need the no index meta tag for these files. I have been reading up on Google Web Tools and it implies that Google doesn't crawl files that it doesn't have access to. It is logical that these files don't get crawled. Thanks.