The greater pertinent pages are coupled with each other when crawled continuously, and as the crawling frequency rises, so does the overall rank in search engines like yahoo.A robots meta tag is definitely an HTML snippet that tells search engines like yahoo the way to crawl or index a certain page. It’s positioned in to the segment of a webpage