Google Modifies Robots.txt’s Unsupported Fields Not Crawled Rules
Google just announced significant updates in its robots.txt policy: unsupported fields within the file will be totally neglected. Such actions thrill business owners and marketers because it has to do with how their websites are crawled by Google bots and indexed, so the change does not go too far in business. Still, the file is […]