Improving on Robots Exclusion Protocol

<<Web publishers often ask us how they can maximize their visibility on the web. Much of this has to do with search engine optimization — making sure a publisher’s content shows up on all the search engines. However, there are some cases in which publishers need to communicate more information t…>>   (more…)

See original post by Chris Gilmer