Search rivals buddy up for better crawling

16-11-2006

by Silicon.com

Search engine rivals Google, Microsoft and Yahoo are teaming up to make it easier for website owners to make sure their sites get included in their web indexes.

The trio is adopting Google's Sitemaps protocol, available since June 2005, which allows website owners to manually feed their pages to Google and to check whether their sites have been crawled. Website owners have had to follow similar processes at each of the other major search engines separately.

Now, website owners will be able to go to one place for alerting the search engines to their web pages, something they have been requesting for some time, said Tim Mayer, director of product management at Yahoo Search.

A joint news release said: "In the first joint and open initiative to improve the web crawl process for search engines, Google, Yahoo and Microsoft today announced support for Sitemaps 0.90, a free and easy way for webmasters to notify search engines about their websites and be indexed more comprehensively and efficiently, resulting in better representation in search indices. For users, Sitemaps enables higher quality, fresher search results."

The effort was initially started by Google and Yahoo. Mayer said: "We thought it would be great for publishers and webmasters to be able to submit their content in one format for all the different search engines. We are proposing the format together and inviting other search engines to adopt it."

Ken Moss, general manager of Windows Live Search at Microsoft, said: "Windows Live Search is happy to be working with Google and Yahoo on Sitemaps to not only help webmasters but also help consumers by delivering more relevant search results so they can find what they're looking for faster. I am sure this will be the first of many industry initiatives you will see us working and collaborating on."

The manual web page submission process supplements the traditional web crawling and does not automatically guarantee the pages will be included in a search engine's index.

Elinor Mills writes for CNET News.com

Reprinted with permission from Silicon.com