Yahoo's Slurp is All New

Yahoo released its new and polished spider Slurp to crawl it's index. Reports were slow coming in...But they are all over now. Some sites gained position and some lost, just like any other shuffle.

Most of mine remained relatively the same. I think there are some important things to note in the new Slurp. The first thing is you will probably notice in your logs, as I have noticed in all of my sites, Slurp rolled in grabbed the robots.txt, the y key and maybe one or 2 files. At no time did Slurp access any of my sitemap.xml files. With what Slurp accessed I am guessing this was just a verification preliminary crawl. Now mind you, I only have the data from 11 sites that I maintain to use. However, in each and in every one of those site's logs that is what I saw. I also saw a great deal of action from Yahoo Feed Seeker on my site with feeds, these visits were fairly relative to Slurp's visit.

So my summation is this, if your site was dropped from queried results as a result of this update then look at your Yahoo Site Explorer for errors, check your sitemap..maybe make a new one and ping, and verify your robots.txt.

It is very important that you are allowing Slurps new protocol in your robots.txt if you specify access by bot. Below are the old and new bots and their information.

Old Slurp
72.30.76.134 - - [22/Aug/2007:17:03:40 -0500] "GET / HTTP/1.1" 500 627 "-" "Yahoo! Slurp/Site Explorer"

New Slurp
74.6.24.202 - - [23/Aug/2007:23:16:37 -0500] "GET /robots.txt HTTP/1.0" 200 28 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

Yahoo Feed Seeker
216.39.58.17 - - [21/Aug/2007:22:33:13 -0500] "GET /robots.txt HTTP/1.0" 304 - "-" "YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)"

I think its also important to note that Yahoo appears to be totally supporting wild cards in your robots.txt now. I found this page while searching for the new information on Slurp.

Keeping my ear to the ground on this one. Yahoo is making some bold moves, as they need to do. However, our jobs just got a little more hectic.

Peace and SEO

Melanie Prough
"Baby"


**We Require a Link Back Please.