Spiders are software programs developed by the search engines that crawl the web for websites just to add them to their database. Spiders "read" your meta tags and see how far they are allowed to crawl into your website. (index-follow tag).
The list of search engine spiders you can expect to find in your log-file:
108.Muscat Ferret
109.Mwd.Search
110.NetCarta WebMap Engine
111.NetMechanic
112.NetScoop
113.newscan-online
114.NHSE Web Forager
115.Nomad
116.The NorthStar Robot
117.Occam
118.HKU WWW Octopus
119.Orb Search
120.Pack Rat
121.PageBoy
122.Patric
123.The Peregrinator
124.PerlCrawler 1.0
125.Phantom
126.PiltdownMan
127.Pioneer
128.html_analyzer
129.Portal Juice Spider
130.PGP Key Agent
131.PlumtreeWebAccessor
132.GetterroboPlus Puu
133.The Python Robot
134.RBSE Spider
135.Resume Robot
136.RoadHouse Crawling System
137.Road Runner
138.Robbie the Robot
139.ComputingSite Robi/1.0
140.Roverbot
141.SafetyNet Robot
142.Scooter
143.Search.Aus-AU.COM
144.SearchProcess
145.Senrigan
146.SG-Scout
147.Shai'Hulud
148.Sift
149.Simmany Robot Ver1.0
150.Open Text Index Robot
151.SiteTech-Rover
152.Inktomi Slurp
153.Smart Spider
154.Snooper
155.Solbot
156.Spanner
157.SpiderBot
158.SpiderMan
159.Spry Wizard Robot
160.Site Searcher
161.Suke
162.Sven
163.TACH Black Widow
164.Tarantula
165.tarspider
166.Tcl W3 Robot
167.TechBOT
168.Templeton
169.TitIn
170.TITAN
171.The TkWWW Robot
172.TLSpider
173.UCSD Crawl
174.UdmSearch
175.URL Check
176.URL Spider Pro
177.Valkyrie
178.Victoria
179.vision-search
180.Voyager
181.VWbot
182.The NWI Robot
183.W3M2
184.the World Wide Web Wanderer
185.WebBandit Web Spider
186.WebCatcher
187.WebCopy
188.webfetcher
189.The Webfoot Robot
190.weblayers
191.WebLinker
192.WebMirror
193.The Web Moose
194.WebQuest
195.Digimarc MarcSpider
196.WebReaper
197.webs
198.Websnarf
199.WebSpider
200.WebVac
201.webwalk
202.WebWalker
203.WebWatch
204.Wget
205.WhoWhere Robot
206.w3mir
207.WebStolperer
208.The Web Wombat
209.The World Wide Web Worm
210.WWWC Ver 0.2.5
211.WebZinger
212.XGET
213.Nederland.zoek
Meta tags Summary - Important SEO things to remember
Gone through all the steps and made changes to your website? Did you add relevant meta tags to your website ? We suggest that you use the Meta tags in your index.htm or index.html page. Create relevant different meta tags for each and every page.