User-agent: *
Disallow: 
BizBot04 kirk.overleaf.com
HappyBot (gserver.kw.net)
CaliforniaBrownSpider
EI*Net/0.1  libwww/0.1
Ibot/1.0 libwww-perl/0.40    
Merritt/1.0
StatFetcher/1.0
TeacherSoft/1.0  libwww/2.17
WWW Collector
processor/0.0ALPHA libwww-perl/0.20
wobot/1.0 from 206.214.202.45
Libertech-Rover         www.libertech.com?
WhoWhere Robot
ITI Spider
w3index
MyCNNSpider
SummyCrawler
OGspider
linklooker
CyberSpyder (amant@www.cyberspyder.com)
SlowBot
heraSpider
Surfbot
Bizbot003
WebWalker
SandBot
EnigmaBot
spyder3.microsys.com
www.freeloader.com.
205.252.60.71
194.20.32.131
198.5.209.201
acke.dc.luth.se
dallas.mt.cs.cmu.edu
darkwing.cadvision.com
waldec.com
www2000.ogsm.vanderbilt.edu
unet.ca
murph.cais.net (rapid fire... sigh)
spyder3.microsys.com
# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism

User-agent: webcrawler
Disallow:/

User-agent: lycra
Disallow: /

User-agent: *
Disallow: /tmp
Disallow: /logs 
Disallow: /admin
Disallow: /includes
Disallow: /Connections
Disallow: /images