クローラーの邪魔をしたい場合

.htaccess

BrowserMatchNoCase Googlebot     robot
BrowserMatchNoCase Slurp         robot
BrowserMatchNoCase msnbot        robot
BrowserMatchNoCase proodleBot    robot
BrowserMatchNoCase psbot         robot
BrowserMatchNoCase ScSpider      robot
BrowserMatchNoCase TutorGigBot   robot
BrowserMatchNoCase YottaShopping robot
BrowserMatchNoCase Faxobot       robot
BrowserMatchNoCase Gigabot       robot
BrowserMatchNoCase MJ12bot       robot
BrowserMatchNoCase Baidu         robot
deny from env=robot

確認（Perlモジュールを使う）

$ lwp-request -mHEAD -H 'User-Agent: slurp' http://sample.com/img/1.jpg
$ lwp-request -mHEAD -H 'User-Agent: slurp' http://user:password@testserver.com/img/1.jpg