Welcome, Guest.
Please login or register.

Welcome to the E-Blah Community!
We would like to welcome you to our community and invite you to register an account or login.
Being a registered member is important, as it gives you several advantages over the normal Guest status. After registering you will be able to download files and images, post messages, and access member-only portions of the forum - just to name a few. Registration is quick and simple, and only takes about a minute of your time.

E-Blah Community  ›  General Discussion  ›  Chit - Chat  ›  MSN bot
Users Browsing Forum
Googlebot and 1 Guests

MSN bot  This thread currently has 928 views. Print
1 Pages 1 Recommend Thread
xantica
May 5, 2006, 7:41pm Report to Moderator Report to Moderator

Working hard whilst Justin and Craig fix my board
Posts: 74
Posts Per Day: 0.06
Reputation: 100.00%
Reputation Score: +3 / -0
Time Online: 1 days 11 hours 59 minutes
Anyone know how to stop the ggogle and msn bots crawling my forum , I dont want the hits or the traffic they bring.

Thanks
Logged Offline
Private Message Private message
Justin
May 5, 2006, 8:16pm Report to Moderator Report to Moderator

The E-Blah Developer
E-Blah Programmer
Posts: 15,070
Gender: Male
Posts Per Day: 6.55
Reputation: 93.40%
Reputation Score: +297 / -21
Time Online: 36 days 23 hours 14 minutes
Location: Tallassee, AL
Age: 22
robots.txt files help.


I do installs for $25 and upgrades for $20.
Technical support is always free.

  Donate to E-Blah!  

My Websites: Revolution Reality (My Blog)  |  MinistryTalk.com  |  Portfolio

"But you, O Lord, are a compassionate and gracious God, slow to anger, abounding in love and faithfulness." — Psalm 86:15 NIV
Logged Offline
Site Site Private Message Private message Reply: 1 - 7
xantica
May 9, 2006, 5:46am Report to Moderator Report to Moderator

Working hard whilst Justin and Craig fix my board
Posts: 74
Posts Per Day: 0.06
Reputation: 100.00%
Reputation Score: +3 / -0
Time Online: 1 days 11 hours 59 minutes
Thanks justin, reply appreciated will look later
Logged Offline
Private Message Private message Reply: 2 - 7
xantica
May 9, 2006, 4:47pm Report to Moderator Report to Moderator

Working hard whilst Justin and Craig fix my board
Posts: 74
Posts Per Day: 0.06
Reputation: 100.00%
Reputation Score: +3 / -0
Time Online: 1 days 11 hours 59 minutes
Ok, im thick, i found the file , but what do I put in it ?. or do.
Logged Offline
Private Message Private message Reply: 3 - 7
Mike Brazier
May 9, 2006, 5:13pm Report to Moderator Report to Moderator

E-Blah Member
Posts: 60
Gender: Male
Posts Per Day: 0.06
Reputation: 100.00%
Reputation Score: +2 / -0
Time Online: 2 days 8 hours 36 minutes
Location: Birmingham
Age: 34
Have a look at http://www.robotstxt.org/, it is a useful reference.

Mike.
Logged Offline
Site Site Private Message Private message Reply: 4 - 7
xantica
May 9, 2006, 5:13pm Report to Moderator Report to Moderator

Working hard whilst Justin and Craig fix my board
Posts: 74
Posts Per Day: 0.06
Reputation: 100.00%
Reputation Score: +3 / -0
Time Online: 1 days 11 hours 59 minutes
Thanks Mike, will do
Logged Offline
Private Message Private message Reply: 5 - 7
Martin
May 10, 2006, 1:16am Report to Moderator Report to Moderator

I love E-Blah
10 Series Support Team
Posts: 4,558
Gender: Male
Posts Per Day: 3.16
Reputation: 98.60%
Reputation Score: +211 / -3
Time Online: 35 days 19 hours 17 minutes
Location: UK
Age: 50
Here are a couple of free robot.txt generators.

http://www.webtoolcentral.com/webmaster/tools/robots_txt_file_generator/
http://www.1-hit.com/all-in-one/tool-robots.txt-generator.htm
http://www.yellowpipe.com/yis/tools/robots.txt/

Or

This will stop all robots:

# go away
User-agent: *
Disallow: /


I have attached a robots.txt file so you can see how big they can get. This guy wants to block the nasties but keep Google, Yahoo and MSN etc so had to block each one individually, these came from his server logs.

It is my opinion though that lots of bots ignore this file, I have seen activity on my own site where a banned bot was busy spidering away.

I have had no problems since also adding the following to my template.html

Code
<meta name="robots" content="noindex,nofollow">



This post contains attachments; to download them you must login.




Martin's Reputation: Increase
Logged Offline
Site Site Private Message Private message Reply: 6 - 7
xantica
May 10, 2006, 12:11pm Report to Moderator Report to Moderator

Working hard whilst Justin and Craig fix my board
Posts: 74
Posts Per Day: 0.06
Reputation: 100.00%
Reputation Score: +3 / -0
Time Online: 1 days 11 hours 59 minutes
Thank you Martin as always it is appreciated.
Logged Offline
Private Message Private message Reply: 7 - 7
1 Pages 1 Recommend Thread
Print

E-Blah Community  ›  General Discussion  ›  Chit - Chat  ›  MSN bot

© 2001-2008 E-Blah
About Us - Contact