Welcome to the E-Blah Community!
We would like to welcome you to our community and invite you to register an account or login.
Being a registered member is important, as it gives you several advantages over the normal Guest status. After registering you will be able to download files and images, post messages, and access member-only portions of the forum - just to name a few. Registration is quick and simple, and only takes about a minute of your time.

E-Blah Community    Informational    Forum Updates  ›  Making Your Forum Bot Friendly
Users Browsing Forum
No Members and 2 Guests

Making Your Forum Bot Friendly  This thread currently has 2,306 views. Print
1 Pages 1 Recommend Thread
Justin
September 11, 2005, 1:02pm Report to Moderator Report to Moderator

The E-Blah Developer
E-Blah Programmer
Posts: 15,000
Gender: Male
Posts Per Day: 6.79
Reputation: 93.38%
Reputation Score: +296 / -21
Time Online: 36 days 16 hours 47 minutes
Location: Tallassee, AL
Age: 21
This tuturial uses the Apache mod_rewrite modification to your forum in versions of E-Blah greater than Platinum 9.3.

This post is for users wishing to make their forums bot friendly.

First and formost, versions of E-Blah prior to Platinum 9.3 are NOT bot friendly, therefore this tuturial will not be benifical to you until you upgrade.

First thing you should do is create a Google Sitemap.  A good tuturial for doing this can be found here: The CLiX Network.

The next thing you should do is tell Google NOT to index the following directories (as they shall be called).  To do this, we must make a robots.txt file.  This file is simple, yet very effective.  Google likes to index all of your pages.  This is both good and bad.  For instance, you don't want all the quote pages indexed so when users search for "How to use Quote" they get a listing of 2,000 pages refrencing the actual quote page.  You also don't want your post page, period, to be index.  This is how your robots.txt file should look:


Code
User-agent: *
Disallow: /forum/v-post/
Disallow: /blahdocs/
Disallow: /forum/v-report/
Disallow: /forum/v-recommend/
Disallow: /forum/v-memberpanel/
Disallow: /forum/v-members/
Disallow: /forum/v-cal/
Disallow: /forum/v-search/
Disallow: /forum/v-register/
Disallow: /forum/v-login/
Disallow: /forum/v-print/a-1/



Of course if you are using, for instance, /bb/ for your directory use /bb/v-post/.

This code will be updated periodically so that you can use block various diffrent URL's which are not needed in the index.  All you really want in your index is the Board Index, Message Index, and Message Display pages.

Applying this simple scheme will help your forum be indexed by Google, and, hopefully, gain you new members and also help Google figure out with pages to index and not index.  Over the next few weeks I hope to get the eblah.com domain cleaned up of trash on Google and get this new scheme of things working.  I also plan to add a Google site search link so that users can search via Google and -- probably -- get more results than if using the E-Blah built in search engine.  Stay tuned for that.  

- Justin


I do installs for $25 and upgrades for $20.
Technical support is always free.

  Donate to E-Blah!  

My Websites: Revolution Reality (My Blog)  |  MinistryTalk.com  |  Portfolio

"But you, O Lord, are a compassionate and gracious God, slow to anger, abounding in love and faithfulness." — Psalm 86:15 NIV


Revision History (3 edits)
admin  -  September 12, 2005, 8:12pm
admin  -  September 11, 2005, 4:17pm
admin  -  September 11, 2005, 1:02pm
Logged Offline
Site Site Private Message Private message
Tim Linden
September 11, 2005, 1:05pm Report to Moderator Report to Moderator

Who IS that guy?
Administrator
Posts: 100
Posts Per Day: 0.04
Reputation: 100.00%
Reputation Score: +152 / -0
Time Online: 43 days 7 hours 41 minutes
Hey

Make sure you create the sitemap AFTER doing the mod_rewrite thing, so your sitemap contains only the new stuff

Tim


Logged Offline
Site Site Private Message Private message Reply: 1 - 3
Craig
September 14, 2005, 2:36pm Report to Moderator Report to Moderator

SQL Support Team
Posts: 3,619
Gender: Male
Posts Per Day: 2.25
Reputation: 97.73%
Reputation Score: +215 / -5
Time Online: 14 days 14 hours 57 minutes
Location: Germany
Age: 38
...and of course Justin will have to realease 9.3 before any of this is useful


Don't just ask a question.
Instead ask a question, give us a screen shot and post a link to your forum.
Help us help you make your forum better!


If I helped...increase my Reputation by clicking here.
Logged Offline
Site Site Private Message Private message ICQ ICQ Reply: 2 - 3
Justin
December 4, 2005, 1:22pm Report to Moderator Report to Moderator

The E-Blah Developer
E-Blah Programmer
Posts: 15,000
Gender: Male
Posts Per Day: 6.79
Reputation: 93.38%
Reputation Score: +296 / -21
Time Online: 36 days 16 hours 47 minutes
Location: Tallassee, AL
Age: 21


I do installs for $25 and upgrades for $20.
Technical support is always free.

  Donate to E-Blah!  

My Websites: Revolution Reality (My Blog)  |  MinistryTalk.com  |  Portfolio

"But you, O Lord, are a compassionate and gracious God, slow to anger, abounding in love and faithfulness." — Psalm 86:15 NIV

Logged Offline
Site Site Private Message Private message Reply: 3 - 3
1 Pages 1 Recommend Thread
Print

E-Blah Community    Informational    Forum Updates  ›  Making Your Forum Bot Friendly