Welcome, Guest

Want to take part in these discussions? Sign in if you have an account, or apply for one below

• Forgot your password?
• Apply for membership

1. • CommentAuthorBravos
   • CommentTimeMay 1st 2007

    permalinkReport Post
   I know this sounds like a contradiction but I want to keep the search engines and the public out of a blog. I want to use it for managing my own business.
   I believe there is a robots.txt file I can use for this in the blog folder?
   
   Can I also password protect a blog?
   
   Regards
   Bruce
2. • CommentAuthorDenis
   • CommentTimeMay 2nd 2007

    permalinkBuy a DrinkReport Post

   See Options / Privacy.

   To password protect the entire blog, there are a variety of solutions. Among the simplest:

   • one of the password protection plugins from wp-plugins.net
   • amember

   D.

3. • CommentAuthorBravos
   • CommentTimeMay 2nd 2007 edited

    permalinkReport Post
   Denis
   Does password protection definitely keep the search engines out?
   
   Regards
   B
   
   Sorry - I really should read all the Options available!
4. • CommentAuthorDenis
   • CommentTimeMay 2nd 2007

    permalinkBuy a DrinkReport Post

   Yes. It generates an artificial robots.txt file that only search engines can access, which disallows the indexing of the entire site.

   D.

5. • CommentAuthorClaire
   • CommentTimeMay 2nd 2007

    permalinkReport Post

   Don't know if this helps, but I have a private blog myself. I password protected it by going into my web hosting's cPanel, and then password protected the directory in which I had installed the blog.

   Claire

6. • CommentAuthorDenis
   • CommentTimeMay 2nd 2007

    permalinkBuy a DrinkReport Post

   That works too indeed, if your control panel offers the option. :-)

   D.

7. • CommentAuthorBravos
   • CommentTimeMay 5th 2007

    permalinkReport Post
   Many thanks to both of you.
   
   Bruce
8. •  
     CommentAuthorJeff
   • CommentTimeMay 26th 2008

    permalinkBuy a DrinkReport Post

   The privacy option in the WP admin simply turns privacy all on or all off. I want to limit access to places like /wp-content by humans and robots, not the entire blog. How do I limit access to specific places using robits.txt files and .htaccess?

   Is there a way?

9. •  
     CommentAuthorDarrell Ridley
   • CommentTimeMay 26th 2008

    permalinkBuy a DrinkReport Post
   To limit spiders through you site's directories with your robot.txt, I use the following, just change the directories you want....
   
   User-agent: *
   # disallow all files in these directories
   Disallow: /cgi-bin/
   Disallow: /z/j/
   Disallow: /z/c/
   Disallow: /stats/
   Disallow: /dh_
   Disallow: /wp-admin/
   Disallow: /wp-includes/
   Disallow: /contact/
   Disallow: /tag/
   Disallow: /wp-content/b
   Disallow: /wp-content/p
   Disallow: /wp-content/themes/askapache/4
   Disallow: /wp-content/themes/askapache/c
   Disallow: /wp-content/themes/askapache/d
   Disallow: /wp-content/themes/askapache/f
   Disallow: /wp-content/themes/askapache/h
   Disallow: /wp-content/themes/askapache/in
   Disallow: /wp-content/themes/askapache/p
   Disallow: /wp-content/themes/askapache/s
   Disallow: /trackback/
   Disallow: /*?*
   Disallow: */trackback/
   User-agent: Googlebot
   # disallow all files ending with these extensions
   Disallow: /*.php$
   Disallow: /*.js$
   Disallow: /*.inc$
   Disallow: /*.css$
   Disallow: /*.gz$
   Disallow: /*.cgi$
   Disallow: /*.wmv$
   Disallow: /*.png$
   Disallow: /*.gif$
   Disallow: /*.jpg$
   Disallow: /*.cgi$
   Disallow: /*.xhtml$
   Disallow: /*.php*
   Disallow: */trackback*
   Disallow: /*?*
   Disallow: /z/
   Disallow: /wp-*
   Allow: /wp-content/uploads/
   # allow google image bot to search all images
   User-agent: Googlebot-Image
   Allow: /*
   # allow adsense bot on entire site
   User-agent: Mediapartners-Google*
   Disallow: /*?*
   Allow: /z/
   Allow: /about/
   Allow: /contact/
   Allow: /wp-content/
   Allow: /tag/
   Allow: /manual/*
   Allow: /docs/*
   Allow: /*.php$
   Allow: /*.js$
   Allow: /*.inc$
   Allow: /*.css$
   Allow: /*.gz$
   Allow: /*.cgi$
   Allow: /*.wmv$
   Allow: /*.cgi$
   Allow: /*.xhtml$
   Allow: /*.php*
   Allow: /*.gif$
   Allow: /*.jpg$
   Allow: /*.png$
   # disallow archiving site
   User-agent: ia_archiver
   Disallow: /
   # disable duggmirror
   User-agent: duggmirror
   Disallow: /
   
   And with the .htaccess you can restrict access with rewrite conditions. Checkout this site to see how to implement them....
   
   http://www.udel.edu/topics/internet/WWW/restricting.html
   
   I hope this helps.
   
   Darrell
10. •  
      CommentAuthorJeff
    • CommentTimeMay 26th 2008

     permalinkBuy a DrinkReport Post

    Thanks a lot Darrell!

    You are a huge time saver! I am not sure what some of that is, but I put most of it in my robots.txt file.

    Thanks!

    Jeff