I have been for a while wondering why when you search google for my .com site it only comes up with the link and nothing else, afterall I have enough SEO (Search Engine Optimisation) plugins runing.
It appears that my robots.txt file has been blocking google – doh! It was something I set up a year or so ago.
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:
User-agent: *
Disallow: /
The “User-agent: *” means this section applies to all robots. The “Disallow: /” tells the robot that it should not visit any pages on the site.
There are two important considerations when using /robots.txt:
robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention. The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don’t want robots to use. So don’t try to use /robots.txt to hide information.
I have now deleted those two lines and hopefully it will make a big difference to the search listings.
Visit www.robotstxt.org to learn more.
