Now, the actual content of the page is the driving factor which the keyword is just that: a keyword. For a list of known web spiders, see http://www.user-agents.org/. If you have content that is no longer useful or is just so terrible that it's not worth the …

more ...