diff options
| author | luxagraf <sng@luxagraf.net> | 2019-05-04 15:48:55 -0500 |
|---|---|---|
| committer | luxagraf <sng@luxagraf.net> | 2019-05-04 15:48:55 -0500 |
| commit | 79fafe2f44f5e31522dd93013950474342bfdfb0 (patch) | |
| tree | bc9ccf5b4eadeebf3a2f86b21f9b382edfa41735 /published/Webmonkey/Monkey_Bites/2007/02.19.07/Thu/robotstxt.txt | |
| parent | 62167091560c908db0613bcb35ff9ae8292f5961 (diff) | |
archived all the stuff from freelancing for wired
Diffstat (limited to 'published/Webmonkey/Monkey_Bites/2007/02.19.07/Thu/robotstxt.txt')
| -rw-r--r-- | published/Webmonkey/Monkey_Bites/2007/02.19.07/Thu/robotstxt.txt | 12 |
1 files changed, 0 insertions, 12 deletions
diff --git a/published/Webmonkey/Monkey_Bites/2007/02.19.07/Thu/robotstxt.txt b/published/Webmonkey/Monkey_Bites/2007/02.19.07/Thu/robotstxt.txt deleted file mode 100644 index fc88079..0000000 --- a/published/Webmonkey/Monkey_Bites/2007/02.19.07/Thu/robotstxt.txt +++ /dev/null @@ -1,12 +0,0 @@ -The Google blog has a nice ongoing set of tutorials on how to use the Robots Exclusion Protocol rules to control how and what search engines index on your site. The first part was [published last month][1] and this afternoon they [posted a sequel][2]. - -Most of the information in the little tutorials applies to all search engines that follow robots.txt, though a couple of things are specific to Google. - -And even if you think you know everything about robots.txt already there still might be a few surprises for you in these tutorials. For instance I never knew that it was possible to stop Google from displaying the little summary text snippets below the results links. I still can't think of a situation where that would be helpful, but it's good to know should the need arise. - -Today's post promises at least one more short tutorial detailing common exclusion problems that and how to solve them so stay tuned. Also worth checking out is Google's overall [guide to the Robots Exclusion Protocol][3] as well as the more search engine neutral [guidelines at robotstxt.org][4]. - -[1]: http://googleblog.blogspot.com/2007/01/controlling-how-search-engines-access.html "Controlling how search engines access and index your website" -[2]: http://googleblog.blogspot.com/2007/02/robots-exclusion-protocol.html "The Robots Exclusion Protocol" -[3]: http://www.google.com/support/webmasters/bin/topic.py?topic=8843 "How Google crawls my site" -[4]: http://www.robotstxt.org/wc/exclusion.html "Robots Exclusion"
\ No newline at end of file |