Search Engine Robots Txt Files ~ Google Plus One News, Google Plus One Project, Google +1 Information
Google Plus One News, Google Plus One Project, Google +1 Information: Search Engine Robots Txt Files

Thursday, November 10, 2011

Search Engine Robots Txt Files

Google Robots Txt File

Url: https://www.google.com/robots.txt
Disallow: /search
Disallow: /groups
Disallow: /images
Disallow: /catalogs
Allow: /catalogs/about
Disallow: /catalogues
Disallow: /news
Allow: /news/directory
Disallow: /nwshp
Disallow: /setnewsprefs?
Disallow: /index.html?
Disallow: /?
Disallow: /addurl/image?
Disallow: /pagead/
Disallow: /relpage/
Disallow: /relcontent
Disallow: /imgres
Disallow: /imglanding
Disallow: /sbd
Disallow: /keyword/
Disallow: /u/
Disallow: /univ/
Disallow: /cobrand
Disallow: /custom
Disallow: /advanced_group_search
Disallow: /googlesite
Disallow: /preferences
Disallow: /setprefs
Disallow: /swr
Disallow: /url
Disallow: /default
Disallow: /m?
Disallow: /m/?
Disallow: /m/blogs?
Disallow: /m/directions?
Disallow: /m/ig
Disallow: /m/images?
Disallow: /m/imgres?
Disallow: /m/local?
Disallow: /m/movies?
Disallow: /m/news?
Disallow: /m/news/i?
Disallow: /m/place?
Disallow: /m/products?
Disallow: /m/products/
Disallow: /m/setnewsprefs?
Disallow: /m/search?
Disallow: /m/swmloptin?
Disallow: /m/trends
Disallow: /m/video?
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /sprint_xhtml
Disallow: /sprint_wml
Disallow: /pqa
Disallow: /palm
Disallow: /gwt/
Disallow: /purchases
Disallow: /hws
Disallow: /bsd?
Disallow: /linux?
Disallow: /mac?
Disallow: /microsoft?
Disallow: /unclesam?
Disallow: /answers/search?q=
Disallow: /local?
Disallow: /local_url
Disallow: /shihui?
Disallow: /shihui/
Disallow: /froogle?
Disallow: /products?
Disallow: /products/
Disallow: /froogle_
Disallow: /product_
Disallow: /products_
Disallow: /products;
Disallow: /print
Disallow: /books/
Disallow: /bkshp?*q=*
Disallow: /books?*q=*
Disallow: /books?*output=*
Disallow: /books?*pg=*
Disallow: /books?*jtp=*
Disallow: /books?*jscmd=*
Disallow: /books?*buy=*
Disallow: /books?*zoom=*
Allow: /books?*q=related:*
Allow: /books?*q=editions:*
Allow: /books?*q=subject:*
Allow: /books/about
Allow: /booksrightsholders
Allow: /books?*zoom=1*
Allow: /books?*zoom=5*
Disallow: /ebooks/
Disallow: /ebooks?*q=*
Disallow: /ebooks?*output=*
Disallow: /ebooks?*pg=*
Disallow: /ebooks?*jscmd=*
Disallow: /ebooks?*buy=*
Disallow: /ebooks?*zoom=*
Allow: /ebooks?*q=related:*
Allow: /ebooks?*q=editions:*
Allow: /ebooks?*q=subject:*
Allow: /ebooks?*zoom=1*
Allow: /ebooks?*zoom=5*
Disallow: /patents?
Allow: /patents?id=
Allow: /patents?vid=
Disallow: /scholar
Disallow: /citations?
Allow: /citations?user=
Allow: /citations?view_op=new_profile
Disallow: /complete
Disallow: /s?
Disallow: /sponsoredlinks
Disallow: /videosearch?
Disallow: /videopreview?
Disallow: /videoprograminfo?
Disallow: /maps?
Disallow: /mapstt?
Disallow: /mapslt?
Disallow: /maps/stk/
Disallow: /maps/br?
Disallow: /mapabcpoi?
Disallow: /maphp?
Disallow: /mapprint?
Disallow: /maps/api/js/StaticMapService.GetMapImage?
Disallow: /maps/api/staticmap?
Disallow: /mld?
Disallow: /staticmap?
Disallow: /places/
Allow: /places/$
Disallow: /maps/place
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /lochp?
Disallow: /center
Disallow: /ie?
Disallow: /sms/demo?
Disallow: /katrina?
Disallow: /blogsearch?
Disallow: /blogsearch/
Disallow: /blogsearch_feeds
Disallow: /advanced_blog_search
Disallow: /reader/
Allow: /reader/play
Disallow: /uds/
Disallow: /chart?
Disallow: /transit?
Disallow: /mbd?
Disallow: /extern_js/
Disallow: /calendar/feeds/
Disallow: /calendar/ical/
Disallow: /cl2/feeds/
Disallow: /cl2/ical/
Disallow: /coop/directory
Disallow: /coop/manage
Disallow: /trends?
Disallow: /trends/music?
Disallow: /trends/hottrends?
Disallow: /trends/viz?
Disallow: /notebook/search?
Disallow: /musica
Disallow: /musicad
Disallow: /musicas
Disallow: /musicl
Disallow: /musics
Disallow: /musicsearch
Disallow: /musicsp
Disallow: /musiclp
Disallow: /browsersync
Disallow: /call
Disallow: /archivesearch?
Disallow: /archivesearch/url
Disallow: /archivesearch/advanced_search
Disallow: /base/reportbadoffer
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /codesearch?
Disallow: /codesearch/feeds/search?
Disallow: /wapsearch?
Disallow: /safebrowsing
Allow: /safebrowsing/diagnostic
Allow: /safebrowsing/report_badware/
Allow: /safebrowsing/report_error/
Allow: /safebrowsing/report_phish/
Disallow: /reviews/search?
Disallow: /orkut/albums
Allow: /jsapi
Disallow: /views?
Disallow: /c/
Disallow: /cbk
Allow: /cbk?output=tile&cb_client=maps_sv
Disallow: /recharge/dashboard/car
Disallow: /recharge/dashboard/static/
Disallow: /translate_a/
Disallow: /translate_c
Disallow: /translate_f
Disallow: /translate_static/
Disallow: /translate_suggestion
Disallow: /profiles/me
Allow: /profiles
Disallow: /s2/profiles/me
Allow: /s2/profiles
Allow: /s2/photos
Allow: /s2/static
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /gcc/
Disallow: /aclk
Disallow: /cse?
Disallow: /cse/home
Disallow: /cse/panel
Disallow: /cse/manage
Disallow: /tbproxy/
Disallow: /imesync/
Disallow: /shenghuo/search?
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /ppob/?
Disallow: /ppob?
Disallow: /ig/add?
Disallow: /adwordsresellers
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /topicsearch?q=
Disallow: /xfx7/
Disallow: /squared/api
Disallow: /squared/search
Disallow: /squared/table
Disallow: /toolkit/
Allow: /toolkit/*.html
Disallow: /globalmarketfinder/
Allow: /globalmarketfinder/*.html
Disallow: /qnasearch?
Disallow: /app/updates
Disallow: /sidewiki/entry/
Disallow: /quality_form?
Disallow: /labs/popgadget/search
Disallow: /buzz/post
Disallow: /compressiontest/
Disallow: /analytics/reporting/
Disallow: /analytics/admin/
Disallow: /analytics/web/
Disallow: /analytics/feeds/
Disallow: /analytics/settings/
Disallow: /alerts/
Disallow: /ads/preferences/
Allow: /ads/preferences/html/
Allow: /ads/preferences/plugin
Disallow: /phone/compare/?
Allow: /alerts/manage
Disallow: /travel/clk
Disallow: /commercesearch/services/search/
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.google.com/hostednews/sitemap_index.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml
Sitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml
Sitemap: http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml

Yahoo Robots Txt File

Url: http://www.yahoo.com/robots.txt

User-agent: *
Disallow: /p/
Disallow: /r/
Disallow: /*?

Bing Robots Txt File

Url: http://www.bing.com/robots.txt

User-agent: *
Disallow: /bmi/
Disallow: /BVFrame.aspx
Disallow: /BVSandbox.aspx
Disallow: /cashback/admin
Disallow: /cashback/go
Disallow: /challenge
Disallow: /community/forums/tags
Disallow: /community/login.aspx?
Disallow: /fd/
Disallow: /history
Disallow: /images/search?
Disallow: /images?
Disallow: /kids/
Disallow: /maps/clicks.ashx?
Disallow: /math/
Allow: /news/search?*p1=%5bNewsVertical+Category%3d*%5d
Disallow: /news/search?
Disallow: /nutrition/
Disallow: /previewx
Disallow: /products/?q
Disallow: /products/accelerator
Disallow: /products/search?q
Disallow: /profile/
Disallow: /Proxy.ashx
Disallow: /results
Disallow: /search
Disallow: /Search
Disallow: /settings
Disallow: /shopping/*/c/*?q
Disallow: /shopping/*/search
Disallow: /shopping/accelerator
Disallow: /shopping/cashback/go
Disallow: /shopping/cashback/search?q
Disallow: /shopping/sa/
Disallow: /shopping/search?
Disallow: /shopping/twitter
Disallow: /social/search?
Disallow: /spbasic
Disallow: /spresults
Disallow: /static/
Disallow: /travel/css
Disallow: /travel/flight/flightSearch
Disallow: /travel/flight/flightSearchAction
Disallow: /travel/hotel/hotelMiniSearchRequest
Disallow: /travel/hotel/hotelSearch
Disallow: /travel/scripts
Disallow: /travel/secure
Disallow: /twitter/search?
Disallow: /videos/search?
Disallow: /videos?
Disallow: /w3c/
Disallow: /xrank/search?

Sitemap: http://sitemap.video.msn.com/sitemap/sitemap-en-us-index.xml
Sitemap: http://www.bing.com/browse/sitemap.xml
Sitemap: http://www.bing.com/celebrities/sitemap.xml
Sitemap: http://www.bing.com/finance/sitemapIndex.xml
Sitemap: http://www.bing.com/ges/2/sitemap.xml
Sitemap: http://www.bing.com/movies/search/sitemap.xml
Sitemap: http://www.bing.com/music/sitemaps.xml
Sitemap: http://www.bing.com/news/sitemap.xml
Sitemap: http://www.bing.com/onlinegames/sitemap.xml
Sitemap: http://www.bing.com/recipe/sitemap.xml
Sitemap: http://www.bing.com/shopping/static/sitemapindex.xml
Sitemap: http://www.bing.com/travel/sitemap.xml
Sitemap: http://www.bing.com/travel/sitemap/sitemapDestinations
Sitemap: http://www.bing.com/tv/sitemap.xml
Sitemap: http://www.bing.com/videogames/sitemap.xml
Sitemap: http://www.bing.com/weather/sitemap.xml

No comments:

Post a Comment

Web Analytics