You may not want certain pages of your site crawled because they might not be useful to users if found in a search engine's search results. If you do want to prevent search engines from crawling your pages, Google Search Console has a friendly robots.txt generator to help you create this file. Note that if your site uses subdomains and you wish to have certain pages not crawled on a particular subdomain, you'll have to create a separate robots.txt file for that subdomain. For more information on robots.txt, we suggest this Webmaster Help Center guide on using robots.txt files13.
To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]
You can confer some of your site's reputation to another site when your site links to it. Sometimes users can take advantage of this by adding links to their own site in your comment sections or message boards. Or sometimes you might mention a site in a negative way and don't want to confer any of your reputation upon it. For example, imagine that you're writing a blog post on the topic of comment spamming and you want to call out a site that recently comment spammed your blog. You want to warn others of the site, so you include the link to it in your content; however, you certainly don't want to give the site some of your reputation from your link. This would be a good time to use nofollow.
23snaps Amikumu aNobii AsianAve Ask.fm Badoo Cloob Cyworld Diaspora Draugiem.lv Ello Facebook Foursquare Gab Hello Hi5 Highlight Houseparty Idka Instagram IGTV IRC-Galleria Keek LiveJournal Lifeknot LockerDome Marco Polo Mastodon MeetMe Meetup Miaopai micro.blog Minds MixBit Mixi Myspace My World Nasza-klasa.pl Nextdoor OK.ru Path Peach Periscope Pinterest Pixnet Plurk Qzone Readgeek Renren Sina Weibo Slidely Snapchat SNOW Spaces Streetlife StudiVZ Swarm Tagged Taringa! Tea Party Community TikTok Tinder Tout Tuenti TV Time Tumblr Twitter Untappd Vero VK Whisper Xanga Yo
Optimizing a website may involve editing its content, adding content, modifying HTML, and associated coding to both increase its relevance to specific keywords and to remove barriers to the indexing activities of search engines.[citation needed] Promoting a site to increase the number of backlinks, or inbound links, is another SEO tactic. By May 2015, mobile search had surpassed desktop search.[3]
×