To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots (usually ). When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.[47]

Mobile devices have become increasingly popular, where 5.7 billion people are using them worldwide [13]. This has played a role in the way consumers interact with media and has many further implications for TV ratings, advertising, mobile commerce, and more. Mobile media consumption such as mobile audio streaming or mobile video are on the rise – In the United States, more than 100 million users are projected to access online video content via mobile device. Mobile video revenue consists of pay-per-view downloads, advertising and subscriptions. As of 2013, worldwide mobile phone Internet user penetration was 73.4%. In 2017, figures suggest that more than 90% of Internet users will access online content through their phones.[14]


Since social media marketing first came to be, strategists and markets have been getting smarter and more careful with the way they go about collecting information and distributing advertisements. With the presence of data collecting companies, there is no longer a need to target specific audiences. This can be seen as a large ethical gray area. For many users, this is a breach of privacy, but there are no laws that prevent these companies from using the information provided on their websites. Companies like Equifax, Inc., TransUnion Corp, and LexisNexis Group thrive on collecting and sharing personal information of social media users.[107] In 2012, Facebook purchased information from 70 million households from a third party company called Datalogix. Facebook later revealed that they purchased the information in order to create a more efficient advertising service.[108]

Webmasters and content providers began optimizing websites for search engines in the mid-1990s, as the first search engines were cataloging the early Web. Initially, all webmasters only needed to submit the address of a page, or URL, to the various engines which would send a "spider" to "crawl" that page, extract links to other pages from it, and return information found on the page to be indexed.[5] The process involves a search engine spider downloading a page and storing it on the search engine's own server. A second program, known as an indexer, extracts information about the page, such as the words it contains, where they are located, and any weight for specific words, as well as all links the page contains. All of this information is then placed into a scheduler for crawling at a later date.

×