Regulating Crawlers: sitemap.xml and robots.txt


Sitemap example: studiofineartz.com - SEO

Exhibit 18.15   Sitemap for studiofineartz.com.

Sitemaps maintain a complete list of the website’s pages for crawlers to effortlessly crawl the entire site. They are formatted as XLM files (Exhibit 18.15) that crawlers are designed to read and follow.

If no sitemap is provided, crawlers must comb pages to seek and follow internal links, moving page to page until the entire site is crawled. As this is a more tedious process, search algorithms favourably rank sites that maintain sitemaps.

According to Google, sitemaps are particularly helpful if:

  • Your site has dynamic content. In other words pages are dynamically created by passing variables to the server. (Examples: view.php?id=123, https://www.studiofineartz.com/artist.php?name=Sangeeta%20Charan).
  • Your site has pages that are not easily discovered by robots during the crawl process — for example, pages featuring rich AJAX or Flash.
  • Your site is new and relatively isolated. (Spiders like Googlebot crawl the web by following links from one page to another, so if your site is not well linked, it may be hard for browsers to discover it.)
  • Your site has a large archive of content pages that are not well linked to each other, or are not linked at all.


XML-Sitemaps.com — freeware for generating sitemaps

Exhibit 18.16   XML-Sitemaps.com — freeware for generating sitemaps.

It is easy to generate sitemaps with freeware like XML-Sitemaps shown in Exhibit 18.16.


Submission of sitemap via Google’s Search Console - SEO

Exhibit 18.17   Submission of sitemap via Google’s Search Console.

Sitemaps are be submitted to Google via Google’s Search Console (see Exhibit 18.17).

In addition to sitemaps, crawlers look for the robots.txt file on websites. This file keeps a list of restricted pages and directories that are not to be crawled or indexed.

Previous     Next

Note: To find content on MarketingMind type the acronym ‘MM’ followed by your query into the search bar. For example, if you enter ‘mm consumer analytics’ into Chrome’s search bar, relevant pages from MarketingMind will appear in Google’s result pages.







Digital Marketing Workshop

Digital Marketing Workshop

Two-day hands-on coaching on Digital Marketing and Advertising, to train participants in developing and executing effective digital marketing strategies.