Regulating Crawlers: sitemap.xml and robots.txt

Sitemap example: - SEO

Exhibit 18.15   Sitemap for

Sitemaps maintain a complete list of the website’s pages for crawlers to effortlessly crawl the entire site. They are formatted as XLM files (Exhibit 18.15) that crawlers are designed to read and follow.

If no sitemap is provided, crawlers must comb pages to seek and follow internal links, moving page to page until the entire site is crawled. As this is a more tedious process, search algorithms favourably rank sites that maintain sitemaps.

According to Google, sitemaps are particularly helpful if:

  • Your site has dynamic content. In other words pages are dynamically created by passing variables to the server. (Examples: view.php?id=123,
  • Your site has pages that are not easily discovered by robots during the crawl process — for example, pages featuring rich AJAX or Flash.
  • Your site is new and relatively isolated. (Spiders like Googlebot crawl the web by following links from one page to another, so if your site is not well linked, it may be hard for browsers to discover it.)
  • Your site has a large archive of content pages that are not well linked to each other, or are not linked at all. — freeware for generating sitemaps

Exhibit 18.16 — freeware for generating sitemaps.

It is easy to generate sitemaps with freeware like XML-Sitemaps shown in Exhibit 18.16.

Submission of sitemap via Google’s Search Console - SEO

Exhibit 18.17   Submission of sitemap via Google’s Search Console.

Sitemaps are be submitted to Google via Google’s Search Console (see Exhibit 18.17).

In addition to sitemaps, crawlers look for the robots.txt file on websites. This file keeps a list of restricted pages and directories that are not to be crawled or indexed.

Previous     Next

Note: To find content on MarketingMind type the acronym ‘MM’ followed by your query into the search bar. For example, if you enter ‘mm consumer analytics’ into Chrome’s search bar, relevant pages from MarketingMind will appear in Google’s result pages.

Marketing Analytics Workshop

Marketing Analytics Workshop

In an analytics-driven business environment, this analytics-centred consumer marketing workshop is tailored to the needs of consumer analysts, marketing researchers, brand managers, category managers and seasoned marketing and retailing professionals.

Digital Marketing Workshop

Digital Marketing Workshop

Two-day hands-on training on Digital Marketing, conducted at the NUS Business School. Designed to make you more effective in developing and executing digital marketing strategies. You learn to use Google Search Console, Google Analytics and Google Ads to execute online marketing initiatives, run search and display advertising campaigns, and track and optimize performance.

What they SHOULD TEACH at Business Schools

What they SHOULD TEACH at Business Schools

Is marketing education fluffy too?

Experiential Learning via Simulators | Best Way to Train Marketers

Experiential Learning via Simulators | Best Way to Train Marketers

Marketing simulators impart much needed combat experiences, equipping practitioners with the skills to succeed in the consumer market battleground. They combine theory with practice, linking the classroom with the consumer marketplace.