Logo

Robots.txt & XML Sitemap Builder

Generate the essential files search engines need to crawl and index your website effectively.

robots.txt
User-agent: *
Disallow: /api/
Disallow: /admin/
Disallow: /private/
Allow: /

Sitemap: https://www.example.com/sitemap.xml
sitemap.xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2026-08-15</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://www.example.com/about</loc>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  <url>
    <loc>https://www.example.com/blog</loc>
    <changefreq>weekly</changefreq>
    <priority>0.9</priority>
  </url>
</urlset>

The Role of Robots.txt and Sitemap.xml

When a search engine like Google or Bing discovers your website, its "spider" or "crawler" bot will look for two specific files in your root directory to understand how to index your content: robots.txt and sitemap.xml.

How Robots.txt Works

The robots.txt file acts as the bouncer for your website. It uses the Robots Exclusion Protocol to tell web crawlers which URLs they are allowed to access.

  • User-agent: * means the rules apply to all web crawlers.
  • Disallow: /private/ prevents bots from indexing sensitive or duplicate paths (like an admin dashboard or user account page).

How XML Sitemaps Work

While robots.txt tells crawlers where not to go, the XML Sitemap tells them exactly where they should go. It acts as a roadmap of all your important pages.

The sitemap schema allows you to specify metadata about each URL:

  • lastmod: When the page was last updated.
  • changefreq: How often the content changes (e.g., daily for a news homepage, monthly for an about page).
  • priority: A relative scale from 0.0 to 1.0 indicating how important the page is compared to other pages on your site (typically 1.0 for the homepage).