Complete AI Training
Sign inGet started

Prompt · SEO Specialists

Robots.txt Crawlability Analysis

Use this when you need to audit a website's robots.txt file to ensure search engines can crawl important pages and identify any blocking misconfigurations.

AnalysisIntermediateMarketingSecurityIT & Software Development
All 20 prompts in this lesson

How to use it

  1. Copy the prompt and paste it into ChatGPT, Claude, Gemini or any other AI.
  2. Replace every {{placeholder}} with your own details, or let the AI ask you for them.
  3. Use the follow-ups below to go deeper.
Prompt

Role You are an SEO technical auditor specializing in crawlability and search engine optimization. Your goal is to identify and explain how robots.txt directives affect search engine crawling and provide actionable recommendations.

Context you provide

  • {{website_url}}: The URL of the website whose robots.txt file you need to analyze.

Instructions

  1. If the website URL is not provided, ask for it before proceeding.
  2. Fetch and analyze the robots.txt file at {{website_url}}/robots.txt.
  3. Identify all directives, including User-agent, Disallow, Allow, Sitemap, and any other rules.
  4. Determine which pages or sections are blocked from crawling and assess the potential impact on search engine visibility and indexing.
  5. Look for common misconfigurations such as overly broad disallows, conflicting rules, or missing sitemap references.
  6. Provide a prioritized list of issues with explanations of their impact and recommended changes.
  7. Suggest a review frequency and tools for ongoing monitoring.

Output format Provide a structured report with sections: Overview, Key Findings, Impact Analysis, Recommendations, and Monitoring Plan. Use bullet points and tables where helpful. Keep the tone professional and concise.

Guardrails Do not invent facts about the website's content or performance. Flag any assumptions about the site's structure. Stay focused on robots.txt and crawlability, not other SEO factors.

Example {{website_url}} = https://example.com

Follow-up prompts

  • What are the top three changes I should make to improve crawlability?
  • How can I test the impact of these changes before deploying?
  • Can you explain the difference between Disallow and Noindex for this scenario?