How to Fix Missing robots.txt
What it is
robots.txt is a small text file at the root of your site (yourdomain.com/robots.txt) that gives crawlers instructions, such as which areas to skip and where your sitemap is.
Why it matters
- Crawlers look for it first; a missing file is not an error but means you give no guidance.
- It is the standard place to point crawlers at your sitemap.
- Be careful: a wrong rule can hide pages from search, so keep it simple.
How to fix it
- Create a text file named robots.txt.
- A simple safe version is: User-agent: * then Allow: / then Sitemap: https://yourdomain.com/sitemap.xml.
- Upload it to your site root.
- Open yourdomain.com/robots.txt in a browser to check, then re-scan.
Platform tips
| WordPress | WordPress serves a virtual robots.txt by default; SEO plugins (Yoast, Rank Math) let you edit it. Check Settings > Reading that search engines are not discouraged. |
|---|---|
| Wix | Wix creates one automatically; edit it under Marketing & SEO > SEO Tools > Robots.txt Editor. |
| Squarespace | Squarespace generates it automatically and it cannot be fully replaced; check that the site is not set to private. |
| Shopify | Shopify generates it automatically; customise through the robots.txt.liquid template if needed. |
| Plain HTML | Save a file called robots.txt in the same folder as your homepage and upload it. |
Related guides
Disclaimer: This guide is general, informational advice only and is not legal, medical, financial, compliance or professional advice. Menu names on platforms change over time; check the platform's own help if something looks different.