Your robots.txt file is a tiny text file that lives at the root of your website and tells search engines where they are allowed to go. It is one of the first things Google looks for when it visits your site, and getting it right (or wrong) can quietly shape how much of your business shows up in search.
The good news is that robots.txt is simple once you understand what it actually does. In this guide we will walk through what it is, how to set one up, and the mistakes that can accidentally hide your whole site from Google.
What is a robots.txt file?
Robots.txt is a plain text file that gives instructions to web crawlers, the automated programs (often called bots or spiders) that search engines send out to read the pages on your site. When Googlebot arrives at your domain, it checks for a file at yourdomain.com/robots.txt before it crawls anything else.
Think of it as a polite sign at the entrance to your property. It says, "You are welcome to look around here, but please skip these areas." Crawlers from the major search engines respect these instructions, which makes robots.txt a useful part of your technical SEO setup.
One important thing to understand up front: robots.txt controls crawling, not indexing. It asks bots not to visit certain pages. It does not guarantee those pages stay out of Google's results. We will come back to that distinction, because it trips people up.
What robots.txt is actually for
For most small businesses, robots.txt is not something you need to fuss over constantly. Its main jobs are:
- Keeping crawlers out of pages that do not belong in search. Think admin logins, internal search results, or a thank-you page after someone fills out a form.
- Helping crawlers spend their time wisely. Large sites have a limited "crawl budget," and robots.txt can steer bots away from low-value URLs so they focus on the pages that matter.
- Pointing search engines to your sitemap. You can list the location of your XML sitemap right inside robots.txt, which helps Google discover your important pages faster.
What robots.txt is not for is hiding sensitive information. Because the file is public (anyone can read yours), listing a private folder there actually advertises its location. For anything truly private, use password protection instead.
How to read a robots.txt file
A robots.txt file is made of small blocks of rules. Each block names a crawler and then lists what it can and cannot access. Here are the pieces you will see most often.
| Directive | What it does |
|---|---|
User-agent | Names the crawler the rule applies to. An asterisk (*) means all crawlers. |
Disallow | Tells the crawler not to access a path or folder. |
Allow | Makes an exception, letting crawlers into a specific page inside a disallowed folder. |
Sitemap | Lists the full URL of your XML sitemap. |
A simple, healthy robots.txt for a small business site often looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap.xmlRead out loud, that says: "All crawlers, please stay out of the admin folder, except for this one file you need, and here is my sitemap." That is genuinely all most sites require.
A quick word on wildcards
You can use a * to match any sequence of characters and a $ to mark the end of a URL. For example, Disallow: /*.pdf$ would ask crawlers to skip every PDF file. These are handy, but they are also where accidents happen, so use them carefully and test your changes.
How to create and add a robots.txt file
Setting one up is straightforward, and the exact steps depend on how your site is built.
- Check if you already have one. Type your domain followed by
/robots.txtinto a browser. Many platforms and WordPress setups create a basic version automatically. - Create or edit the file. On WordPress, an SEO plugin like Yoast or Rank Math gives you an editor for it. On other website platforms, look under SEO or advanced settings. If you manage your own hosting, you can upload a plain text file named exactly
robots.txtto your root directory. - Keep it simple. Start with the basic example above. Only add rules you actually understand and need.
- Add your sitemap line. This is an easy win that helps Google find and crawl your pages.
- Test it. Use the robots.txt report inside Google Search Console to confirm the file is reachable and that your important pages are not blocked.
Tip: Whenever you change your robots.txt, view a few of your key pages in Search Console's URL Inspection tool afterward. It tells you plainly whether Google can crawl each page, which catches mistakes before they cost you traffic.
Common robots.txt mistakes (and how to avoid them)
Because one line can affect your entire site, robots.txt is worth double-checking. These are the errors we see most.
Accidentally blocking your whole site
The single most damaging line is Disallow: / under User-agent: *. That tells every crawler to stay off every page. Website builders sometimes add it while a site is under construction and forget to remove it at launch. If your pages suddenly vanish from Google, this is the first thing to check.
Using robots.txt to hide a page from search
As mentioned earlier, blocking a page in robots.txt does not reliably remove it from Google. If other sites link to that page, Google may still list it (usually without a description). To keep a page out of results, let crawlers reach it and add a noindex tag instead. If you need a page gone quickly and permanently, look into redirects or removal tools rather than robots.txt.
Blocking your CSS, JavaScript, or images
Some older advice suggested blocking script and style folders. Do not do this. Google needs those files to see how your page looks and works, which affects mobile-friendliness and page speed. Blocking them can hurt your rankings.
Forgetting robots.txt is not a security tool
Never rely on it to protect confidential data. It is a set of requests that well-behaved crawlers follow, not a lock. Sensitive areas belong behind a password.
Where robots.txt fits in the bigger picture
Robots.txt is one small piece of getting your site found. It works best alongside a clear site structure, a submitted sitemap, and solid on-page SEO. For most small business owners, the right approach is to set up a simple, correct file once, point it at your sitemap, and then leave it alone unless something changes.
If you are not sure whether yours is helping or hurting, checking it takes five minutes and can save you from an invisible traffic leak.
Frequently asked questions
Do I really need a robots.txt file?
You are not strictly required to have one. If you do not, crawlers simply assume they can access everything. That said, having a clean robots.txt that points to your sitemap is a low-effort best practice, and most modern platforms create a basic version for you anyway.
Will robots.txt keep a page out of Google?
Not reliably. It stops crawlers from reading the page, but the URL can still appear in results if other sites link to it. To truly keep a page out of search, allow crawling and use a noindex meta tag, or protect the page with a password.
Where does the robots.txt file need to go?
It must sit in the root of your domain, reachable at yourdomain.com/robots.txt. Search engines only look for it there. A robots.txt placed in a subfolder will be ignored, so make sure it lives at the top level of your site.