Robots.txt is a text file on your website that search engines (including Google) read before crawling your website. The file tells search engines if there are pages they should not crawl. It is important that your website has a robots.txt file.


What is robots.txt?

Robots.txt is a text file in the root of your web server that guides Google and other search engines in crawling your website. The file specifies whether Google may or may not crawl parts of the website by allowing (allow) or block (disallow) access to specific files and folders.

Your website should have a robots.txt file, as there will almost always be content that you don't want Google to burden your web server with crawling. For example:

  • internal search pages
  • administrator pages that should not be publicly accessible
  • sections under development
  • pages with sensitive information.

Don't block Google from JavaScript-CSS and image files used on your website, as these files are necessary for Google to see the correct version of the website. Also, don't block Google from the content you want Google to index.

Please note that a block in robots.txt does not guarantee that a page will not appear in search results. Google may still index the page if it can find it by other means (for example, via a link from another website). To avoid a page being indexed, add a noindex-tag on the page. Read more on support.google.com.

Field compressed

Mark Molgaard

Partner & Senior SEO Specialist

Se forfatter

Get help with SEO

Do you need help with search engine optimisation (SEO), or are you considering whether it makes sense for your business to focus on SEO? Contact Henning Madsen for a no-obligation discussion about your SEO project.

Vi har modtaget din forespørgsel

Tak for din forespørgsel. Vi sætter pris på muligheden for at drøfte dit projekt. Du hører fra os inden for 1-2 hverdage

På gensyn

Step 1 / 3

close icon

    Step 1 / 3 - Select project type

    Get a dialogue about your project

    Send us an inquiry and have a dialogue about how we can help you with your project and your objectives.

    SEOGoogle AdsSocial MediaMarketing automationData & InsightsOther things

    Step 2 / 3 - Project information

    Describe your project as best you can

    Step 3 / 3 - Your information

    Please enter your information below

    Download our new SEO book
    Gain new knowledge about technical SEO