What is robots.txt | Why it is too Important?

robots.txt is a text file located at the root directory of a website. Its primary purpose is to instruct web crawlers (also known as robots or spiders) about which pages or sections of the website should not be crawled or indexed. This file acts as a guideline for search engines, helping to manage and optimize the crawling process.

A bot is an automated computer program that interacts with websites and applications. There are good bots and bad bots, and one type of good bot is called a web crawler bot. These bots “crawl” webpages and index the content so that it can show up in search engine results. A robots.txt file helps manage the activities of these web crawlers so that they don’t overtax the web server hosting the website, or index pages that aren’t meant for public view.

The robots.txt file extension refers to a text file that doesn’t contain HTML markup code. Similar to other files on the website, the robots.txt file is stored on the web server. Actually, you can usually view the robots.txt file for any given website by typing the homepage’s full URL followed by /robots.txt, such as https://templatelane.com/robots.txt. Although users are unlikely to come across the file because it isn’t linked to anywhere else on the website, the majority of web crawler bots will look for it before exploring the rest of the site.

Although it contains instructions for bots, a robots.txt file is unable to enforce its commands. Before viewing any other pages on a domain, a good bot—like a web crawler or news feed bot—will try to visit the robots.txt file and will obey its instructions. A malicious bot will either process the robots.txt file to locate the prohibited webpages or ignore it.

A web crawler bot will follow the most specific set of instructions in the robots.txt file. If there are contradictory commands in the file, the bot will follow the more granular command. The robots.txt file uses a straightforward syntax to provide instructions to web crawlers.

Here’s an example of a basic robots.txt file:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

• User-agent: Specifies which web crawler the rules apply to. An asterisk (*) means the rules apply to all web crawlers.
• Disallow: Indicates the directories or pages that should not be crawled.
• Allow: Specifies the directories or pages that can be crawled, even if they fall under a disallowed directory.

Any person or program using the internet has a “user agent,” which is like a name assigned to them. For people, this includes details like the browser they’re using and their operating system, but no personal information. This helps websites display content that works well with the user’s system. For bots, the user agent helps website administrators identify which bots are visiting the site.

In a robots.txt file, website administrators can give specific instructions to different bots by writing separate rules for each bot’s user agent. For example, if an admin wants a page to appear in Google search results but not in Bing, they can include two sets of instructions in the robots.txt file: one starting with “User-agent: Bingbot” and another with “User-agent: Googlebot”.

Common search engine bot user agent names include:

• Google:
  • Googlebot
  • Googlebot-Image (for images)
  • Googlebot-News (for news)
  • Googlebot-Video (for video)
• Bing
  • Bingbot
  • MSNBot-Media (for images and video)
• Baidu
  • Baiduspider

The “Disallow” command is a key part of the robots exclusion protocol. It tells bots not to visit the webpage or set of webpages specified after the command. These pages aren’t hidden; they just aren’t shown in search results because they aren’t useful for most Google or Bing users. However, if someone knows the link, they can still visit these pages on the website.

The “Allow” command specifies which parts of a website can be accessed by web crawlers despite any broader restrictions set by the Disallow command. It grants permission for specific URLs or directories within a section that might otherwise be restricted. This command is particularly useful for giving access to certain content within a generally restricted area, ensuring that important pages are indexed while keeping other sections private. For example, you might disallow all content in a directory but allow a specific file within that directory to be crawled.

1. Control Over Crawling: With robots.txt, you can specify which parts of your website you want to keep private or restrict from being indexed by search engines. This can be useful for hiding confidential information, staging areas, or duplicate content.
2. Optimize Crawl Budget: Search engines allocate a certain amount of resources (crawl budget) to each website. By directing crawlers to the most important pages and away from low-priority or resource-heavy sections, you can ensure that search engines efficiently use their resources on your site.
3. Prevent Duplicate Content: If your website has multiple versions of the same content, robots.txt can help prevent duplicate content issues by blocking crawlers from accessing redundant pages.
4. Enhance Site Security: By disallowing crawlers from accessing sensitive directories, you can add an extra layer of security to your website, protecting against potential data breaches.

1. Keep It Simple: Avoid overly complex rules. Simple, clear instructions are more effective and easier to manage.
2. Regularly Update: As your website evolves, update your robots.txt file to reflect any new pages or directories you want to control.
3. Test Your File: Use tools like Google’s robots.txt Tester to ensure your file is correctly formatted and functioning as intended.
4. Monitor Crawling: Keep an eye on your server logs or use tools like Google Search Console to monitor how search engines are interacting with your site.
5. Don’t Rely Solely on robots.txt for Security: While robots.txt can help protect sensitive areas, it should not be your only security measure. Use proper authentication and authorization mechanisms for sensitive data.
6. Allow Important Pages: Make sure critical pages are not accidentally disallowed. Blocking important pages can negatively impact your SEO.

1. Disallowing All Crawlers: Avoid using Disallow: / for all user-agents unless you want to block your entire site from being crawled.
2. Misplaced robots.txt File: Ensure the robots.txt file is in the root directory. Placing it elsewhere will render it ineffective.
3. Forgetting Important Directories: Regularly review your robots.txt to ensure all important directories are included or excluded as per your requirements.

Creating a robots.txt file is simple. Here are the steps:

• Open a Text Editor: Use a basic text editor like Notepad (Windows) or TextEdit (Mac) to create the file.
• Write the Rules: Define the user-agents and the directories or pages you want to disallow or allow.
• Save the File: Save the file as robots.txt.
• Upload to Root Directory: Place the robots.txt file in the root directory of your website. For example, if your website is www.example.com, the robots.txt file should be accessible at www.example.com/robots.txt.

Here’s an example of a basic robots.txt file:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

1. Using FTP:
   • Use an FTP client (e.g., FileZilla, Cyberduck) to connect to your web server.
   • Navigate to the root directory of your website (usually the public_html or www folder).
   • Upload the robots.txt file to this root directory.
2. Using cPanel:
   • Log in to your cPanel account.
   • Go to the File Manager.
   • Navigate to the root directory of your website.
   • Click on Upload and select the robots.txt file from your local computer.
3. Using a CMS (e.g., WordPress):
   • Log in to your WordPress admin dashboard.
   • Go to the plugin section and install a plugin like “RankMath“,”Yoast SEO” or “All in One SEO Pack” which allows you to manage robots.txt.
   • Use the plugin’s interface to create and upload the robots.txt file.
4. Verify the Upload:
   • Open a web browser and navigate to https://yourwebsite.com/robots.txt.
   • Ensure the file is accessible and the rules are correctly displayed.