Robots.txt file is very important file for search engine crawlers it’s a mechanism for keeping a webpage out of google its mandatory to check robots.txt file and test robots.txt file after implementing on your website and if any urls are blocked via robots.txt file then can be crawled if url can be found from other resources on web.
Robots.txt file is the only file which tells crawlers which section of files or pages to crawl and how much crawl delay of web crawlers and also used to avoid overloading your website requests.
Perform live url inspection of the url in search console and it will tell you url is blocked by robots txt or any other source like meta tags noindex, or blocked by cdn or any other issues with url.
Once you have successfully created robots.txt file for your website and decided which section of webpages to be crawled by google then the next step is to test and check your robots.txt file directives are exactly working as you have created or not. If any things goes wrong google will not be crawling your website which leads to disasters and not getting your website in google search indexing sometimes.
Related Robots.txt Coverage:
1. How to Create Robots.txt file for SEO To allow or disallow Website For Google Crawlers
2. How to Fix URLs Blocked by Robots.txt File in Google Search Console
3. What is robots.txt File in SEO and Uses of Robots File
4. Indexed though blocked by robots.txt file issue Fix Errors
Test robots.txt file:
Google provides a handy tool for website owners and google search console is on of the best tool provided by google to test robots.txt file. Its very simple to test robots (check robots.txt file) file via search console and check whether you are blocking any google bot or not. Google has many search engine bots and google search console provides checking all google bots at a time.
Steps to Check Robots.txt File:
The robots.txt tester tool in search console whether your robots.txt file blocks google web crawler from specific section of your website or not.
Steps to Follow to Test and Check Robots.txt File Blocked by Google or Not:
- Go to Robots tester tool section, and scroll through the robots.txt code to locate the highlighted syntax warnings and logic errors. The number of syntax warnings and logic errors is shown immediately below the editor.
- Type in the URL of a page on your site in the text box at the bottom of the page.
- Select the user-agent you want to simulate in the dropdown list to the right of the text box.
- Click the TEST button to test access.
- Check to see if TEST button now reads ACCEPTED or BLOCKED to find out if the URL you entered is blocked from Google web crawlers.
- Edit the file on the page and retest as necessary. Note that changes made in the page are not saved to your site! See the next step.
- Copy your changes to your robots.txt file on your site. This tool does not make changes to the actual file on your site, it only tests against the copy hosted in the tool.
Source: https://support.google.com/webmasters/answer/6062598?hl=en