Link: https://support.brilliantdirectories.com/support/solutions/articles/12000099791
A robots.txt file is a text file placed on a website's server that instructs web crawlers and other automated processes which pages or sections of the site should or should not be accessed. It's a protocol used to communicate with web robots and is commonly used for managing website indexing and crawling by search engines like Google, Bing, SEO tools and others.
Adding a Web Robot to the site
Having a web robot crawl a site involves granting permission for it to access and index the site's content.
To allow third-party tools to crawl a website, they must be included in the Allowed list within the website's Robots.txt file.
The ROBOTS.TXT file can be accessed through:
- Navigate to the Developer Hub sidebar menu link.
- Click on the Robots File option.
To incorporate https://ahrefs.com/ into the list, please add the following lines to the beginning of the code:
Lastly, click on "Save Changes". Now, Ahrefs has permission to crawl the website without encountering any blockages.
For additional assistance:
Email support@brilliantdirectories.com or create a ticket Inside Your Account Dashboard.
Was this article helpful and accurate? Yes No
Can't find what you're looking for?