How to Recrawl a Website in Ahrefs Console

Estimated reading time: 2 minutes

Running a manual re-crawl in Ahrefs can help ensure that your data is up-to-date, especially after making significant changes to your website. Here’s how you can do it:

1. Log in to Ahrefs

• Go to the Ahrefs website and log in with your account credentials.

2. Access Your Project

• Once logged in, navigate to the Site Audit section.
• Select the project (website) you want to re-crawl from the list of your projects.

3. Run a Re-crawl

• Within your selected project, you should see an overview of your last crawl results.
• Look for a button or option labelled “Re-crawl” or “Start new crawl”. The exact label might vary slightly based on the interface updates.

Detailed Steps:

1. Open Site Audit:
   • From the Ahrefs dashboard, click on Site Audit in the top navigation menu.
2. Select Your Project:
   • Choose the specific project (your website) that you want to re-crawl.
3. Start a New Crawl:
   • In your project dashboard, you will see an option to “Start new crawl”. Click on this button.
   • You may also find an option to “Re-crawl pages” or “Re-crawl entire site” depending on what you need.
4. Configure Crawl Settings (Optional):
   • Before starting the crawl, you can adjust the settings such as crawl speed, user-agent, and other options if necessary.
   • Review and confirm your settings.
5. Start Crawl:
   • Click the “Start Crawl” button to initiate the manual re-crawl.
   • Ahrefs will then begin re-crawling your website based on the configured settings.

4. Monitor the Crawl Progress

• After initiating the crawl, you can monitor its progress in the Site Audit dashboard.
• Ahrefs will provide updates on the status and once complete, you’ll be able to see the updated data.

Tips for Effective Re-crawl:

• Ensure Correct Settings: Verify that the crawl settings (like crawl speed and depth) are configured properly to avoid overwhelming your server.
• Check Robots.txt and Sitemaps: Ensure that your robots.txt file and sitemaps are correctly set up to guide the crawler effectively.
• Exclude Unnecessary Pages: Use the settings to exclude certain pages or sections that you don’t need in your audit to save crawl budget and time.

Conclusion

By following these steps, you can run a manual re-crawl in Ahrefs and ensure your website data is current and accurate. If you encounter any issues, Ahrefs’ support and detailed help documentation can provide additional assistance.