Skip to content

Web Crawl Errors

Error Message Resolution
Invalid URL syntax
  • Verify the scheme of the URL. Ensure that the URL starts with a valid scheme like "http://" or "https://”.
  • Check for typographical errors in the URL.
  • Ensure that the URL points to an accessible destination.
  • Clear your browser cache
  • Disable any browser extensions that might be interfering with the request.
URL unreachable
  • Verify the URL for any typographical errors.
  • Ensure that the URL is complete and absolute.
  • Check if the URL is accessible.
  • Check for any network connectivity issues.
  • Ensure that the IP address of the SearchAI application is whitelisted.
No links found to crawl after adhering to Robots.txt directives. The Robots.txt file in a website instructs the crawler about the pages that can be accessed. This error is shown when the crawler is not able to find any page that adheres to both the crawl configuration and the robots.txt directives. You can uncheck Respect Robots.txt directives in which case the crawler will ignore the contents in the file and crawl all the URLs that are reachable. Alternatively, contact the website administrator.
Web crawling failed due to timeout.
  • Check for network connectivity issues
  • Ensure your requests are not overwhelming the server.
No suitable XMLs were found. Verify that the URLs listed in the XML sitemap point to valid pages on your website and that there are no typographical errors in the URLs.
Improper column names in the CSV file. Please check the column names of CSV uploaded.
Authorization or Authentication Issues
  • Ensure the URL is correct.
  • Ensure that the browser's cache/cookies are cleared.
  • Verify the authentication credentials and ensure they have the necessary permissions to access the requested resource.
  • Verify if your IP address is whitelisted by the website. Contact the website's support team or your administrator for further assistance.
Server Unavailable If you encounter this error, the server is likely overwhelmed with requests or undergoing maintenance.
  • Wait for some time and try accessing the website again later.
  • Check the website or service for any scheduled maintenance.

    Contact the website's support team or your administrator for further assistance.

Too Many Redirects Some websites require specific headers or cookies to proceed past certain redirects.
  • Ensure that your requests include necessary headers and cookies that mimic a real browser's behavior.
  • Enable JS rendered crawling option.
Request Handling Error Contact SearchAI team for this.