Website Crawler Tool and Google Sitemap Generator


Your privacy is important to us. We will never share your information with third parties.
If you would like to prevent this free tool from crawling your website, please add the following lines to your robots.txt file:
User-agent:NinjaBot
    Disallow: /

See how to use Website Crawler Tool and Google Sitemap Generator

This free crawler (designed by Jim Boykin) lets you generate Google Sitemap, spell-check, identify your site crawl issues and errors; Crawl as deep as 1000 pages! It can take a while to crawl and analyze the whole website: Feel free to provide your email and we'll message you the results as soon as it is done!

Disclaimer: We don't store or use your information in any way

Links are one of the most essential elements of any website. Not only do they allow pages to connect with other related pages and sites, they are also essential in optimizing pages for SEO. The Google Sitemap Generator makes it easy for SEOs and webmasters to check the external and internal links on their website to find any errors and identify link rot and redirects. The results are downloadable.

To begin using the sitemap generator paste or type the home page URL of your website into the box and select how many pages you would like scanned. You can select 500, 1,000, or 10,000. The site will start immediately and run in real time. The process can take up to 30 minutes or longer depending on the number of pages being crawled. Because of the amount of time it can take to run you can opt to have an email sent upon completion of the crawl as notification of completion.  The online sitemap generator will provide a variety of options and can act as an HTML or XML sitemap generator.

The data that results from the online sitemap generator is interactive. Most items are included as links. Most non-URL data can be hovered over to view complete results.

Results

Upon starting the tool a results bar will be presented which includes the following:

  • Status of the tool (Crawling or Done)
  • Number of Internal URLs crawled
  • Number of External links found
  • Number of Internal HTTP Redirects found
  • Number of External HTTP Redirects found
  • Number of Internal HTTP error codes found
  • Number of External HTTP error codes found

If you need a sitemap in XML or HTML the following options are also provided:

  • Download XML Sitemap button
  • Download tool results in Excel format
  • Download tool results in HTML format

Results will be organized into the following six tables:

  1. Internal links
  2. External links
  3. Internal errors (a subset of Internal Links)
  4. Internal redirects (another subset of Internal Links)
  5. External errors (a subset of External Links)
  6. External redirects (another subset of External Links)

Sorting can be done by clicking on the column headers.

Tables

Internal Links - This table includes all of the following: Crawled URLs, the On Page Optimization Analysis for the URL, the level of the URL off of the root domain, status code, internal links belonging to the URL (these can be viewed by clicking), link text, number of internal links on that page (these can be viewed by clicking), number of external links on that page (these can be viewed by clicking), total page size (the page load speed test results can be viewed by clicking), title tag text, meta description tag text, meta keywords tag text, “rel=” attribute contents if used.

Internal HTTP Code Errors - This table includes all of the following: HTTP status code of pages returning HTTP code errors, number of times each URL is linked on the site (these URLs can be viewed by clicking), internal URL used in the link, anchor text used for the URL, internal page where the link was initially found. This table is a subset of the Internal Links table.

External Links - This table includes all of the following: HTTP status code of the URL, number of times the URL is linked within the website (these can be viewed by clicking), external URL the link is directing to, anchor text used for the link, page where the link is first found.

External HTTP Code Errors - This table includes all of the following: Status code of URLs, times that URL is linked to within the website (these can be viewed by clicking), internal URL used in the link, link text used, redirect’s target URL, page where the URL was first found. This table is a subset of the External Links table.

External HTTP Redirects - This table includes all of the following: Status code of URLs, times that URL is linked to within the website (these can be viewed by clicking), internal URL used in the link, link text used, redirect’s target URL, page where the URL was first found. This table is a subset of the External Links table.


Feedback

 

Users comments:

Login or register to post comments and rate

Not rated yet.

jovan aja
2014-11-23
thanks for the tool very helping me to find the wrong in my http://benoabaysand-bali.blogspot.com/