Getting Started with Screaming Frog SEO Spider
Screaming Frog is a UK-based search agency but their name has become synonymous with a website crawler they created called SEO Spider. It is extremely powerful and an asset for any technical SEO audits you are performing, especially considering that it is completely free if you only need to scan 500 URLs or less. Some of the most notable use cases include:
Analyze page titles, meta data, alt tags, etc.
Understand status codes and redirects
Uncover issues like broken links, duplicate content or missing data
Downloading Screaming Frog
The crawler is not web-based and will require you to download the application to your Mac or PC. To do so, click on the Download link shown below in the image and it will open a model that detects your device and provides you with a giant download button.
Once you have done that, run through the typical steps to install a new application on your computers. After it has finished installing, launch the application.
Optional - Purchasing & Entering License
With the Screaming Frog application now open, you can click on “License” in the top navigation and select “Buy License” that will take you to the website to purchase a one-year license for £149 (roughly $200USD). Once you have the license information you can go back to the top nav > License > Enter License and you will get this modal to enter the details you received from your purchase.
Run Your First Crawl
Note: This step is the same regardless of whether or not you bought a license.
This part is about as easy as entering a website in a web browser. There are a lot of advanced settings that can be done prior to starting the crawl but we aren’t going to cover those at the moment. Simply enter the full URL that you want to start the crawl where it says “Enter URL to spider” and click the Start button.
Reviewing the Results
Once your crawl is complete, you should see results that look similar to my example below with Amazon’s site. However, if you jumped straight in to a full crawl of a very large site with the paid version, it could still be running or have stalled out, requiring some of the customization mentioned earlier.
There is so much that this tool can do that it definitely cannot be covered in one post. As you are exploring the results of your first crawl, I recommend you simply start by clicking through all of the results in all of the tabs (so many tabs!) to get a feel for the kind of information now available to you.