Run your first crawl | Screpy
Run your first crawl
Run your first Screpy website crawl and use the resulting technical SEO data.
Your first crawl creates Screpy's technical picture of a website. It discovers pages on the project domain, follows links within the crawl scope, and turns the completed data into actionable on-page SEO findings.
Screpy starts the initial crawl when you create a project. Later, you can start a new crawl from your project when you need fresh data after a site release, content update, migration, or technical fix.
What Screpy analyzes
After a successful crawl, Screpy organizes the latest results into focused feature areas:
- Quick Wins helps you find prioritized SEO opportunities.
- Pages shows the crawled URLs and their on-page data.
- Links helps you inspect internal and external links, including source pages.
- Images surfaces image-related SEO data and the pages that reference each image.
- On-page SEO on the dashboard summarizes the health of the latest completed crawl.
These views use completed crawl data. If a crawl is still running, keep the project open and return when the analysis finishes.
Set the right crawl scope
The setup flow asks for two controls: maximum URLs and depth.
| Setting | Use it for | Practical guidance |
|---|---|---|
| Max URLs | The number of pages Screpy can include | Choose enough capacity for the part of the site you want to evaluate. Increase it for a full-site audit when your plan allows it. |
| Depth | The number of link levels Screpy follows | Use a lower depth for a focused crawl and a higher depth when important pages sit further from the homepage. |
Depth and URL capacity work together. A large website can have more discoverable pages than your initial capacity, while a low depth can intentionally keep the analysis close to the main navigation. Update these controls in Project settings before the next crawl if your audit needs change.
When to run another crawl
Run a new crawl after changes that can affect technical SEO, including:
- A redesign, migration, domain change, or new template.
- A large content release or a new site section.
- Changes to redirects, canonical tags, robots rules, internal links, or structured data.
- A fix you want to verify in Quick Wins, Pages, Links, or Images.
Use the latest completed crawl
Compare your changes against the most recent completed crawl. Starting a new crawl before a previous one finishes will not give you a second parallel analysis for the same project.
Crawl troubleshooting
| Situation | What to do |
|---|---|
| The crawl is blocked or fails quickly | Confirm the website is reachable, allow Screpy's setup-screen IP addresses in your firewall or WAF, and allow ScrepyBot in robots.txt. |
| A crawl is already running | Wait for the current analysis to complete before starting another one. |
| You recently started a manual crawl | Manual crawl starts are limited to one successful start per hour. Wait for the available window before retrying. |
| Pages, links, or images are empty | Check that the latest crawl completed successfully and that its URL capacity and depth were sufficient for the area you expected to find. |
| Crawling is unavailable | Check that website audit access is available on the project's plan and that crawling is enabled in Project settings. |
Turn crawl data into work
Do not try to fix every finding at once. Start with the issues most likely to improve crawlability, indexability, page quality, and internal linking. Then rerun a crawl to confirm the change is visible in the latest data.