Crawly – Custom Website Crawler
Crawly is a powerful desktop application designed to systematically scan and analyze all internal links across your website. Map your entire site structure, identify broken links, and audit your web pages with precision and control.
Core Functionality
Complete Internal Link Discovery
The tool automatically discovers and crawls all internal pages starting from your specified URL, recursively following links to map your entire website architecture. Track crawl depth and page hierarchy to understand your site structure at a granular level.
Key Features
Configurable Crawl Settings
- Custom Start URL: Enter any website domain to begin crawling from your homepage or specific entry point
- Adjustable Delay: Set request delays (in milliseconds) to control crawl speed and respect server load, preventing rate limiting or server strain
- Multi-threaded Crawling: Configure up to 10 concurrent threads for faster crawling of large websites while maintaining stability
Advanced Targeting Options
- HTML Selectors: Use comma-separated CSS selectors to target specific page elements, allowing precise content extraction from targeted sections
- Container Selectors: Optionally limit crawling scope to specific HTML containers, focusing analysis on main content areas while excluding headers, footers, or sidebars
Multi-Format Scanning
- Image Detection: Optionally scan and catalog all images across your website
- PDF Discovery: Identify and list all PDF documents linked throughout your site
- Media Files: Detect audio and video files embedded or linked on your pages
- External Link Tracking: Enable external link scanning to identify outbound connections and third-party resources
Flexible Input Methods
- Custom URL Lists: Import specific URLs from a file to crawl only targeted pages, ideal for selective audits or testing specific site sections
Use Cases
This web crawler is ideal for SEO audits, site migration planning, content inventory management, broken link detection, website architecture analysis, and quality assurance testing across digital properties.