Crawler

The Crawler starts from one or more known URLs and follows links to discover more pages on those sites. Unlike the Scraper, it does not depend on a search engine index, so it can find public routes that search results miss.

What It Does

• Visits seed URLs and follows discovered links within the configured scope.
• Maps traditional static sites, server-rendered applications, and JavaScript single-page applications.
• Finds deeper pages, routes, and parameterized URLs that can be reviewed or passed to another module.
• Produces a reusable URL inventory for the target.

Coverage depends on the seed URLs, link structure, rendering, authentication, and scope rules. A crawler cannot guarantee that it will find routes that are not linked or otherwise exposed.

What You Can Use It For

• Creating an inventory of reachable pages on a website you own or are authorized to assess.
• Finding unindexed pages during a web application review.
• Mapping a JavaScript-heavy application before manual testing.
• Expanding an approved seed list before running the Vulnerability Scanner.

Typical Input and Output

Input: one or more authorized domains or seed URLs, plus crawl scope and limits.

Output: discovered pages and endpoints, including URLs not present in search-engine results.

When to Choose It

Choose the Crawler for depth within known sites. Choose the Scraper for broad discovery across search engines. A common workflow is Scraper → scope review → Crawler, with an authorization check before crawling any discovered domain.