Threads and Timeout
The following numbers are here as reference, know that depending on the server provider, hardware and network those settings might change
Minimum 4 vCPU / 8 GB RAM - Maximum one task at the time
| Module | Recommendation |
|---|---|
| Scraper / Parser | 50–250 threads, 15,000 ms timeout |
| Dumper | 10-25 main and secondary threads |
| Vulnerability scan | 50–75 threads, 15,000+ ms timeout |
| Other tasks | 100–250 threads, 15,000 ms timeout |
Minimum 8 vCPU / 16 GB RAM
| Module | Recommendation |
|---|---|
| Scraper / Parser | 100–250 threads, 7,500 ms timeout |
| Dumper | 25-50 main and secondary threads |
| Vulnerability scan | 100–250 threads, 15,000+ ms timeout |
| Other tasks | 250–500 threads, 15,000 ms timeout |
Minimum 16 vCPU / 32 GB RAM
| Module | Recommendation |
|---|---|
| Scraper / Parser | 250–500 threads, 15,000 ms timeout |
| Dumper | 50-100 main and secondary threads |
| Vulnerability scan | 250–500 threads, 30,000+ ms timeout |
| Other tasks | 500+ threads, 7,500 ms timeout |
These figures assume modern hardware and at least a 500 Mbps network connection. Older machines or congested links may need lower thread counts and higher timeouts to avoid rate limiting. Increase threads only when both CPU and network utilisation remain low.