Search the Community

crawlergo is a browser crawler that uses chrome headless mode for URL collection. It hooks key positions of the whole web page with DOM rendering stage, automatically fills and submits forms, with intelligent JS event triggering, and collects as many entries exposed by the website as possible. The built-in URL de-duplication module filters out a large number of pseudo-static URLs, still maintains a fast parsing and crawling speed for large websites, and finally gets a high-quality collection of request results. crawlergo currently supports the following features: chrome browser environment rendering Intelligent form filling, automated submission Full DOM event collection with automated triggering Smart URL de-duplication to remove most duplicate requests Intelligent analysis of web pages and collection of URLs, including javascript file content, page comments, robots.txt files and automatic Fuzz of common paths Support Host binding, automatically fix and add Referer Support browser request proxy Support pushing the results to passive web vulnerability scanners [hide][Hidden Content]]