itsMe Posted October 16, 2021 Share Posted October 16, 2021 This is the hidden content, please Sign In or Sign Up crawlergo is a browser crawler that uses chrome headless mode for URL collection. It hooks key positions of the whole web page with DOM rendering stage, automatically fills and submits forms, with intelligent JS event triggering, and collects as many entries exposed by the website as possible. The built-in URL de-duplication module filters out a large number of pseudo-static URLs, still maintains a fast parsing and crawling speed for large websites, and finally gets a high-quality collection of request results. crawlergo currently supports the following features: chrome browser environment rendering Intelligent form filling, automated submission Full DOM event collection with automated triggering Smart URL de-duplication to remove most duplicate requests Intelligent analysis of web pages and collection of URLs, including javascript file content, page comments, robots.txt files and automatic Fuzz of common paths Support Host binding, automatically fix and add Referer Support browser request proxy Support pushing the results to passive web vulnerability scanners This is the hidden content, please Sign In or Sign Up Link to comment Share on other sites More sharing options...
Recommended Posts