Search the Community

Intro Basics of data extraction (4 videos) 1. Is scraping legal? 2. Data extraction methods 3. Scraping DOM with code (PHP) 4. Scraping DOM without code 5. Extract data visually, without code 1. Scraping Instagram for profit (3 videos) 1. Deep dive #1 2. Deep dive #2 3. Putting it all together 2. Scraping Shopify for profit 1. Collecting keywords on the Shopify app store 2. Fetching results, ratings and reviews from the Shopify App Store 3. Find leads with technical marketing (5 videos) 1. Scraping competitor reviews (with code) 2. Exporting leads to a spreadsheet (with code) 3. Scraping competitor reviews (without code) 4. Using the power of Google to find lists of leads 5. Finding leads through technology lookups 4. Proxying requests (3 videos) 1. Intro to proxies 2. Proxying requests (no code) 3. Setting up a data collector (no code) [Hidden Content] [hide][Hidden Content]]

pagodo (Passive Google Dork) – Automate Google Hacking Database scraping The goal of this project was to develop a passive Google dork script to collect potentially vulnerable web pages and applications on the Internet. There are 2 parts. The first is ghdb_scraper.py that retrieves Google Dorks and the second portion is pagodo.py that leverages the information gathered by ghdb_scraper.py. What are Google Dorks? The awesome folks at Offensive Security maintain the Google Hacking Database (GHDB) found here: [Hidden Content]. It is a collection of Google searches, called dorks, that can be used to find potentially vulnerable boxes or other juicy info that is picked up by Google’s search bots. Changelog v2.3.1 Added switches to optionally provide a file name for both text and JSON output. Thank you @cgeopapa for #68 [hide][Hidden Content]]

pagodo (Passive Google Dork) – Automate Google Hacking Database scraping The goal of this project was to develop a passive Google dork script to collect potentially vulnerable web pages and applications on the Internet. There are 2 parts. The first is ghdb_scraper.py that retrieves Google Dorks and the second portion is pagodo.py that leverages the information gathered by ghdb_scraper.py. What are Google Dorks? The awesome folks at Offensive Security maintain the Google Hacking Database (GHDB) found here: [Hidden Content]. It is a collection of Google searches, called dorks, that can be used to find potentially vulnerable boxes or other juicy info that is picked up by Google’s search bots. [hide][Hidden Content]]

Follow on course: Scrape images, master regular expressions, create dynamic charts and graphs. OOP and AJAX too! What you'll learn Scraping title and images, displaying them in a gallery using bootstrap cards, regular expressions, ajax, front end design, and dynamic charts/graphs. Requirements Ideally, take our basic (zero to pro) web scraping series as a prerequisite. Suggested but not required. Description Don't waste your time on courses that hide behind hundreds of hours. In just five lessons, you will be scraping titles and images with confidence, and using flask, bootstrap and chartist.js to display your scraped data in galleries and dynamically changing charts! You will also finally walk away confident with regular expressions, write OOP code and AJAX and learn a hell of a lot in an extremely short time frame. Happy learning and I do hope you'll really enjoy and benefit from this course. Who this course is for: Beginner Python developers, advanced developers looking to learn web scraping and new techniques Course content 5 sections • 5 lectures • 1h 35m total length [Hidden Content] [hide][Hidden Content]]

Powerful Telegram Members Scraping and Adding Toolkit. Feature ADDS IN BULK[by user id, not by username] Scrapes and adds to public groups Works in Windows systems You can run unlimited accounts at the same time in order to add members CSV files auto-distributer based on number of accounts to use Powerful scraping tool that can scrape active members from any public group You can add members both by username and by user ID Least chances of account ban Script auto-joins public group from all accounts for faster adding Filters banned accounts and remove them, making things easy Genisys can also store unlimited accounts for adding purposes Adding scripts launches automatically based on the number of accounts to use [hide][Hidden Content]]

Dorks Eye Google Hacking Dork Scraping and Searching Script Dorks Eye is a script I made in python 3. With this tool, you can easily find Google Dorks. Dork Eye collects potentially vulnerable web pages and applications on the Internet or other awesome info that is picked up by Google's search bots. Dorks Eye has tested it so far on: Kali Linux Parrot Security BlackArch Termux This list would be expanded [hide][Hidden Content]]

OSINT Tool to find Media Links in Tor Sites Features Download Media Scrape From Single Url Face Recognition Scraping From Files Txt Csv Excel Tested On Kali Linux 2019.2 Ubuntu 18.04 Nethunter Arc Linux [HIDE][Hidden Content]]

Twint is an advanced Twitter scraping tool written in python that allows for scraping Tweets and pictures from Twitter profiles without using Twitter’s API. Twint utilizes Twitter’s search operators to let you scrape Tweets from specific users, scrape Tweets relating to certain topics, hashtags & trends, or sort out sensitive information from Tweets like e-mail and phone numbers. I find this very useful, and you can get really creative with it too. Twint also makes special queries to Twitter allowing you to also scrape a Twitter user’s followers, Tweets a user has liked, and who they follow without any authentication, API, Selenium, or browser emulation. Benefits Some of the benefits of using Tweep vs Twitter API: Can fetch almost all Tweets (Twitter API limits to last 3200 Tweets only) Fast initial setup Can be used anonymously and without Twitter sign up No rate limitations Changelog v2.1 New features: added user_rt: name of the user at the time which retweeted extended user_rt_id to any retweet and not just specific ones added retweet_id added reply_to: list of {user_id, username} of users which you’d reply to Basic Examples and Combos. python3 twint.py -u username – Scrape all the Tweets from user‘s timeline. python3 twint.py -u username -s pineapple – Scrape all Tweets from the user‘s timeline containing pineapple. python3 twint.py -s pineapple – Collect every Tweet containing pineapple from everyone’s Tweets. python3 twint.py -u username --year 2014 – Collect Tweets that were tweeted before 2014. python3 twint.py -u username --since 2015-12-20 – Collect Tweets that were tweeted since 2015-12-20. python3 twint.py -u username -o file.txt – Scrape Tweets and save to file.txt. python3 twint.py -u username -o file.csv --csv – Scrape Tweets and save as a csv file. python3 twint.py -u username --fruit – Show Tweets with low-hanging fruit. python3 twint.py -s "Donald Trump" --verified --users – List verified users that Tweet about Donald Trump. python3 twint.py -g="48.880048,2.385939,1km" -o file.csv --csv – Scrape Tweets from a radius of 1km around a place in Paris and export them to a csv file. python3 twint.py -u username -es localhost:9200 – Output Tweets to Elasticsearch python3 twint.py -u username -o file.json --json – Scrape Tweets and save as a json file. python3 twint.py -u username --database tweets.db – Save Tweets to an SQLite database. python3 twint.py -u username --followers – Scrape a Twitter user’s followers. python3 twint.py -u username --following – Scrape who a Twitter user follows. python3 twint.py -u username --favorites – Collect all the Tweets a user has favorited [HIDE][Hidden Content]]

Dark Scrape OSINT Tool to find Media Links in Tor Sites Features Download Media Scrape From Single Url Face Recognition Scraping From Files Txt Csv Excel Tested On Kali Linux 2019.2 Ubuntu 18.04 Nethunter Arc Linux [HIDE][Hidden Content]]

Dirble is a website directory scanning tool for Windows and Linux. It’s designed to be fast to run and easy to use. Changelog v1.4 Added Add ability to do scans using HEAD and POST requests Directories which return 401 and 403 codes are no longer scanned by default Not found detection now can detect response lengths that vary by the URL length Added dockerfile to git repository Commit hashes are now displayed with the version number in local builds Changed Wordlist items now have a leading and trailing slash removed Default wordlist location checks have been improved SimpleLog crate now used to print additional scanning information Silent and verbose flags now affect logging level Output for listable directories now has a bold L Wordlist splitting of initial URLs is increased Fixed Disable recursion flag now works as intended Validator always defaulting to 404 [HIDE][Hidden Content]]

OSINT Tool For Scraping Dark Websites Tested On Kali Linux 2019.2 Ubuntu 18.04 Nethunter Arc Linux [HIDE][Hidden Content]]