Explore projects
-
COSSAS / bigphish
Apache License 2.0BigPhish - An all-in-one solution to identify and monitor phishing kit deployment
Updated -
Solebull / dwse
GNU General Public License v3.0 onlyDark Web Search Engine : a dark net crawler and database viewer.
Updated -
Crawl the LEGO Pick A Brick store to create a JSON representation of the available products.
UpdatedUpdated -
Laurent Arnoud / validate-website
MIT LicenseWeb crawler for checking the validity of your documents.
Updated -
Alex / srogue11
BSD 3-Clause "New" or "Revised" Licenserefactor of RogueLikeRestorationProject/srogue to make it meet C11 standard
Updated -
marzzzello / gplaycrawler
MIT LicenseDiscover apps by different mehtods. Mass download app packages and metadata.
Updated -
Daniel Snider / crawler
GNU Affero General Public License v3.0A Python app for scanning large data sets of URLs for a given signature and storing the results to an ElasticSearch index. Useful applications for CERTs and security researchers, maybe others.
Updated -
-
-
This script is created for scraping Data from MCGM web-site
Updated -
experiments / node-simplecrawler-example
Apache License 2.0An experiment that implements simplcrawler.
Updated -
LorneWu / e-Gental-downloader
GNU General Public License v3.0 onlyThis is a e-gental downloader based on python3. #crawler
Updated -
-
-
From a base URL, crawl a websites URLs (via href link traversal) and generate a Sitemap. Supports checking and cleaning up sitemap, and checking paths on a different base/host.
Updated -
Adrian B.G. / warmcache
GNU General Public License v3.0 or laterA simple tool to scan your website to keep your cache hot & ready. Helper tool for Prerender, Squid, CDN etc..
Updated -
Crawler which can crawl and download videos from YouTube and Vimeo
Updated -
explore and download movies much EASIER
Updated -