W

web-scraping

Postgres DB + Crawlers and Scrapers for Apache HOP + Img Proxy + Fast API

NeuralNiche is a modern full-stack SaaS platform that delivers weekly validated AI and micro-SaaS ideas directly to builders who want to ship fast and build profitably. Built with Next.js 15, Flask, and PostgreSQL, it features a stunning responsive landing page with glassmorphism design, smooth Framer Motion animations, and a robust waitlist system with email/WhatsApp integration. The platform uses a 12-point validation framework to analyze engagement patterns across 50+ platforms like Reddit, Twitter, and ProductHunt, delivering only the top 3-5 ideas (validation score 70+) with detailed execution playbooks, revenue models, and source links. With comprehensive form validation, real-time error handling, SEO optimization, and professional UI/UX, NeuralNiche transforms the way indie makers and entrepreneurs discover and validate their next big idea, moving them from endless scrolling to profitable building.

A Rag application that keeps you updated about AI news, scrapped from MIT AI newsletter website

Since avherald.com does not have a RESTful API, RSS Feed, or other ways to get data without visiting the website, this Python 3 script will extract the data for you using website scraping.

Download media from Instagram.

این پروژه یک API مبتنی بر جنگو و Django REST Framework است که امکان دریافت لیست اخبار را فراهم می‌کند. در این API، اخبار شامل عنوان، متن، تگ‌ها و منبع می‌باشند. همچنین قابلیت فیلتر کردن اخبار بر اساس تگ‌ها، کلیدواژه‌های موجود و کلیدواژه‌های حذف شده فراهم شده است. این پروژه شامل طراحی مدل‌های دیتابیس و نوشتن تست‌های واحد برای اطمینان از عملکرد صحیح است.

Zoomit.ir news scraper using Scrapy | Scrapy جمع‌آوری اخبار زومیت با

This is simple Python email scraper that finds and writes into file all emails found on provided websites. You can also speify "deepness" of search, time-out and much more. Check program for full manual.

Aggregator and frontend for news.ycombinator.com

Bug fixes for the abandoned python Wikipedia project to warn the user when the Wikpedia suggestion engine is corrupting the titles of valid Wikipedia articles. Required for the examples in Natural Language Processing in Action, 2nd Edition by Maria Dyshel and Hobson Lane (and a community of more than 30 contributing authors and editors).

This repo is a mix of several data science tools. There is a mix of web-scraping of data that is then cleaned, and used to analyze the property market in malta, using prediction models, visualisations and statistical analysis.

There is also visualisations for chess data from the 1980's till 2021. Moreover, there is twitter data, which is then stored in the neo4J nosql dbms.

No data is presented in the git, only the results. Code with the data can be found at: https://drive.google.com/file/d/15EQnRtsngDsFDD_A7g4N1fwCuXI0f_Xi/view?usp=sharing

Este proyecto esta realizado para obtener el precio del dolar en venezuela, escaneando la pagina del Banco Central de Venezuela http://www.bcv.org.ve/

Alguns exemplos simples de Web Scrapping com Python, para retornar no console dados desejados e salvá-los em arquivos CSV.

Three parts for advanced scraping. Part 1, being able to reach search pages, when only some results are returned and you need to cycle to get all of them. Part2, using Django to handle database reads and saves via APi's, Part 3. Scraping pages with a lot of data, validating via Pydantic and saving via the Django APi

NFinance is a web scraper for finance news from Yahoo Finance and WSJ, useful for any trader or investor.

Extracts Slow Food Switzerland locations from the official website, and places them into KML format.

Image scraper for Norwegian Institute for Nature Research (NINA)s wildlife camera database at https://viltkamera.nina.no/

Their image license prohibits distribution of the images to third parties, as well as commercial use, and publishing. I believe downloading them yourself for research use should be okay, but I'm not a lawyer.

This tool was designed to extract member institution locations from the official North American Reciprocal Association (NARM) website and places them into KML format. There are a few Google maps versions of this data created by individuals out there (like this one) that are probably mostly accurate, but the reciprocal agreements evidently update at least annually, which means they have a higher potential for inaccuracy every year they are not updated. Hopefully that dynamism within the NARM locations makes this script a particularly useful one, as (in theory) until NARM makes a major update to their website source code, this scraper will always produce a kml file with the most updated map at any time of any year.