In this repository, I try to share some of the little tips and tricks and amazing spiders that I used to work with on the scrapy framework.

Since their inception, websites are used to share information. Whether it is a Wikipedia article, YouTube channel, Instagram account, or a Twitter handle. They all are packed with interesting data ...

At the time I used ChatGPT to complete the above Python Scrapy crawler, I had not yet learned the basic skills of prompt engineering. After this experience, I am determined to learn the usage and ...

Python tools like Scrapy and Selenium help scrape large or interactive websites easily New AI tools like Firecrawl simplify complex scraping tasks with smart automation. Static websites are best ...