Colly

Colly is a fast and lightweight web scraping framework for Go that helps developers build crawlers, spiders, and data extraction tools. They provide a clean API for collecting structured information from websites for data mining, processing, and archiving.

They support synchronous, asynchronous, and parallel scraping, allowing developers to choose the approach that fits their project. Colly can also manage request delays, domain concurrency, cookies, sessions, and non-Unicode responses automatically.

Colly includes built-in features for caching, distributed scraping, and robots.txt support. Its performance can exceed 1,000 requests per second on a single CPU core, making it suitable for projects that need fast and efficient web crawling.

They are designed for developers who want a business-ready and open-source scraping solution. Colly is free for commercial use, while its community-driven development keeps the framework publicly accessible and continuously evolving.

Colly is free and open source, with no paid subscription required for its core framework. Developers can get started through the documentation, explore its examples, or access the project directly through its GitHub repository.