Project
Distributed web crawler
A horizontally scalable web crawler in Java: Kafka distributes URLs across crawler processes, Cassandra keeps the metadata, and S3 or MinIO stores the content.
A crawler designed to scale out: any number of crawler processes share one URL queue, and storage for metadata and content is separate and scalable.
What it does
-
Distributes URLs across crawler instances through Kafka.
-
Respects robots.txt, applies politeness delays, and crawls concurrently with configurable thread pools.
-
Filters by domain and URL pattern, with depth limits.
-
Deduplicates content by SHA-256 hash.
-
Stores page metadata and crawl state in Cassandra and raw content in S3 or MinIO, organised by date and hash.
How it is built
Java and Spring Boot. Each worker takes a URL from Kafka, checks it against depth limits, filters, and robots.txt, fetches it with JSoup, hashes the content, stores it, extracts and normalises the links, filters them, and enqueues the survivors back onto Kafka — so discovery and crawling are one continuous loop shared by every worker.