Distributed Web Crawler

This project is a scalable, distributed web crawler built with Java that supports multiple crawler processes and horizontal scaling. It utilizes technologies such as Kafka for URL queue management, Cassandra for metadata storage, S3/MinIO for content storage, and Spring Boot for its core engine and framework. The crawler features concurrent craw...