GitHub - noarche/crawler: Url crawler spider

docker compose up -d

docker exec -it crawler /bin/bash

Name		Name	Last commit message	Last commit date
Latest commit History 28 Commits
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
blacklist.py		blacklist.py
blacklist.txt		blacklist.txt
crawler.py		crawler.py
docker-compose.yml		docker-compose.yml
listgov.txt		listgov.txt
remove first or second half by delimiter.py		remove first or second half by delimiter.py
sites_found.txt		sites_found.txt
user_agents.py		user_agents.py

Provide feedback