crawler

Web Crawler

Async web crawler

Features

• asynchronous
• search Rest API

Usage

1. Create venv and install dependencies

   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt

2. Run docker-compose

   docker-compose up -d

3. Crawl task

   python main.py -e http://localhost:9200 -c 8 -p 8 docs.python.org

4. Get result

   curl http://localhost:8080/api/v1/search?q=asyncio&limit=20&offset=5