Web Research AI Agent

A Python-based AI agent that searches the web for information on given topics, fetches relevant content, and generates intelligent summaries using a local Ollama model.

Features

• 🔍 Web Search: Search for information using DuckDuckGo
• 📄 Content Fetching: Asynchronously fetch and extract content from multiple URLs
• 🤖 AI Summarization: Generate intelligent summaries using local Ollama models
• 💬 Interactive Chat: Ask questions about research topics
• ⚡ Async/Await: Non-blocking concurrent operations for faster research
• 📊 Comprehensive Results: Combine search results with fetched content and AI insights

Project Structure

Basic-AI-Agent/
├── src/
│   ├── ai_agent.py           # Main AI agent class
│   ├── web_search.py          # Web searching and content fetching
│   └── example_usage.py       # Example usage demonstrations
├── requirements.txt           # Python dependencies
├── .env.example              # Environment variable template
└── README.md                 # This file

Installation

1. Clone/Setup the project:

   cd Basic-AI-Agent

2. Create a virtual environment (recommended):

   python -m venv venv
   # On Windows:
   venv\Scripts\activate
   # On macOS/Linux:
   source venv/bin/activate

3. Install dependencies:

   pip install -r requirements.txt

4. Install Ollama locally:

   • Follow the instructions at https://ollama.com/docs/installation
   • The agent will automatically pull the configured Ollama model when it first runs if it is not already installed locally.

Usage

Basic Web Search

from src.ai_agent import WebResearchAgent

agent = WebResearchAgent()

# Search for a topic
results = agent.search_topic("machine learning trends", num_results=5)
for result in results:
    print(f"Title: {result['title']}")
    print(f"URL: {result['link']}")
    print(f"Summary: {result['body']}\n")

Full Research with AI Summary

import asyncio
from src.ai_agent import WebResearchAgent

async def research():
    agent = WebResearchAgent()
    
    # Perform complete research
    results = await agent.full_research("artificial intelligence", num_results=5)
    
    print("Topic:", results["topic"])
    print("\nAI Summary:")
    print(results["summary"])

asyncio.run(research())

Interactive Chat About Topics

from src.ai_agent import WebResearchAgent

agent = WebResearchAgent()

response = agent.chat_about_topic(
    topic="blockchain technology",
    question="What are the main use cases of blockchain?"
)

print(response)

Run Examples

cd src
python example_usage.py

API Reference

WebResearchAgent

Main class for researching topics.

Methods:

• search_topic(topic: str, num_results: int = 5) -> List[Dict]

  • Search the web for a topic
  • Returns list of search results with title, link, and body

• async research_topic(topic: str, num_results: int = 5, fetch_content: bool = True) -> Dict

  • Research a topic by searching and optionally fetching content
  • Returns dictionary with search results and fetched content

• generate_summary(research_data: Dict) -> str

  • Generate an AI summary from research data
  • Returns summary text

• async full_research(topic: str, num_results: int = 5) -> Dict

  • Perform complete research: search, fetch, and summarize
  • Returns comprehensive research data with AI summary

• chat_about_topic(topic: str, question: str) -> str

  • Have a conversation about a specific topic
  • Returns AI response

WebSearcher

Handles web searches and content extraction.

Methods:

• search(query: str, max_results: int = 5) -> List[Dict]

  • Search using DuckDuckGo
  • Returns search results

• async fetch_content(url: str) -> Optional[str]

  • Fetch and extract text from a URL
  • Returns extracted text (max 2000 chars)

• async fetch_multiple(urls: List[str]) -> Dict[str, Optional[str]]

  • Fetch content from multiple URLs concurrently
  • Returns dictionary mapping URLs to content

Dependencies

• requests: HTTP library for web requests
• beautifulsoup4: HTML/XML parsing
• duckduckgo-search: DuckDuckGo search integration
• python-dotenv: Environment variable management
• aiohttp: Async HTTP client for concurrent requests
• Ollama CLI: Local LLM runtime installed separately

Configuration

Environment Variables

You can configure the local Ollama model with a .env file in the project root:

OLLAMA_MODEL=llama2

Customization

Modify the agent behavior by adjusting parameters:

# Use a different local Ollama model
agent = WebResearchAgent(model="llama2-mini")

# Adjust number of search results
results = await agent.full_research(topic, num_results=10)

# Control content fetching timeout
searcher = WebSearcher(timeout=15)

Error Handling

The agent includes comprehensive error handling:

• Invalid API keys are caught on initialization
• Network errors during content fetching are logged
• LLM call failures are handled gracefully
• Search failures return empty results with logging

Performance Tips

• Use async operations (full_research) for multiple concurrent URL fetches
• Limit max_results for faster searches
• Cache results if researching similar topics
• Use smaller local Ollama models for faster inference and larger models for higher-quality summaries

Limitations

• Web search results depend on DuckDuckGo availability
• Content extraction works best with text-heavy HTML
• LLM summaries are limited to token constraints
• Some websites may block automated requests

Future Enhancements

• Support for multiple LLM providers (Claude, Anthropic, etc.)
• Search result caching and deduplication
• PDF and document support
• Custom prompt templates
• Integration with knowledge bases
• Multi-language support
• Advanced filtering and ranking of results

License

MIT License

Contributing

Feel free to submit issues and enhancement requests!

Support

For issues or questions:

1. Check the examples in example_usage.py
2. Review error logs for debugging
3. Verify Ollama is installed and the selected model is available locally
4. Ensure internet connectivity for web searches and model downloads

---

Happy researching! 🚀