🧠 Humanity's Last Exam - Human Benchmarking Quiz App

A comprehensive web-based quiz application that adapts the Humanity's Last Exam (HLE) dataset to benchmark human performance instead of AI models.

🌟 Features

• 📝 Interactive Quizzes: Take quizzes across multiple academic subjects
• 📊 Detailed Analytics: Track your performance with comprehensive statistics
• 🎯 Subject, Type & Difficulty Filtering: Filter by HLE category (subject), question type and difficulty
• 📈 Progress Tracking: Monitor your improvement over time
• 🔄 Resumable Sessions: Continue where you left off
• 📱 Responsive Design: Works on desktop and mobile devices
• 🗃️ Persistent Storage: Questions persisted in SQLite (hle_quiz.db)
• 🔌 Decoupled Architecture: Streamlit app consumes a local API (api_server.py) that serves questions from the DB
• 🧭 Adaptive Mode (beta): One-question-at-a-time flow with simple difficulty steering
• 🔐 API key auth + ⏱️ Rate limiting: Optional protection for all data endpoints

🚀 Quick Start

1. Install Dependencies

pip install -r quiz_requirements.txt

2. Set your Hugging Face token (required for gated HLE)

export HF_TOKEN=hf_...   # or run: python setup_hf_token.py

3. Start the Question API (serves from SQLite)

python -m uvicorn api.api_server:app --host 0.0.0.0 --port 8000

Optional configuration before starting:

# Protect API with a shared key (client will send X-API-Key)
export HLE_API_KEY=your_secret_key

# Enable Redis for rate limiting and served-ID tracking
export REDIS_URL=redis://localhost:6379/0

4. Run the Quiz App (consumes the API)

streamlit run app/human_quiz_app.py

If your API is not on http://localhost:8000, set:

export HLE_API_URL=http://your-api-host:8000
# If API key auth is enabled on the server
export HLE_API_KEY=your_secret_key

5. Open Your Browser

Navigate to http://localhost:8501 to start taking quizzes!

📋 How to Use

Taking a Quiz

1. Start Quiz: Click "Start New Quiz" and choose the number of questions
2. Answer Questions: Type your answers in the text fields
3. Submit: Click "Submit Quiz" when finished
4. Review Results: See your score, accuracy, and detailed explanations

Adaptive Mode (beta)

• Toggle "Adaptive mode" before starting a quiz to receive one question at a time.
• After each answer, the app locally checks correctness and requests the next question from the API, steering toward easier or harder questions using a simple length-based difficulty proxy.
• The server uses Redis (if configured) to avoid repeats per session_id.

Analytics Dashboard

• Overall Statistics: Total quizzes, average accuracy, best score
• Performance Over Time: Track your progress with line charts
• Subject Performance: See how you perform in different subjects
• Difficulty Analysis: Compare performance across difficulty levels
• Recent Results: View your latest quiz attempts

🔧 Configuration

Using the Full HLE Dataset

To use the actual HLE dataset (requires authentication):

1. Get Access: Request access to the HLE dataset on Hugging Face
2. Authenticate: export HF_TOKEN=... or run python setup_hf_token.py
3. Run the stack: Start the API and the Streamlit app (see Quick Start). On first run, the API/database loader will ingest the HLE dataset into hle_quiz.db and all subsequent queries will be served from the DB.

No code changes are required.

Custom Questions

You can add your own questions by modifying the SAMPLE_QUESTIONS list in human_quiz_app.py:

SAMPLE_QUESTIONS = [
    {
        "id": "custom_001",
        "question": "Your question here?",
        "answer": "Correct answer",
        "subject": "Your Subject",
        "difficulty": "Basic/Intermediate/Advanced",
        "explanation": "Explanation of the answer"
    },
    # ... more questions
]

📊 Data Structure

Each question follows this format:

{
    "id": "unique_identifier",
    "question": "The question text",
    "answer": "Correct answer",
    "subject": "HLE category (e.g., Physics, Math, CS/AI, …)",
    "raw_subject": "Fine-grained subject from HLE (e.g., Applied Mathematics)",
    "difficulty": "Basic/Intermediate/Advanced",
    "explanation": "Explanation of the answer",
    "image": "URL to image (for multi-modal questions)",
    "question_type": "text/image"
}

📈 Benchmarking Features

Performance Metrics

• Accuracy: Percentage of correct answers
• Speed: Time taken per question
• Subject Mastery: Performance by academic subject
• Difficulty Progression: Performance across difficulty levels
• Consistency: Standard deviation of scores

Analytics Dashboard

The app provides comprehensive analytics including:

• Performance trends over time
• Subject-wise breakdown
• Difficulty level analysis
• Comparative statistics
• Progress tracking

🎯 Use Cases

Educational Institutions

• Student Assessment: Evaluate student knowledge across subjects
• Curriculum Planning: Identify areas needing more focus
• Progress Tracking: Monitor student improvement over time

Research

• Human Performance Studies: Compare human vs AI performance
• Cognitive Science: Study learning patterns and knowledge retention
• Educational Psychology: Analyze difficulty progression

Personal Development

• Self-Assessment: Test your knowledge across various subjects
• Learning Goals: Set and track learning objectives
• Skill Development: Focus on specific subject areas

🔒 Data Privacy

• All quiz results are stored locally in human_benchmark_results.json
• No data is sent to external servers
• You can delete the results file at any time to clear your data

🛠️ Technical Details

Dependencies

• Streamlit: Web application framework
• Pandas: Data manipulation and analysis
• Plotly: Interactive visualizations
• NumPy: Numerical computations
• FastAPI + Uvicorn: Local API to serve questions from DB
• datasets: Hugging Face loader for gated HLE dataset
• Redis (optional): Rate limiting and served-ID tracking

Environment Variables

• HF_TOKEN: Hugging Face access token to load the HLE dataset
• HLE_API_URL: Base URL for the API used by the Streamlit client
• HLE_API_KEY: If set, API endpoints require X-API-Key: <HLE_API_KEY> header
• REDIS_URL: If set, enables rate limiting and served-ID tracking (e.g., redis://localhost:6379/0)

File Structure

├── app/
│   └── human_quiz_app.py      # Main Streamlit application (consumes API)
├── api/
│   └── api_server.py          # FastAPI service serving questions from DB
├── core/
│   ├── database_manager.py    # SQLite schema/queries and analytics
│   └── hle_database_loader.py # Ingests HLE dataset into SQLite
├── data/
│   ├── hle_quiz.db            # Persistent SQLite database (created on first run)
│   └── human_benchmark_results.json  # Quiz results (created after first quiz)
├── scripts/
│   ├── run_quiz.py            # Launcher for Streamlit app
│   └── setup_hf_token.py      # Helper to set HF token
├── docs/
│   └── hle/                   # Upstream HLE docs and assets (vendor)
├── quiz_requirements.txt      # Python dependencies
├── README.md                  # This file

🤝 Contributing

Feel free to contribute to this project by:

1. Adding more sample questions
2. Improving the UI/UX
3. Adding new analytics features
4. Enhancing the question filtering system
5. Adding support for different question types

📚 About HLE

Humanity's Last Exam (HLE) is a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. It consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and natural sciences.

• Website: lastexam.ai
• Paper: lastexam.ai/paper
• Dataset: Hugging Face

📄 License

This project is based on the HLE benchmark and follows the same MIT license as the original repository.

🙏 Acknowledgments

• Center for AI Safety for creating the HLE benchmark
• Hugging Face for hosting the dataset
• Streamlit for the web application framework

---

Note: This application is designed for educational and research purposes. The HLE dataset should never appear in training corpora for AI models.

🗺️ Planned Implementations

• Adaptive question serving

  • Per-user session state and simple difficulty curve (Elo/IRT-lite) to choose next question
  • Prevent repeats by tracking served IDs (e.g., Redis) per session

• API hardening

  • API key/JWT auth, rate limiting, CORS allowlist
  • Deterministic sampling (seed + filters → reproducible ordering)
  • Admin endpoints: refresh dataset, warm indexes, purge caches

• Data and storage

  • Keep SQLite for serving; add DuckDB read-only for analytics directly over Parquet
  • Lightweight DB migrations and ETL validation (pydantic)

• Performance and UX

  • Cache facet counts (/stats, /subjects) and serve instantly
  • SSE/WebSocket channel for timed rounds and lower latency
  • Image proxy with cache headers, lazy loading in UI

• Observability and ops

  • Structured logs, basic metrics (latency, QPS, cache hit rate)
  • Health/readiness probes; Docker Compose for local dev (API + Streamlit + Redis)
  • Contract tests from OpenAPI-generated client used in Streamlit

• Safety and data hygiene

  • Whitelist/strip sensitive fields (e.g., canary) from responses/UI
  • Content length caps and sanitization