🚀 Go Tasks - Distributed Task Queue System

Production-ready distributed task queue and job scheduler system, similar to Celery, built with Go.

🚀 Quick Start

Prerequisites

• Go 1.24 or higher
• Redis server (optional, will use mock queue if unavailable)

Installation

# Clone the repository
git clone https://github.com/mstfsu/go-tasks.git
cd go-tasks

# Install dependencies
go mod download

# Build the application
make build
# Binary will be created at: ./bin/go-tasks

Running

# Start Redis (if using Docker)
docker run -d -p 6379:6379 redis:alpine

# Run the server
./bin/go-tasks

# Or with custom Redis configuration
REDIS_ADDR=localhost:6379 REDIS_PASSWORD=yourpassword ./bin/go-tasks

Access Points

• 🌐 Dashboard: http://localhost:8080
• 📡 REST API: http://localhost:8080/api/v1/
• 📊 Metrics: http://localhost:8080/metrics
• 🔌 gRPC: localhost:50051

---

📸 Screenshots

Dashboard

Real-time task monitoring with live status updates

Task Creation

Simple and intuitive task creation interface

Running Tasks

Active task processing with status updates

---

---

🏗️ Architecture

┌──────────────────┐
│   Web Dashboard  │
│  (Real-time UI)  │
└────────┬─────────┘
         │
         ▼
┌─────────────────────────────────────────────────────┐
│                    API Layer                         │
│  ┌──────────────┐              ┌──────────────┐    │
│  │  REST API    │              │  gRPC Server │    │
│  │   (Gin)      │              │  (Protocol)  │    │
│  └──────┬───────┘              └──────┬───────┘    │
└─────────┼──────────────────────────────┼───────────┘
          │                              │
          ▼                              ▼
┌─────────────────────────────────────────────────────┐
│              Queue Layer (Redis)                     │
│  • Priority Queue  • Distributed  • Persistent      │
└────────┬────────────────────────────────┬───────────┘
         │                                │
         ▼                                ▼
┌─────────────────┐              ┌─────────────────┐
│  Worker Pool    │              │   Scheduler     │
│  • Concurrent   │              │  • Cron Jobs    │
│  • Retry Logic  │              │  • Recurring    │
│  • Callbacks    │              │  • Timezone     │
└────────┬────────┘              └─────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────┐
│              Monitoring & Metrics                    │
│  • Prometheus Export  • Task Stats  • Health Check  │
└─────────────────────────────────────────────────────┘

---

📚 API Examples

REST API

Create a Task

curl -X POST http://localhost:8080/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "name": "process_user_data",
    "payload": {"user_id": 123, "action": "export"},
    "priority": 10
  }'

Get Task Status

curl http://localhost:8080/api/v1/tasks/{task_id}

Schedule a Job

curl -X POST http://localhost:8080/api/v1/scheduler/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "name": "daily_cleanup",
    "schedule": "0 0 * * *",
    "task_name": "cleanup_task",
    "payload": {}
  }'

gRPC API

import "github.com/mstfsu/go-tasks/proto"

client := proto.NewTaskServiceClient(conn)
response, err := client.CreateTask(ctx, &proto.CreateTaskRequest{
    Name:     "process_data",
    Payload:  `{"key": "value"}`,
    Priority: 5,
})

---

🛠️ Technology Stack

Component	Technology
Language	Go 1.24+
Queue	Redis 6.0+
REST Framework	Gin
RPC	gRPC + Protocol Buffers
Scheduler	robfig/cron
Metrics	Prometheus
Testing	testify
Frontend	HTML5 + Vanilla JS

---

📁 Project Structure

┌─────────────────────────────────────────────────────┐
│              Queue Layer (Redis)                     │
│  • Priority Queue  • Distributed  • Persistent      │
└────────┬────────────────────────────────┬───────────┘
         │                                │
         ▼                                ▼
┌─────────────────┐              ┌─────────────────┐
│  Worker Pool    │              │   Scheduler     │
│  • Concurrent   │              │  • Cron Jobs    │
│  • Retry Logic  │              │  • Recurring    │
│  • Callbacks    │              │  • Timezone     │
└────────┬────────┘              └─────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────┐
│              Monitoring & Metrics                    │
│  • Prometheus Export  • Task Stats  • Health Check  │
└─────────────────────────────────────────────────────┘
```un the server (with Redis)
REDIS_ADDR=localhost:6379 ./bin/go-tasks

# Run without Redis (uses mock queue)
./bin/go-tasks

# Access dashboard at http://localhost:8080
# API endpoints at http://localhost:8080/api/v1/
# Metrics at http://localhost:8080/metrics

Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Client    │────▶│   Broker    │────▶│   Worker    │
│  (REST/gRPC)│     │   (Redis)   │     │    Pool     │
└─────────────┘     └─────────────┘     └─────────────┘
                           │                    │
                           ▼                    ▼
                    ┌─────────────┐     ┌─────────────┐
                    │  Scheduler  │     │  Metrics    │
                    │   (Cron)    │     │(Prometheus) │
                    └─────────────┘     └─────────────┘

Project Structure

go-tasks/
├── cmd/
│   └── server/              # Main server application
├── internal/
│   ├── task/                # Task models and lifecycle
│   ├── queue/               # Queue interface + Redis/Mock implementations
│   ├── worker/              # Worker pool with retry logic
│   ├── retry/               # Exponential backoff retry handler
│   ├── scheduler/           # Cron-based job scheduler
│   ├── api/                 # REST API (Gin) + recent tasks tracking
│   ├── grpc/                # gRPC server implementation
│   └── metrics/             # Prometheus metrics collector
├── proto/                   # Protocol buffer definitions
├── web/
│   └── templates/           # Dashboard HTML templates
├── Makefile                 # Build and test commands
└── README.md

---

🧪 Testing

# Run all tests
make test

# Run tests with coverage
go test -v -cover ./...

# Run specific package tests
go test -v ./internal/worker/

Test Coverage: 72 comprehensive tests across all modules

• ✅ Task lifecycle and state management
• ✅ Queue operations (enqueue, dequeue, priority)
• ✅ Worker pool concurrency and shutdown
• ✅ Retry mechanism with backoff
• ✅ Scheduler job execution
• ✅ API endpoints (REST + gRPC)
• ✅ Metrics collection and export

---

📦 Configuration

Environment Variables

# Redis Configuration
REDIS_ADDR=localhost:6379       # Redis server address
REDIS_PASSWORD=                 # Redis password (optional)
REDIS_DB=0                      # Redis database number

# Server Configuration
SERVER_PORT=8080                # HTTP server port
GRPC_PORT=50051                # gRPC server port

# Worker Configuration
NUM_WORKERS=2                   # Number of concurrent workers
POLL_INTERVAL=1s               # Queue polling interval
SHUTDOWN_TIMEOUT=10s           # Graceful shutdown timeout

Worker Configuration

workerConfig := worker.WorkerConfig{
    NumWorkers:      2,                    // Concurrent workers
    PollInterval:    time.Second,          // How often to check queue
    ShutdownTimeout: 10 * time.Second,     // Graceful shutdown time
    RetryConfig: retry.RetryConfig{
        MaxRetries:      3,
        InitialInterval: time.Second,
        MaxInterval:     time.Minute,
        Multiplier:      2.0,
    },
}

---

🎯 Use Cases

• Microservices: Decouple services with async task processing
• ETL Pipelines: Process large datasets in background
• Email/Notifications: Send bulk communications asynchronously
• Data Export: Generate and deliver reports on demand
• Scheduled Tasks: Run periodic cleanup, backups, aggregations
• Webhook Processing: Handle webhook callbacks reliably
• Image/Video Processing: Process media files in background

---

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the project
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Acknowledgments

• Inspired by Celery (Python)
• Built with love using Go
• Queue powered by Redis

---

⭐ Star this repo if you find it useful!

Made with ❤️ by mstfsu

└── web/ # Dashboard UI ```

Getting Started

Prerequisites

• Go 1.21+
• Redis 6.0+
• Docker (optional)

Installation

go mod download

Running Tests

go test ./...

Running the Server

go run cmd/server/main.go

Running a Worker

go run cmd/worker/main.go

Development Progress

This project is being built step-by-step with comprehensive testing at each stage.

License

MIT