Log levels and filtering

[ralyodio]

📋 Product Requirements Document

PRD: Log levels and filtering

Issue: #163
Milestone: Phase 6: Observability
Labels: logging, hacktoberfest

---

PRD: Log Levels and Filtering

Overview

The MeshHook project aims to introduce log levels and filtering to enhance its observability features, aligning with the project's goal of providing a reliable and scalable webhook-first, deterministic workflow engine. This feature is critical for debugging and monitoring the system's health, enabling users and administrators to efficiently diagnose and resolve issues. By implementing structured logging with log levels and dynamic filtering capabilities, MeshHook will offer improved insights into its operations, thereby supporting its core objective of delivering a robust workflow engine.

Functional Requirements

1. Log Levels: Implement structured logging with predefined log levels (DEBUG, INFO, WARN, ERROR, FATAL) to categorize log messages according to their severity and importance.
2. Dynamic Log Filtering: Develop functionality to filter logs dynamically based on log levels, timestamps, and potentially other metadata like component or workflow identifiers, both through the UI and API.
3. User-configurable Log Levels: Enable users to set default log levels via the MeshHook UI and through configuration files, which will determine the verbosity of logging output for their environment.
4. Log Persistence and Real-time Viewing: Store logs in a structured and queryable format within Supabase, ensuring they are accessible for real-time viewing in the MeshHook console, facilitated by Supabase Realtime.

Non-Functional Requirements

• Performance: Ensure that the logging mechanism, including collection, filtering, and querying, is optimized for high performance and minimal impact on workflow execution.
• Reliability: Design the logging system to be fault-tolerant, making sure that no logs are lost during normal and high-load operations.
• Security: Implement measures to redact sensitive information from logs and enforce role-based access control for viewing logs.
• Maintainability: The logging framework should be easy to extend and maintain, adhering to MeshHook's coding standards and best practices.

Technical Specifications

Architecture Context

MeshHook leverages SvelteKit for its UI and API layers, and Supabase for database and real-time functionalities. The logging system must integrate seamlessly within this architecture, using Supabase for log storage and enabling real-time log viewing through the MeshHook console.

Implementation Approach

1. Analysis: Conduct an audit of current logging practices within MeshHook to identify gaps and integration points for structured logging.
2. Design:
   • Define a logging interface that encapsulates structured logging functionalities, including log levels and metadata.
   • Design the integration with Supabase for efficient log storage and querying.
   • Design the UI components for log configuration and filtering, ensuring a user-friendly experience.
3. Implementation:
   • Update the existing logging system to support structured logging with the defined log levels.
   • Create backend services for storing logs in Supabase and enabling real-time log viewing.
   • Implement UI components for log level configuration and log filtering.
4. Integration: Ensure seamless integration of the logging features with existing MeshHook components, focusing on the UI, backend services, and Supabase integration.
5. Testing: Execute comprehensive testing, including unit tests for new logic, integration tests for system components, and performance tests to meet benchmarks.
6. Documentation: Update project documentation to cover the new logging features, usage instructions, and configuration options.

Data Model Changes

• Logs Table: Introduce a logs table in Supabase with columns for level, timestamp, message, and metadata (JSON type to store additional context).

API Endpoints

• GET /api/logs: Fetch logs based on specified filter criteria, such as level and date range.
• POST /api/logs/config: Allow updating the system-wide default log level.

Acceptance Criteria

• Log levels (DEBUG, INFO, WARN, ERROR, FATAL) are fully implemented across MeshHook's components and can be dynamically configured.
• Logs are filterable by level, timestamp, and other specified criteria through both the UI and API.
• Logs are persistently stored in a structured format within Supabase and can be viewed in real-time through the MeshHook console.
• The logging system's performance meets predefined benchmarks, ensuring minimal impact on the core workflow engine.
• Access to logs is securely controlled, with sensitive information appropriately redacted.

Dependencies

• Access to MeshHook's codebase and development environment.
• A configured Supabase project for log storage and real-time capabilities.

Implementation Notes

Development Guidelines

• Adhere to MeshHook's project structure and coding conventions.
• Ensure new code is well-documented and includes appropriate test coverage.
• Use environment variables for configuration settings to support various deployment environments.

Testing Strategy

• Develop unit tests for new logic, particularly around log level management and filtering.
• Conduct integration tests to confirm seamless functionality with existing MeshHook components and Supabase integration.
• Implement performance testing to verify that logging operations meet efficiency benchmarks.

Security Considerations

• Enforce role-based access control for viewing logs, ensuring only authorized users can access them.
• Implement data sanitization to prevent sensitive information from being logged.

Monitoring & Observability

• Set up monitoring for the logging system to track its performance and reliability.
• Establish alerting mechanisms for critical logging failures and threshold breaches.

---

This PRD was AI-generated using gpt-4-turbo-preview from GitHub issue #163
Generated: 2025-10-10

📎 Generated Documentation

• 📄 PRD Document: 163-log-levels-and-filtering.md
• 🎨 PlantUML Diagram: 163-log-levels-and-filtering.puml
• 🖼️ Diagram Image: 163-log-levels-and-filtering.png

---

This issue body was auto-generated from the PRD. Original issue content is preserved in the PRD document.
Last updated: 2025-10-10