add readme
This commit is contained in:
buenosairesam
214
README.md
Normal file
214
README.md
Normal file
@@ -0,0 +1,214 @@
|
|||||||
|
# sysmonstm
|
||||||
|
|
||||||
|
A real-time distributed system monitoring platform that streams metrics from multiple machines to a central hub with a live web dashboard.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
sysmonstm demonstrates production microservices patterns (gRPC streaming, FastAPI, event-driven architecture) while solving a real problem: monitoring development infrastructure across multiple machines.
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||||
|
│ Collector │ │ Collector │ │ Collector │
|
||||||
|
│ (Machine 1) │ │ (Machine 2) │ │ (Machine N) │
|
||||||
|
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
|
||||||
|
│ │ │
|
||||||
|
│ gRPC Streaming │
|
||||||
|
└───────────────────────┼───────────────────────┘
|
||||||
|
▼
|
||||||
|
┌────────────────────────┐
|
||||||
|
│ Aggregator │
|
||||||
|
│ (gRPC Server + Redis │
|
||||||
|
│ + TimescaleDB) │
|
||||||
|
└────────────┬───────────┘
|
||||||
|
│
|
||||||
|
┌──────────────────┼──────────────────┐
|
||||||
|
│ │ │
|
||||||
|
▼ ▼ ▼
|
||||||
|
┌────────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||||
|
│ Gateway │ │ Alerts │ │ Event Stream│
|
||||||
|
│ (FastAPI + WS) │ │ Service │ │ (Redis PubSub│
|
||||||
|
└────────┬───────┘ └──────────────┘ └──────────────┘
|
||||||
|
│
|
||||||
|
│ WebSocket
|
||||||
|
▼
|
||||||
|
┌────────────────┐
|
||||||
|
│ Browser │
|
||||||
|
│ Dashboard │
|
||||||
|
└────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Real-time streaming**: Collectors stream metrics via gRPC to central aggregator
|
||||||
|
- **Multi-machine support**: Monitor any number of machines from a single dashboard
|
||||||
|
- **Live dashboard**: WebSocket-powered updates with real-time graphs
|
||||||
|
- **Tiered storage**: Redis for hot data, TimescaleDB for historical analysis
|
||||||
|
- **Threshold alerts**: Configurable rules for CPU, memory, disk usage
|
||||||
|
- **Event-driven**: Decoupled services via Redis Pub/Sub
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start the full stack
|
||||||
|
docker compose up
|
||||||
|
|
||||||
|
# Open dashboard
|
||||||
|
open http://localhost:8000
|
||||||
|
```
|
||||||
|
|
||||||
|
Metrics appear within seconds. The collector runs locally by default.
|
||||||
|
|
||||||
|
### Monitor Additional Machines
|
||||||
|
|
||||||
|
Run the collector on any machine you want to monitor:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On a remote machine, point to your aggregator
|
||||||
|
COLLECTOR_AGGREGATOR_URL=your-server:50051 \
|
||||||
|
COLLECTOR_MACHINE_ID=my-laptop \
|
||||||
|
python services/collector/main.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Services
|
||||||
|
|
||||||
|
| Service | Port | Description |
|
||||||
|
|---------|------|-------------|
|
||||||
|
| **Collector** | - | gRPC client that streams system metrics (CPU, memory, disk, network) |
|
||||||
|
| **Aggregator** | 50051 | gRPC server that receives metrics, stores them, publishes events |
|
||||||
|
| **Gateway** | 8000 | FastAPI server with REST API and WebSocket for dashboard |
|
||||||
|
| **Alerts** | - | Subscribes to events, evaluates threshold rules, triggers notifications |
|
||||||
|
|
||||||
|
### Infrastructure
|
||||||
|
|
||||||
|
| Component | Purpose |
|
||||||
|
|-----------|---------|
|
||||||
|
| **Redis** | Current state cache, event pub/sub |
|
||||||
|
| **TimescaleDB** | Historical metrics with automatic downsampling |
|
||||||
|
|
||||||
|
### Key Patterns
|
||||||
|
|
||||||
|
- **gRPC Streaming**: Collectors stream metrics continuously to the aggregator
|
||||||
|
- **Event-Driven**: Services communicate via Redis Pub/Sub for decoupling
|
||||||
|
- **Tiered Storage**: Hot data in Redis, historical in TimescaleDB
|
||||||
|
- **Graceful Degradation**: System continues partially if storage fails
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
sysmonstm/
|
||||||
|
├── proto/
|
||||||
|
│ └── metrics.proto # gRPC service definitions
|
||||||
|
├── services/
|
||||||
|
│ ├── collector/ # Metrics collection (psutil)
|
||||||
|
│ ├── aggregator/ # Central gRPC server
|
||||||
|
│ ├── gateway/ # FastAPI + WebSocket
|
||||||
|
│ └── alerts/ # Threshold evaluation
|
||||||
|
├── shared/
|
||||||
|
│ ├── config.py # Pydantic settings
|
||||||
|
│ ├── logging.py # Structured JSON logging
|
||||||
|
│ └── events/ # Event pub/sub abstraction
|
||||||
|
├── web/
|
||||||
|
│ ├── static/ # CSS, JS
|
||||||
|
│ └── templates/ # Dashboard HTML
|
||||||
|
├── scripts/
|
||||||
|
│ └── init-db.sql # TimescaleDB schema
|
||||||
|
├── docs/ # Architecture diagrams & explainers
|
||||||
|
├── docker-compose.yml
|
||||||
|
└── Tiltfile # Local Kubernetes dev
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
All services use environment variables with sensible defaults:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Collector
|
||||||
|
COLLECTOR_MACHINE_ID=my-machine # Machine identifier
|
||||||
|
COLLECTOR_AGGREGATOR_URL=localhost:50051
|
||||||
|
COLLECTOR_COLLECTION_INTERVAL=5 # Seconds between collections
|
||||||
|
|
||||||
|
# Common
|
||||||
|
REDIS_URL=redis://localhost:6379
|
||||||
|
TIMESCALE_URL=postgresql://monitor:monitor@localhost:5432/monitor
|
||||||
|
LOG_LEVEL=INFO
|
||||||
|
LOG_FORMAT=json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Metrics Collected
|
||||||
|
|
||||||
|
- CPU: Overall percentage, per-core usage
|
||||||
|
- Memory: Percentage, used/available bytes
|
||||||
|
- Disk: Percentage, used bytes, read/write throughput
|
||||||
|
- Network: Bytes sent/received per second, connection count
|
||||||
|
- System: Process count, load averages (1m, 5m, 15m)
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
### Local Development with Hot Reload
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Use the override file for volume mounts
|
||||||
|
docker compose -f docker-compose.yml -f docker-compose.override.yml up
|
||||||
|
```
|
||||||
|
|
||||||
|
### Kubernetes Development with Tilt
|
||||||
|
|
||||||
|
```bash
|
||||||
|
tilt up
|
||||||
|
```
|
||||||
|
|
||||||
|
### Running Services Individually
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install dependencies
|
||||||
|
python -m venv .venv
|
||||||
|
source .venv/bin/activate
|
||||||
|
pip install -r services/collector/requirements.txt
|
||||||
|
|
||||||
|
# Generate protobuf code
|
||||||
|
python -m grpc_tools.protoc -I proto --python_out=. --grpc_python_out=. proto/metrics.proto
|
||||||
|
|
||||||
|
# Run a service
|
||||||
|
python services/collector/main.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
### REST (Gateway)
|
||||||
|
|
||||||
|
| Endpoint | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| `GET /` | Dashboard UI |
|
||||||
|
| `GET /api/machines` | List all monitored machines |
|
||||||
|
| `GET /api/machines/{id}/metrics` | Current metrics for a machine |
|
||||||
|
| `GET /api/machines/{id}/history` | Historical metrics |
|
||||||
|
| `GET /health` | Health check |
|
||||||
|
| `GET /ready` | Readiness check (includes dependencies) |
|
||||||
|
|
||||||
|
### WebSocket
|
||||||
|
|
||||||
|
Connect to `ws://localhost:8000/ws` for real-time metric updates.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
Detailed documentation is available in the `docs/` folder:
|
||||||
|
|
||||||
|
- [Architecture Diagrams](docs/architecture/) - System overview, data flow, deployment
|
||||||
|
- [Building sysmonstm](docs/explainer/sysmonstm-from-start-to-finish.md) - Deep dive into implementation decisions
|
||||||
|
- [Domain Applications](docs/explainer/other-applications.md) - How these patterns apply to payment processing and other domains
|
||||||
|
|
||||||
|
## Tech Stack
|
||||||
|
|
||||||
|
- **Python 3.11+** with async/await throughout
|
||||||
|
- **gRPC** for inter-service communication
|
||||||
|
- **FastAPI** for REST API and WebSocket
|
||||||
|
- **Redis** for caching and pub/sub
|
||||||
|
- **TimescaleDB** for time-series storage
|
||||||
|
- **psutil** for system metrics collection
|
||||||
|
- **Docker Compose** for orchestration
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT
|
||||||
Reference in New Issue
Block a user