Distributed System Monitoring Platform

Project Overview

A real-time system monitoring platform that streams metrics from multiple machines to a central hub with live web dashboard. Built to demonstrate production microservices patterns (gRPC, FastAPI, streaming, event-driven architecture).

Primary Goal: Portfolio project demonstrating real-time streaming with gRPC
Status: Working, deployed at sysmonstm.mcrn.ar

Architecture

┌─────────────┐     ┌─────────────────────────────────────┐     ┌─────────────┐
│  Collector  │────▶│  Aggregator + Gateway + Redis + TS  │────▶│    Edge     │────▶ Browser
│   (mcrn)    │gRPC │            (LOCAL)                  │ WS  │   (AWS)     │ WS
└─────────────┘     └─────────────────────────────────────┘     └─────────────┘
┌─────────────┐                    │
│  Collector  │────────────────────┘
│   (nfrt)    │gRPC
└─────────────┘

Collectors (services/collector/) - gRPC clients on each monitored machine
Aggregator (services/aggregator/) - gRPC server, stores in Redis/TimescaleDB
Gateway (services/gateway/) - FastAPI, bridges gRPC to WebSocket, forwards to edge
Edge (ctrl/edge/) - Simple WebSocket relay for AWS, serves public dashboard

Directory Structure

sms/
├── services/                   # gRPC-based microservices
│   ├── collector/              # gRPC client, streams to aggregator
│   ├── aggregator/             # gRPC server, stores in Redis/TimescaleDB
│   ├── gateway/                # FastAPI, WebSocket, forwards to edge
│   └── alerts/                 # Event subscriber for threshold alerts
│
├── ctrl/                       # Deployment configurations
│   ├── dev/                    # Full stack docker-compose
│   └── edge/                   # Cloud dashboard (AWS)
│
├── proto/                      # Protocol Buffer definitions
├── shared/                     # Shared Python modules (config, logging, events)
└── web/                        # Dashboard templates and static files

Running

Local Development

docker compose up

With Edge Forwarding (to AWS)

EDGE_URL=wss://sysmonstm.mcrn.ar/ws docker compose up

Collector on Remote Machine

docker run -d --network host \
  -e AGGREGATOR_URL=<local-ip>:50051 \
  -e MACHINE_ID=$(hostname) \
  registry.mcrn.ar/sysmonstm/collector:latest

Technical Stack

Python 3.11+
gRPC - Collector to aggregator communication (showcased)
FastAPI - Gateway REST/WebSocket
Redis - Pub/Sub events, current state cache
TimescaleDB - Historical metrics storage
WebSocket - Gateway to edge, edge to browser

Key Files

File	Purpose
`proto/metrics.proto`	gRPC service and message definitions
`services/collector/main.py`	gRPC streaming client
`services/aggregator/main.py`	gRPC server, metric processing
`services/gateway/main.py`	WebSocket bridge, edge forwarding
`ctrl/edge/edge.py`	Simple WebSocket relay for AWS

Portfolio Talking Points

gRPC streaming - Efficient binary protocol for real-time metrics
Event-driven - Redis Pub/Sub decouples processing from delivery
Edge pattern - Heavy processing local, lightweight relay in cloud
Cost optimization - ~$10/mo for public dashboard (data transfer, not requests)

3.8 KiB Raw Blame History