STELLAR AIR

Concurrency model

Everything runs on one OS thread under asyncio — no GIL contention, no thread locks. Shared mutable state (runs: dict, event_hub._clients: set) is safe because mutations are atomic relative to the event loop scheduler, and disconnects happen between awaits so broadcast iteration is race-free. The FCE agent fires five asyncio.create_task calls then asyncio.gather — five MCP tool calls run concurrently but cooperatively. This only breaks once runs grows large enough to want sharding across processes, at which point the in-process guarantees evaporate and a Redis-backed store becomes necessary (see Roadmap).

Stateless API, stateful MCP subprocesses

Each agent run spawns three MCP server subprocesses over stdio. This is wasteful per-request (~500 ms cold-start) but has one decisive advantage: full isolation. No shared scenario state across runs, no mutex on the scenario manager, no "wait, whose data was this?". The path forward is Streamable HTTP transport with long-lived servers — same tool code, different transport — which is a config change rather than a rewrite.

Domain-scoped MCP servers

Three servers — shared, ops, passenger — not one with RBAC filtering. The passenger agent literally cannot call get_crew_duty_status because it never connects to the ops server; the capability isn't even discoverable. Security boundary by architecture, not by authorization. Filter bugs become security bugs; MCP is a capability protocol, so using its native scoping is cleaner than bolting auth on top. If ops tools ever move to a separate team or repo they just become a separately-deployed MCP server — agents update their profile, not their code.

Tools, Resources, and Prompts

All three MCP primitives are used. Tools are actions or queries with potential side effects: get_flight_status, generate_notification. Resources are read-only data with URIs: ops://hubs/{code}, ops://handover/latest — a dynamic resource (updated after each handover) is still a resource because reading it has no side effects. Prompts are server-versioned templates: delay_explainer(cause_code, audience), passenger-notification(tone). The split matters because it lets the server own prompt versioning — update the template on the server and every client picks it up without a redeploy.

Why MCP over function calling, LangChain, or direct APIs

MCP wins when there are multiple consumers of the same tools (here, both a LangGraph agent and Claude Code), when dynamic tool discovery matters, and when protocol-level contracts are worth having. Provider function calling (OpenAI, Anthropic) bakes tool definitions into prompts and locks to one vendor. LangChain tools couple to LangChain's abstractions. Direct API calls are the N×M integration problem. MCP doesn't replace function calling — the LLM still uses its native tool-calling mechanism — it standardises the execution layer underneath.

Approach	Strengths	Weaknesses
MCP	Standard, discoverable, client-agnostic, composable	Extra process, protocol overhead for simple cases
Function calling	Simple, no extra infrastructure	Provider-locked, no runtime discovery, definitions duplicated per call
LangChain tools	Tight framework integration	Coupled to LangChain, not usable outside
Direct API calls	No abstraction overhead	N×M integration problem, no standardisation

LLM provider abstraction

One generate(system_prompt, user_content) function in mcp_servers/shared_llm.py with four backends: Groq (default, free), Anthropic, Bedrock, and any OpenAI-compatible endpoint. Selection happens at runtime via LLM_PROVIDER. LangChain's provider abstraction is heavier than needed here — string in, string out is enough — and switching providers touches one env var rather than the agent code.

Every narrative tool also has a structured template fallback. Response format is identical: {"text": str, "provider": str}. The UI surfaces the provider as a badge, so it's always visible whether a response came from an LLM or the template — honest about what mode the system is in. Tests pass without any API key; the demo works without any API key.

Scenarios in memory, not a database

Scenarios are Python modules, versioned with git, loaded lazily by the scenario manager. They are deliberately designed datasets, not user-generated content — git is more valuable than CRUD for them, and switching scenarios is a config change rather than a data migration. The reload-on-subprocess-spawn pattern sidesteps the cache-invalidation problem entirely. This would break once scenarios became per-tenant or grew beyond ~50 MB — then it's a database.

Dual-mode tests

tests/base.py supports two transports with the same 69 assertions. Default (inprocess) uses httpx.AsyncClient over ASGI — no server needed. live mode runs real HTTP against any CONTRACT_TEST_URL, so the same tests validate a deployed instance. Contract tests are definitionally transport-agnostic; duplicating them into two files would be the bug factory every project eventually regrets.

Kong as additive

The app works with or without Kong. When VITE_KONG_PROXY_URL is empty the UI calls FastAPI directly; when set it routes through Kong Konnect for rate limiting, analytics, and the path to auth. Graceful degradation beats a broken demo — especially relevant when the gateway sits on a trial subscription with a finite lifetime.

Langfuse in a shared cluster

Langfuse runs in its own Kind cluster separate from the app cluster. The v3 stack needs ClickHouse, Redis, MinIO, and a worker — four extra pods that aren't project-specific. Putting it in a shared cluster means every project points LANGFUSE_HOST at the same instance: one dashboard, one set of keys, one upgrade path. That's how Langfuse belongs in production — shared infra, not per-service.

Timeouts, TTL cleanup, error handling

Every MCP tool call is wrapped in asyncio.wait_for with a 15-second timeout — long enough to catch real hangs without false positives from slow-but-alive APIs (OpenMeteo and FAA typically respond in under 2 s). On timeout the span is marked ERROR in Langfuse, the error is added to the run's error list, and the agent continues with partial data. The notification prompt is told which sources are missing and omits them rather than hallucinating.

The in-memory run store is pruned by a background task that removes completed or errored runs older than one hour. Errors surface with proper HTTP status codes — HTTPException(404, …) for missing resources, 400 for invalid requests — rather than 200 responses with an error body, so clients can distinguish failure without parsing the payload.

Roadmap

Items deferred intentionally — the system works without them, and each is a clean extension rather than a rewrite.

• MCP over Streamable HTTP. Replace subprocess-per-run with long-lived server processes. Becomes worthwhile once cold-start latency matters in aggregate or once MCP needs to serve multiple API replicas.
• Redis-backed run store and event bus. Enables multi-instance WebSocket broadcast and survives API restarts. Necessary as soon as the API scales past a single process.
• Database-backed scenarios. Replace the in-memory modules with a datastore once scenarios need to be per-tenant or grow beyond what fits comfortably in git.
• Circuit breakers on external APIs. Exponential backoff and breakers on FAA and OpenMeteo via tenacity. Worth doing once those APIs have their first real outage.
• Kong Key Auth. Per-consumer access control and per-agent rate limits. Unlocks multi-tenant use and a formal API-key lifecycle.