# Observability

> CloudWatch logs, structured JSON, X-Ray, Lambda Insights, EMF. Brief Prometheus/Grafana orientation.

## CloudWatch Logs — what you get for free

Every Lambda function automatically writes to a CloudWatch Log Group named `/aws/lambda/<function-name>`. Each execution environment gets its own Log Stream. Lambda writes two special lines automatically:

```
START RequestId: abc-123 Version: $LATEST
END RequestId: abc-123
REPORT RequestId: abc-123  Duration: 312.45 ms  Billed Duration: 313 ms
        Memory Size: 256 MB  Max Memory Used: 89 MB
        Init Duration: 423.12 ms   # only on cold starts
```

The REPORT line is your free performance telemetry. `Init Duration` appears only on cold invocations. `Max Memory Used` helps right-size memory configuration.

**Retention:** Default is "Never Expire." Set it explicitly — 7, 14, or 30 days covers most needs. Every MB of retained logs costs money.

## Structured logging

Emit JSON instead of plain strings. CloudWatch Logs Insights can filter and aggregate JSON fields efficiently; plain strings require regex and are slow. Example:

```python
import json, logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def handler(event, context):
    logger.info(json.dumps({
        "event": "pdf_scan_start",
        "bucket": BUCKET,
        "prefix": PREFIX,
        "request_id": context.aws_request_id,
    }))
```

With this, Logs Insights can run: `filter event = "pdf_scan_start" | stats count() by bin(5m)` in seconds.

## X-Ray tracing

X-Ray gives you request traces across services — how long the Lambda itself ran vs how long S3 calls took. Three things must all be true:

1. **Tracing enabled on the function** — console toggle or `TracingConfig: Active` in SAM/CDK
2. **X-Ray SDK instrumented in your code** — `from aws_xray_sdk.core import patch_all; patch_all()` wraps boto3 calls automatically
3. **IAM permission** — execution role needs `xray:PutTraceSegments` and `xray:PutTelemetryRecords`

Without all three, traces are either absent or incomplete. People flip one and conclude X-Ray is broken.

## Lambda Insights

Lambda Insights is a CloudWatch feature (not a separate service) that surfaces system-level metrics: CPU usage, memory utilisation, network I/O, disk I/O — things the REPORT line doesn't include. To enable it:

- Add the Lambda Insights extension layer (`arn:aws:lambda:<region>:580247275435:layer:LambdaInsightsExtension:38`)
- Add `cloudwatch:PutMetricData` to the execution role

It's useful when you suspect memory or CPU contention but the REPORT line's "Max Memory Used" isn't granular enough.

## EMF — Embedded Metrics Format

EMF lets you emit custom CloudWatch metrics by writing structured JSON to stdout. No `PutMetricData` API call needed — the Lambda runtime parses the log line and publishes the metric asynchronously. This is far more efficient than calling CloudWatch from inside the handler (which adds latency + cost per invocation).

```python
import json

def emit_metric(name, value, unit="Count", **dims):
    print(json.dumps({
        "_aws": {
            "Timestamp": int(time.time() * 1000),
            "CloudWatchMetrics": [{
                "Namespace": "MyApp",
                "Dimensions": [list(dims.keys())],
                "Metrics": [{"Name": name, "Unit": unit}]
            }]
        },
        name: value,
        **dims,
    }))

# usage
emit_metric("PDFsProcessed", count, Unit="Count", Function="pdf-scanner")
```

## Prometheus & Grafana (brief)

Prometheus uses a **pull model** — it scrapes HTTP endpoints. Lambda functions are ephemeral and have no persistent HTTP endpoint, so Prometheus can't scrape them directly. Approaches:

- **EMF → CloudWatch → Grafana CloudWatch plugin** — easiest; Grafana queries CW as a data source
- **Amazon Managed Prometheus (AMP) + remote_write** — Lambda pushes metrics to AMP via the Prometheus remote write API; Grafana (or Amazon Managed Grafana) reads from AMP. Requires the `prometheus_client` library and SIGV4 signing on the remote_write request.
- **Statsd/push gateway** — Lambda pushes to a persistent push gateway; Prometheus scrapes the gateway. More infra to manage, stale metric risk if the push gateway isn't flushed between invocations.

For Lambda-centric dashboards, the CloudWatch → Grafana path is usually the simplest to operate.