soleprint/station/tools/tester/ENHANCEMENT_DESIGN.md

Tester Enhancement Design

Problem Statement

Current tester filter UI "sucks" because:

1. Code-centric filtering - organizes by Python modules/classes, not user behavior
2. No Gherkin integration - can't filter by scenarios or features
3. No pulse variables - can't filter by:
   • User roles (VET, USER/petowner, ADMIN)
   • Flow stages (coverage check, service selection, payment, turno)
   • Data states (has_pets, has_coverage, needs_payment)
   • Service types, mock behaviors
4. Clunky manual testing - checkbox-based selection, not "piano playing" rapid execution
5. Backend tests only - no frontend (Playwright) test support
6. No video captures - critical for frontend test debugging

Solution Overview

Transform tester into a Gherkin-driven, behavior-first test execution platform with:

1. Gherkin-First Organization

• Import/sync feature files from album/book/gherkin-samples/
• Parse scenarios and tags
• Map tests to Gherkin scenarios via metadata/decorators
• Filter by feature, scenario, tags (@smoke, @critical, @payment-flow)

2. Pulse Variables (Amar-specific filters)

Enable filtering by behavioral dimensions:

User Context:

• Role: VET, USER, ADMIN, GUEST
• State: new_user, returning_user, has_pets, has_coverage

Flow Stage:

• coverage_check, service_selection, cart, payment, turno_confirmation

Service Type:

• medical, grooming, vaccination, clinical

Mock Behavior:

• success, failure, timeout, partial_failure

Environment:

• local, demo, staging, production

3. Rapid Testing UX ("Piano Playing")

• Quick filters - one-click presets (e.g., "All payment tests", "Smoke tests")
• Keyboard shortcuts - run selected with Enter, navigate with arrows
• Test chains - define sequences to run in order
• Session memory - remember last filters and selections
• Live search - instant filter as you type
• Batch actions - run all visible, clear all, select by pattern

4. Frontend Test Support (Playwright)

• Detect and run .spec.ts tests via Playwright
• Capture video/screenshots automatically
• Display videos inline (like jira vein attachments)
• Attach artifacts to test results

5. Enhanced Test Results

@dataclass
class TestResult:
    test_id: str
    name: str
    status: TestStatus
    duration: float
    error_message: Optional[str] = None
    traceback: Optional[str] = None

    # NEW FIELDS
    gherkin_feature: Optional[str] = None      # "Reservar turno veterinario"
    gherkin_scenario: Optional[str] = None     # "Verificar cobertura en zona"
    tags: list[str] = field(default_factory=list)  # ["@smoke", "@coverage"]
    artifacts: list[TestArtifact] = field(default_factory=list)  # videos, screenshots
    pulse_context: dict = field(default_factory=dict)  # {role: "USER", stage: "coverage"}

@dataclass
class TestArtifact:
    type: str  # "video", "screenshot", "trace", "log"
    filename: str
    path: str
    size: int
    mimetype: str
    url: str  # streaming endpoint

Architecture Changes

Directory Structure

ward/tools/tester/
├── core.py                 # Test discovery/execution (existing)
├── api.py                  # FastAPI routes (existing)
├── config.py               # Configuration (existing)
├── base.py                 # HTTP test base (existing)
├── gherkin/                # NEW - Gherkin integration
│   ├── parser.py           # Parse .feature files
│   ├── mapper.py           # Map tests to scenarios
│   └── sync.py             # Sync from album/book
├── pulse/                  # NEW - Pulse variable system
│   ├── context.py          # Define pulse dimensions
│   ├── filters.py          # Pulse-based filtering
│   └── presets.py          # Quick filter presets
├── playwright/             # NEW - Frontend test support
│   ├── runner.py           # Playwright test execution
│   ├── discovery.py        # Find .spec.ts tests
│   └── artifacts.py        # Handle videos/screenshots
├── templates/
│   ├── index.html          # Runner UI (existing)
│   ├── filters.html        # Filter UI (existing - needs redesign)
│   ├── filters_v2.html     # NEW - Gherkin/pulse-based filters
│   └── artifacts.html      # NEW - Video/screenshot viewer
├── tests/                  # Synced backend tests (existing)
├── features/               # NEW - Synced Gherkin features
├── frontend-tests/         # NEW - Synced frontend tests
└── artifacts/              # NEW - Test artifacts storage
    ├── videos/
    ├── screenshots/
    └── traces/

Data Flow

1. Test Discovery:

Backend tests (pytest) → TestInfo
Frontend tests (playwright) → TestInfo
Gherkin features → FeatureInfo + ScenarioInfo
Map tests → scenarios via comments/decorators

2. Filtering:

User selects filters (UI)
  ↓
Filter by Gherkin (feature/scenario/tags)
  ↓
Filter by pulse variables (role/stage/state)
  ↓
Filter by test type (backend/frontend)
  ↓
Return filtered TestInfo list

3. Execution:

Start test run
  ↓
Backend tests: pytest runner (existing)
Frontend tests: Playwright runner (new)
  ↓
Collect artifacts (videos, screenshots)
  ↓
Store in artifacts/
  ↓
Return results with artifact URLs

4. Results Display:

Poll run status
  ↓
Show progress + current test
  ↓
Display results with:
  - Status (pass/fail)
  - Duration
  - Error details
  - Gherkin context
  - Artifacts (inline videos)

Implementation Plan

Phase 1: Gherkin Integration

1. Create gherkin/parser.py - parse .feature files using gherkin-python
2. Create gherkin/sync.py - sync features from album/book
3. Enhance TestInfo with gherkin metadata
4. Add API endpoint /api/features to list features/scenarios
5. Update test discovery to extract Gherkin metadata from docstrings/comments

Phase 2: Pulse Variables

1. Create pulse/context.py - define pulse dimensions (role, stage, state)
2. Create pulse/filters.py - filtering logic
3. Create pulse/presets.py - quick filter configurations
4. Enhance TestInfo with pulse context
5. Add API endpoints for pulse filtering

Phase 3: Frontend Test Support

1. Create playwright/discovery.py - find .spec.ts tests
2. Create playwright/runner.py - execute Playwright tests
3. Create playwright/artifacts.py - collect videos/screenshots
4. Add artifact storage directory
5. Add API endpoint /api/artifact/{run_id}/{artifact_id} for streaming
6. Enhance TestResult with artifacts field

Phase 4: Enhanced Filter UI

1. Design new filter layout (filters_v2.html)
2. Gherkin filter section (features, scenarios, tags)
3. Pulse filter section (role, stage, state, service, behavior)
4. Quick filter presets
5. Live search
6. Keyboard navigation

Phase 5: Rapid Testing UX

1. Keyboard shortcuts
2. Test chains/sequences
3. Session persistence (localStorage)
4. Batch actions
5. One-click presets
6. Video artifact viewer

Quick Filter Presets

PRESETS = {
    "smoke": {
        "tags": ["@smoke"],
        "description": "Critical smoke tests",
    },
    "payment_flow": {
        "features": ["Pago de turno"],
        "pulse": {"stage": "payment"},
        "description": "All payment-related tests",
    },
    "coverage_check": {
        "scenarios": ["Verificar cobertura"],
        "pulse": {"stage": "coverage_check"},
        "description": "Coverage verification tests",
    },
    "frontend_only": {
        "test_type": "frontend",
        "description": "All Playwright tests",
    },
    "vet_role": {
        "pulse": {"role": "VET"},
        "description": "Tests requiring VET user",
    },
    "turnero_complete": {
        "features": ["Reservar turno"],
        "test_type": "all",
        "description": "Complete turnero flow (backend + frontend)",
    },
}

Gherkin Metadata in Tests

Backend (pytest)

class TestCoverageCheck(ContractHTTPTestCase):
    """
    Feature: Reservar turno veterinario
    Scenario: Verificar cobertura en zona disponible
    Tags: @smoke @coverage
    Pulse: role=GUEST, stage=coverage_check
    """

    def test_coverage_returns_boolean(self):
        """When ingreso direccion 'Av Santa Fe 1234, CABA'"""
        # test implementation

Frontend (Playwright)

/**
 * Feature: Reservar turno veterinario
 * Scenario: Verificar cobertura en zona disponible
 * Tags: @smoke @coverage @frontend
 * Pulse: role=GUEST, stage=coverage_check
 */
test('coverage check shows message for valid address', async ({ page }) => {
  // test implementation
});

Pulse Context Examples

# Coverage check test
pulse_context = {
    "role": "GUEST",
    "stage": "coverage_check",
    "state": "new_user",
    "service_type": None,
    "mock_behavior": "success",
}

# Payment test
pulse_context = {
    "role": "USER",
    "stage": "payment",
    "state": "has_pets",
    "service_type": "medical",
    "mock_behavior": "success",
}

# VET acceptance test
pulse_context = {
    "role": "VET",
    "stage": "request_acceptance",
    "state": "has_availability",
    "service_type": "all",
    "mock_behavior": "success",
}

New Filter UI Design

Layout

┌─────────────────────────────────────────────────────────────┐
│ Ward Tester - Gherkin-Driven Test Execution                 │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│ [Quick Filters: Smoke | Payment | Coverage | Frontend]      │
│                                                              │
│ ┌─ Gherkin Filters ────────────────────────────────────┐   │
│ │ Features:   [All ▼] Reservar turno  Pago  Historial │   │
│ │ Scenarios:  [All ▼] Cobertura  Servicios  Contacto  │   │
│ │ Tags:       [@smoke] [@critical] [@payment-flow]    │   │
│ └──────────────────────────────────────────────────────┘   │
│                                                              │
│ ┌─ Pulse Variables (Amar Context) ─────────────────────┐   │
│ │ Role:     [All] VET  USER  ADMIN  GUEST              │   │
│ │ Stage:    [All] coverage  services  cart  payment    │   │
│ │ State:    [All] new  has_pets  has_coverage          │   │
│ │ Service:  [All] medical  grooming  vaccination       │   │
│ │ Behavior: [All] success  failure  timeout            │   │
│ └──────────────────────────────────────────────────────┘   │
│                                                              │
│ ┌─ Test Type ──────────────────────────────────────────┐   │
│ │ [All] Backend (HTTP)  Frontend (Playwright)          │   │
│ └──────────────────────────────────────────────────────┘   │
│                                                              │
│ Search: [________________________]  🔍  [Clear Filters]     │
│                                                              │
│ ┌─ Tests (24 of 156) ──────────────────────────────────┐   │
│ │ ☑ Verificar cobertura en zona disponible             │   │
│ │   Feature: Reservar turno  [@smoke @coverage]        │   │
│ │   Backend + Frontend  •  Role: GUEST  •  Stage: cov  │   │
│ │                                                        │   │
│ │ ☑ Servicios filtrados por tipo de mascota            │   │
│ │   Feature: Reservar turno  [@smoke @services]        │   │
│ │   Backend  •  Role: USER  •  Stage: services         │   │
│ │                                                        │   │
│ │ ... (more tests)                                      │   │
│ └──────────────────────────────────────────────────────┘   │
│                                                              │
│ [▶ Run Selected (24)]  [Select All]  [Deselect All]        │
└─────────────────────────────────────────────────────────────┘

Keyboard Shortcuts

• Enter - Run selected tests
• Ctrl+A - Select all visible
• Ctrl+D - Deselect all
• Ctrl+F - Focus search
• Ctrl+1-9 - Quick filter presets
• Space - Toggle test selection
• ↑/↓ - Navigate tests

Video Artifact Display

When a frontend test completes with video:

┌─ Test Result: Verificar cobertura ─────────────────────┐
│ Status: ✓ PASSED                                        │
│ Duration: 2.3s                                          │
│                                                          │
│ Artifacts:                                              │
│ ┌────────────────────────────────────────────────────┐ │
│ │ 📹 coverage-check-chrome.webm (1.2 MB)             │ │
│ │ [▶ Play inline]  [Download]  [Full screen]        │ │
│ └────────────────────────────────────────────────────┘ │
│                                                          │
│ ┌────────────────────────────────────────────────────┐ │
│ │ 📸 screenshot-before.png (234 KB)                  │ │
│ │ [🖼 View]  [Download]                              │ │
│ └────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────┘

Inline video player (like jira vein):

<video controls width="800">
  <source src="/tools/tester/api/artifact/{run_id}/coverage-check.webm" type="video/webm">
</video>

Benefits

1. Behavior-first filtering - think like a user, not a developer
2. Rapid manual testing - quickly run specific scenarios
3. Better debugging - video captures show exactly what happened
4. Gherkin alignment - tests map to documented behaviors
5. Context-aware - filter by the variables that matter (role, stage, state)
6. Full coverage - backend + frontend in one place
7. Quick smoke tests - one-click preset filters
8. Better UX - keyboard shortcuts, session memory, live search

Next Steps

1. ✅ Design approved
2. Implement Phase 1 (Gherkin integration)
3. Implement Phase 2 (Pulse variables)
4. Implement Phase 3 (Frontend tests)
5. Implement Phase 4 (New filter UI)
6. Implement Phase 5 (Rapid testing UX)