# Tester Enhancement Design ## Problem Statement Current tester filter UI "sucks" because: 1. **Code-centric filtering** - organizes by Python modules/classes, not user behavior 2. **No Gherkin integration** - can't filter by scenarios or features 3. **No pulse variables** - can't filter by: - User roles (VET, USER/petowner, ADMIN) - Flow stages (coverage check, service selection, payment, turno) - Data states (has_pets, has_coverage, needs_payment) - Service types, mock behaviors 4. **Clunky manual testing** - checkbox-based selection, not "piano playing" rapid execution 5. **Backend tests only** - no frontend (Playwright) test support 6. **No video captures** - critical for frontend test debugging ## Solution Overview Transform tester into a **Gherkin-driven, behavior-first test execution platform** with: ### 1. Gherkin-First Organization - Import/sync feature files from `album/book/gherkin-samples/` - Parse scenarios and tags - Map tests to Gherkin scenarios via metadata/decorators - Filter by feature, scenario, tags (@smoke, @critical, @payment-flow) ### 2. Pulse Variables (Amar-specific filters) Enable filtering by behavioral dimensions: **User Context:** - Role: VET, USER, ADMIN, GUEST - State: new_user, returning_user, has_pets, has_coverage **Flow Stage:** - coverage_check, service_selection, cart, payment, turno_confirmation **Service Type:** - medical, grooming, vaccination, clinical **Mock Behavior:** - success, failure, timeout, partial_failure **Environment:** - local, demo, staging, production ### 3. Rapid Testing UX ("Piano Playing") - **Quick filters** - one-click presets (e.g., "All payment tests", "Smoke tests") - **Keyboard shortcuts** - run selected with Enter, navigate with arrows - **Test chains** - define sequences to run in order - **Session memory** - remember last filters and selections - **Live search** - instant filter as you type - **Batch actions** - run all visible, clear all, select by pattern ### 4. Frontend Test Support (Playwright) - Detect and run `.spec.ts` tests via Playwright - Capture video/screenshots automatically - Display videos inline (like jira vein attachments) - Attach artifacts to test results ### 5. Enhanced Test Results ```python @dataclass class TestResult: test_id: str name: str status: TestStatus duration: float error_message: Optional[str] = None traceback: Optional[str] = None # NEW FIELDS gherkin_feature: Optional[str] = None # "Reservar turno veterinario" gherkin_scenario: Optional[str] = None # "Verificar cobertura en zona" tags: list[str] = field(default_factory=list) # ["@smoke", "@coverage"] artifacts: list[TestArtifact] = field(default_factory=list) # videos, screenshots pulse_context: dict = field(default_factory=dict) # {role: "USER", stage: "coverage"} @dataclass class TestArtifact: type: str # "video", "screenshot", "trace", "log" filename: str path: str size: int mimetype: str url: str # streaming endpoint ``` ## Architecture Changes ### Directory Structure ``` ward/tools/tester/ ├── core.py # Test discovery/execution (existing) ├── api.py # FastAPI routes (existing) ├── config.py # Configuration (existing) ├── base.py # HTTP test base (existing) ├── gherkin/ # NEW - Gherkin integration │ ├── parser.py # Parse .feature files │ ├── mapper.py # Map tests to scenarios │ └── sync.py # Sync from album/book ├── pulse/ # NEW - Pulse variable system │ ├── context.py # Define pulse dimensions │ ├── filters.py # Pulse-based filtering │ └── presets.py # Quick filter presets ├── playwright/ # NEW - Frontend test support │ ├── runner.py # Playwright test execution │ ├── discovery.py # Find .spec.ts tests │ └── artifacts.py # Handle videos/screenshots ├── templates/ │ ├── index.html # Runner UI (existing) │ ├── filters.html # Filter UI (existing - needs redesign) │ ├── filters_v2.html # NEW - Gherkin/pulse-based filters │ └── artifacts.html # NEW - Video/screenshot viewer ├── tests/ # Synced backend tests (existing) ├── features/ # NEW - Synced Gherkin features ├── frontend-tests/ # NEW - Synced frontend tests └── artifacts/ # NEW - Test artifacts storage ├── videos/ ├── screenshots/ └── traces/ ``` ### Data Flow **1. Test Discovery:** ``` Backend tests (pytest) → TestInfo Frontend tests (playwright) → TestInfo Gherkin features → FeatureInfo + ScenarioInfo Map tests → scenarios via comments/decorators ``` **2. Filtering:** ``` User selects filters (UI) ↓ Filter by Gherkin (feature/scenario/tags) ↓ Filter by pulse variables (role/stage/state) ↓ Filter by test type (backend/frontend) ↓ Return filtered TestInfo list ``` **3. Execution:** ``` Start test run ↓ Backend tests: pytest runner (existing) Frontend tests: Playwright runner (new) ↓ Collect artifacts (videos, screenshots) ↓ Store in artifacts/ ↓ Return results with artifact URLs ``` **4. Results Display:** ``` Poll run status ↓ Show progress + current test ↓ Display results with: - Status (pass/fail) - Duration - Error details - Gherkin context - Artifacts (inline videos) ``` ## Implementation Plan ### Phase 1: Gherkin Integration 1. Create `gherkin/parser.py` - parse .feature files using `gherkin-python` 2. Create `gherkin/sync.py` - sync features from album/book 3. Enhance `TestInfo` with gherkin metadata 4. Add API endpoint `/api/features` to list features/scenarios 5. Update test discovery to extract Gherkin metadata from docstrings/comments ### Phase 2: Pulse Variables 1. Create `pulse/context.py` - define pulse dimensions (role, stage, state) 2. Create `pulse/filters.py` - filtering logic 3. Create `pulse/presets.py` - quick filter configurations 4. Enhance `TestInfo` with pulse context 5. Add API endpoints for pulse filtering ### Phase 3: Frontend Test Support 1. Create `playwright/discovery.py` - find .spec.ts tests 2. Create `playwright/runner.py` - execute Playwright tests 3. Create `playwright/artifacts.py` - collect videos/screenshots 4. Add artifact storage directory 5. Add API endpoint `/api/artifact/{run_id}/{artifact_id}` for streaming 6. Enhance `TestResult` with artifacts field ### Phase 4: Enhanced Filter UI 1. Design new filter layout (filters_v2.html) 2. Gherkin filter section (features, scenarios, tags) 3. Pulse filter section (role, stage, state, service, behavior) 4. Quick filter presets 5. Live search 6. Keyboard navigation ### Phase 5: Rapid Testing UX 1. Keyboard shortcuts 2. Test chains/sequences 3. Session persistence (localStorage) 4. Batch actions 5. One-click presets 6. Video artifact viewer ## Quick Filter Presets ```python PRESETS = { "smoke": { "tags": ["@smoke"], "description": "Critical smoke tests", }, "payment_flow": { "features": ["Pago de turno"], "pulse": {"stage": "payment"}, "description": "All payment-related tests", }, "coverage_check": { "scenarios": ["Verificar cobertura"], "pulse": {"stage": "coverage_check"}, "description": "Coverage verification tests", }, "frontend_only": { "test_type": "frontend", "description": "All Playwright tests", }, "vet_role": { "pulse": {"role": "VET"}, "description": "Tests requiring VET user", }, "turnero_complete": { "features": ["Reservar turno"], "test_type": "all", "description": "Complete turnero flow (backend + frontend)", }, } ``` ## Gherkin Metadata in Tests ### Backend (pytest) ```python class TestCoverageCheck(ContractHTTPTestCase): """ Feature: Reservar turno veterinario Scenario: Verificar cobertura en zona disponible Tags: @smoke @coverage Pulse: role=GUEST, stage=coverage_check """ def test_coverage_returns_boolean(self): """When ingreso direccion 'Av Santa Fe 1234, CABA'""" # test implementation ``` ### Frontend (Playwright) ```typescript /** * Feature: Reservar turno veterinario * Scenario: Verificar cobertura en zona disponible * Tags: @smoke @coverage @frontend * Pulse: role=GUEST, stage=coverage_check */ test('coverage check shows message for valid address', async ({ page }) => { // test implementation }); ``` ## Pulse Context Examples ```python # Coverage check test pulse_context = { "role": "GUEST", "stage": "coverage_check", "state": "new_user", "service_type": None, "mock_behavior": "success", } # Payment test pulse_context = { "role": "USER", "stage": "payment", "state": "has_pets", "service_type": "medical", "mock_behavior": "success", } # VET acceptance test pulse_context = { "role": "VET", "stage": "request_acceptance", "state": "has_availability", "service_type": "all", "mock_behavior": "success", } ``` ## New Filter UI Design ### Layout ``` ┌─────────────────────────────────────────────────────────────┐ │ Ward Tester - Gherkin-Driven Test Execution │ ├─────────────────────────────────────────────────────────────┤ │ │ │ [Quick Filters: Smoke | Payment | Coverage | Frontend] │ │ │ │ ┌─ Gherkin Filters ────────────────────────────────────┐ │ │ │ Features: [All ▼] Reservar turno Pago Historial │ │ │ │ Scenarios: [All ▼] Cobertura Servicios Contacto │ │ │ │ Tags: [@smoke] [@critical] [@payment-flow] │ │ │ └──────────────────────────────────────────────────────┘ │ │ │ │ ┌─ Pulse Variables (Amar Context) ─────────────────────┐ │ │ │ Role: [All] VET USER ADMIN GUEST │ │ │ │ Stage: [All] coverage services cart payment │ │ │ │ State: [All] new has_pets has_coverage │ │ │ │ Service: [All] medical grooming vaccination │ │ │ │ Behavior: [All] success failure timeout │ │ │ └──────────────────────────────────────────────────────┘ │ │ │ │ ┌─ Test Type ──────────────────────────────────────────┐ │ │ │ [All] Backend (HTTP) Frontend (Playwright) │ │ │ └──────────────────────────────────────────────────────┘ │ │ │ │ Search: [________________________] 🔍 [Clear Filters] │ │ │ │ ┌─ Tests (24 of 156) ──────────────────────────────────┐ │ │ │ ☑ Verificar cobertura en zona disponible │ │ │ │ Feature: Reservar turno [@smoke @coverage] │ │ │ │ Backend + Frontend • Role: GUEST • Stage: cov │ │ │ │ │ │ │ │ ☑ Servicios filtrados por tipo de mascota │ │ │ │ Feature: Reservar turno [@smoke @services] │ │ │ │ Backend • Role: USER • Stage: services │ │ │ │ │ │ │ │ ... (more tests) │ │ │ └──────────────────────────────────────────────────────┘ │ │ │ │ [▶ Run Selected (24)] [Select All] [Deselect All] │ └─────────────────────────────────────────────────────────────┘ ``` ### Keyboard Shortcuts - `Enter` - Run selected tests - `Ctrl+A` - Select all visible - `Ctrl+D` - Deselect all - `Ctrl+F` - Focus search - `Ctrl+1-9` - Quick filter presets - `Space` - Toggle test selection - `↑/↓` - Navigate tests ## Video Artifact Display When a frontend test completes with video: ``` ┌─ Test Result: Verificar cobertura ─────────────────────┐ │ Status: ✓ PASSED │ │ Duration: 2.3s │ │ │ │ Artifacts: │ │ ┌────────────────────────────────────────────────────┐ │ │ │ 📹 coverage-check-chrome.webm (1.2 MB) │ │ │ │ [▶ Play inline] [Download] [Full screen] │ │ │ └────────────────────────────────────────────────────┘ │ │ │ │ ┌────────────────────────────────────────────────────┐ │ │ │ 📸 screenshot-before.png (234 KB) │ │ │ │ [🖼 View] [Download] │ │ │ └────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────┘ ``` Inline video player (like jira vein): ```html ``` ## Benefits 1. **Behavior-first filtering** - think like a user, not a developer 2. **Rapid manual testing** - quickly run specific scenarios 3. **Better debugging** - video captures show exactly what happened 4. **Gherkin alignment** - tests map to documented behaviors 5. **Context-aware** - filter by the variables that matter (role, stage, state) 6. **Full coverage** - backend + frontend in one place 7. **Quick smoke tests** - one-click preset filters 8. **Better UX** - keyboard shortcuts, session memory, live search ## Next Steps 1. ✅ Design approved 2. Implement Phase 1 (Gherkin integration) 3. Implement Phase 2 (Pulse variables) 4. Implement Phase 3 (Frontend tests) 5. Implement Phase 4 (New filter UI) 6. Implement Phase 5 (Rapid testing UX)