412 lines
16 KiB
Markdown
412 lines
16 KiB
Markdown
# Tester Enhancement Design
|
|
|
|
## Problem Statement
|
|
|
|
Current tester filter UI "sucks" because:
|
|
1. **Code-centric filtering** - organizes by Python modules/classes, not user behavior
|
|
2. **No Gherkin integration** - can't filter by scenarios or features
|
|
3. **No pulse variables** - can't filter by:
|
|
- User roles (VET, USER/petowner, ADMIN)
|
|
- Flow stages (coverage check, service selection, payment, turno)
|
|
- Data states (has_pets, has_coverage, needs_payment)
|
|
- Service types, mock behaviors
|
|
4. **Clunky manual testing** - checkbox-based selection, not "piano playing" rapid execution
|
|
5. **Backend tests only** - no frontend (Playwright) test support
|
|
6. **No video captures** - critical for frontend test debugging
|
|
|
|
## Solution Overview
|
|
|
|
Transform tester into a **Gherkin-driven, behavior-first test execution platform** with:
|
|
|
|
### 1. Gherkin-First Organization
|
|
- Import/sync feature files from `album/book/gherkin-samples/`
|
|
- Parse scenarios and tags
|
|
- Map tests to Gherkin scenarios via metadata/decorators
|
|
- Filter by feature, scenario, tags (@smoke, @critical, @payment-flow)
|
|
|
|
### 2. Pulse Variables (Amar-specific filters)
|
|
Enable filtering by behavioral dimensions:
|
|
|
|
**User Context:**
|
|
- Role: VET, USER, ADMIN, GUEST
|
|
- State: new_user, returning_user, has_pets, has_coverage
|
|
|
|
**Flow Stage:**
|
|
- coverage_check, service_selection, cart, payment, turno_confirmation
|
|
|
|
**Service Type:**
|
|
- medical, grooming, vaccination, clinical
|
|
|
|
**Mock Behavior:**
|
|
- success, failure, timeout, partial_failure
|
|
|
|
**Environment:**
|
|
- local, demo, staging, production
|
|
|
|
### 3. Rapid Testing UX ("Piano Playing")
|
|
- **Quick filters** - one-click presets (e.g., "All payment tests", "Smoke tests")
|
|
- **Keyboard shortcuts** - run selected with Enter, navigate with arrows
|
|
- **Test chains** - define sequences to run in order
|
|
- **Session memory** - remember last filters and selections
|
|
- **Live search** - instant filter as you type
|
|
- **Batch actions** - run all visible, clear all, select by pattern
|
|
|
|
### 4. Frontend Test Support (Playwright)
|
|
- Detect and run `.spec.ts` tests via Playwright
|
|
- Capture video/screenshots automatically
|
|
- Display videos inline (like jira vein attachments)
|
|
- Attach artifacts to test results
|
|
|
|
### 5. Enhanced Test Results
|
|
```python
|
|
@dataclass
|
|
class TestResult:
|
|
test_id: str
|
|
name: str
|
|
status: TestStatus
|
|
duration: float
|
|
error_message: Optional[str] = None
|
|
traceback: Optional[str] = None
|
|
|
|
# NEW FIELDS
|
|
gherkin_feature: Optional[str] = None # "Reservar turno veterinario"
|
|
gherkin_scenario: Optional[str] = None # "Verificar cobertura en zona"
|
|
tags: list[str] = field(default_factory=list) # ["@smoke", "@coverage"]
|
|
artifacts: list[TestArtifact] = field(default_factory=list) # videos, screenshots
|
|
pulse_context: dict = field(default_factory=dict) # {role: "USER", stage: "coverage"}
|
|
|
|
@dataclass
|
|
class TestArtifact:
|
|
type: str # "video", "screenshot", "trace", "log"
|
|
filename: str
|
|
path: str
|
|
size: int
|
|
mimetype: str
|
|
url: str # streaming endpoint
|
|
```
|
|
|
|
## Architecture Changes
|
|
|
|
### Directory Structure
|
|
```
|
|
ward/tools/tester/
|
|
├── core.py # Test discovery/execution (existing)
|
|
├── api.py # FastAPI routes (existing)
|
|
├── config.py # Configuration (existing)
|
|
├── base.py # HTTP test base (existing)
|
|
├── gherkin/ # NEW - Gherkin integration
|
|
│ ├── parser.py # Parse .feature files
|
|
│ ├── mapper.py # Map tests to scenarios
|
|
│ └── sync.py # Sync from album/book
|
|
├── pulse/ # NEW - Pulse variable system
|
|
│ ├── context.py # Define pulse dimensions
|
|
│ ├── filters.py # Pulse-based filtering
|
|
│ └── presets.py # Quick filter presets
|
|
├── playwright/ # NEW - Frontend test support
|
|
│ ├── runner.py # Playwright test execution
|
|
│ ├── discovery.py # Find .spec.ts tests
|
|
│ └── artifacts.py # Handle videos/screenshots
|
|
├── templates/
|
|
│ ├── index.html # Runner UI (existing)
|
|
│ ├── filters.html # Filter UI (existing - needs redesign)
|
|
│ ├── filters_v2.html # NEW - Gherkin/pulse-based filters
|
|
│ └── artifacts.html # NEW - Video/screenshot viewer
|
|
├── tests/ # Synced backend tests (existing)
|
|
├── features/ # NEW - Synced Gherkin features
|
|
├── frontend-tests/ # NEW - Synced frontend tests
|
|
└── artifacts/ # NEW - Test artifacts storage
|
|
├── videos/
|
|
├── screenshots/
|
|
└── traces/
|
|
```
|
|
|
|
### Data Flow
|
|
|
|
**1. Test Discovery:**
|
|
```
|
|
Backend tests (pytest) → TestInfo
|
|
Frontend tests (playwright) → TestInfo
|
|
Gherkin features → FeatureInfo + ScenarioInfo
|
|
Map tests → scenarios via comments/decorators
|
|
```
|
|
|
|
**2. Filtering:**
|
|
```
|
|
User selects filters (UI)
|
|
↓
|
|
Filter by Gherkin (feature/scenario/tags)
|
|
↓
|
|
Filter by pulse variables (role/stage/state)
|
|
↓
|
|
Filter by test type (backend/frontend)
|
|
↓
|
|
Return filtered TestInfo list
|
|
```
|
|
|
|
**3. Execution:**
|
|
```
|
|
Start test run
|
|
↓
|
|
Backend tests: pytest runner (existing)
|
|
Frontend tests: Playwright runner (new)
|
|
↓
|
|
Collect artifacts (videos, screenshots)
|
|
↓
|
|
Store in artifacts/
|
|
↓
|
|
Return results with artifact URLs
|
|
```
|
|
|
|
**4. Results Display:**
|
|
```
|
|
Poll run status
|
|
↓
|
|
Show progress + current test
|
|
↓
|
|
Display results with:
|
|
- Status (pass/fail)
|
|
- Duration
|
|
- Error details
|
|
- Gherkin context
|
|
- Artifacts (inline videos)
|
|
```
|
|
|
|
## Implementation Plan
|
|
|
|
### Phase 1: Gherkin Integration
|
|
1. Create `gherkin/parser.py` - parse .feature files using `gherkin-python`
|
|
2. Create `gherkin/sync.py` - sync features from album/book
|
|
3. Enhance `TestInfo` with gherkin metadata
|
|
4. Add API endpoint `/api/features` to list features/scenarios
|
|
5. Update test discovery to extract Gherkin metadata from docstrings/comments
|
|
|
|
### Phase 2: Pulse Variables
|
|
1. Create `pulse/context.py` - define pulse dimensions (role, stage, state)
|
|
2. Create `pulse/filters.py` - filtering logic
|
|
3. Create `pulse/presets.py` - quick filter configurations
|
|
4. Enhance `TestInfo` with pulse context
|
|
5. Add API endpoints for pulse filtering
|
|
|
|
### Phase 3: Frontend Test Support
|
|
1. Create `playwright/discovery.py` - find .spec.ts tests
|
|
2. Create `playwright/runner.py` - execute Playwright tests
|
|
3. Create `playwright/artifacts.py` - collect videos/screenshots
|
|
4. Add artifact storage directory
|
|
5. Add API endpoint `/api/artifact/{run_id}/{artifact_id}` for streaming
|
|
6. Enhance `TestResult` with artifacts field
|
|
|
|
### Phase 4: Enhanced Filter UI
|
|
1. Design new filter layout (filters_v2.html)
|
|
2. Gherkin filter section (features, scenarios, tags)
|
|
3. Pulse filter section (role, stage, state, service, behavior)
|
|
4. Quick filter presets
|
|
5. Live search
|
|
6. Keyboard navigation
|
|
|
|
### Phase 5: Rapid Testing UX
|
|
1. Keyboard shortcuts
|
|
2. Test chains/sequences
|
|
3. Session persistence (localStorage)
|
|
4. Batch actions
|
|
5. One-click presets
|
|
6. Video artifact viewer
|
|
|
|
## Quick Filter Presets
|
|
|
|
```python
|
|
PRESETS = {
|
|
"smoke": {
|
|
"tags": ["@smoke"],
|
|
"description": "Critical smoke tests",
|
|
},
|
|
"payment_flow": {
|
|
"features": ["Pago de turno"],
|
|
"pulse": {"stage": "payment"},
|
|
"description": "All payment-related tests",
|
|
},
|
|
"coverage_check": {
|
|
"scenarios": ["Verificar cobertura"],
|
|
"pulse": {"stage": "coverage_check"},
|
|
"description": "Coverage verification tests",
|
|
},
|
|
"frontend_only": {
|
|
"test_type": "frontend",
|
|
"description": "All Playwright tests",
|
|
},
|
|
"vet_role": {
|
|
"pulse": {"role": "VET"},
|
|
"description": "Tests requiring VET user",
|
|
},
|
|
"turnero_complete": {
|
|
"features": ["Reservar turno"],
|
|
"test_type": "all",
|
|
"description": "Complete turnero flow (backend + frontend)",
|
|
},
|
|
}
|
|
```
|
|
|
|
## Gherkin Metadata in Tests
|
|
|
|
### Backend (pytest)
|
|
```python
|
|
class TestCoverageCheck(ContractHTTPTestCase):
|
|
"""
|
|
Feature: Reservar turno veterinario
|
|
Scenario: Verificar cobertura en zona disponible
|
|
Tags: @smoke @coverage
|
|
Pulse: role=GUEST, stage=coverage_check
|
|
"""
|
|
|
|
def test_coverage_returns_boolean(self):
|
|
"""When ingreso direccion 'Av Santa Fe 1234, CABA'"""
|
|
# test implementation
|
|
```
|
|
|
|
### Frontend (Playwright)
|
|
```typescript
|
|
/**
|
|
* Feature: Reservar turno veterinario
|
|
* Scenario: Verificar cobertura en zona disponible
|
|
* Tags: @smoke @coverage @frontend
|
|
* Pulse: role=GUEST, stage=coverage_check
|
|
*/
|
|
test('coverage check shows message for valid address', async ({ page }) => {
|
|
// test implementation
|
|
});
|
|
```
|
|
|
|
## Pulse Context Examples
|
|
|
|
```python
|
|
# Coverage check test
|
|
pulse_context = {
|
|
"role": "GUEST",
|
|
"stage": "coverage_check",
|
|
"state": "new_user",
|
|
"service_type": None,
|
|
"mock_behavior": "success",
|
|
}
|
|
|
|
# Payment test
|
|
pulse_context = {
|
|
"role": "USER",
|
|
"stage": "payment",
|
|
"state": "has_pets",
|
|
"service_type": "medical",
|
|
"mock_behavior": "success",
|
|
}
|
|
|
|
# VET acceptance test
|
|
pulse_context = {
|
|
"role": "VET",
|
|
"stage": "request_acceptance",
|
|
"state": "has_availability",
|
|
"service_type": "all",
|
|
"mock_behavior": "success",
|
|
}
|
|
```
|
|
|
|
## New Filter UI Design
|
|
|
|
### Layout
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Ward Tester - Gherkin-Driven Test Execution │
|
|
├─────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ [Quick Filters: Smoke | Payment | Coverage | Frontend] │
|
|
│ │
|
|
│ ┌─ Gherkin Filters ────────────────────────────────────┐ │
|
|
│ │ Features: [All ▼] Reservar turno Pago Historial │ │
|
|
│ │ Scenarios: [All ▼] Cobertura Servicios Contacto │ │
|
|
│ │ Tags: [@smoke] [@critical] [@payment-flow] │ │
|
|
│ └──────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ ┌─ Pulse Variables (Amar Context) ─────────────────────┐ │
|
|
│ │ Role: [All] VET USER ADMIN GUEST │ │
|
|
│ │ Stage: [All] coverage services cart payment │ │
|
|
│ │ State: [All] new has_pets has_coverage │ │
|
|
│ │ Service: [All] medical grooming vaccination │ │
|
|
│ │ Behavior: [All] success failure timeout │ │
|
|
│ └──────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ ┌─ Test Type ──────────────────────────────────────────┐ │
|
|
│ │ [All] Backend (HTTP) Frontend (Playwright) │ │
|
|
│ └──────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ Search: [________________________] 🔍 [Clear Filters] │
|
|
│ │
|
|
│ ┌─ Tests (24 of 156) ──────────────────────────────────┐ │
|
|
│ │ ☑ Verificar cobertura en zona disponible │ │
|
|
│ │ Feature: Reservar turno [@smoke @coverage] │ │
|
|
│ │ Backend + Frontend • Role: GUEST • Stage: cov │ │
|
|
│ │ │ │
|
|
│ │ ☑ Servicios filtrados por tipo de mascota │ │
|
|
│ │ Feature: Reservar turno [@smoke @services] │ │
|
|
│ │ Backend • Role: USER • Stage: services │ │
|
|
│ │ │ │
|
|
│ │ ... (more tests) │ │
|
|
│ └──────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ [▶ Run Selected (24)] [Select All] [Deselect All] │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### Keyboard Shortcuts
|
|
- `Enter` - Run selected tests
|
|
- `Ctrl+A` - Select all visible
|
|
- `Ctrl+D` - Deselect all
|
|
- `Ctrl+F` - Focus search
|
|
- `Ctrl+1-9` - Quick filter presets
|
|
- `Space` - Toggle test selection
|
|
- `↑/↓` - Navigate tests
|
|
|
|
## Video Artifact Display
|
|
|
|
When a frontend test completes with video:
|
|
|
|
```
|
|
┌─ Test Result: Verificar cobertura ─────────────────────┐
|
|
│ Status: ✓ PASSED │
|
|
│ Duration: 2.3s │
|
|
│ │
|
|
│ Artifacts: │
|
|
│ ┌────────────────────────────────────────────────────┐ │
|
|
│ │ 📹 coverage-check-chrome.webm (1.2 MB) │ │
|
|
│ │ [▶ Play inline] [Download] [Full screen] │ │
|
|
│ └────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ ┌────────────────────────────────────────────────────┐ │
|
|
│ │ 📸 screenshot-before.png (234 KB) │ │
|
|
│ │ [🖼 View] [Download] │ │
|
|
│ └────────────────────────────────────────────────────┘ │
|
|
└──────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
Inline video player (like jira vein):
|
|
```html
|
|
<video controls width="800">
|
|
<source src="/tools/tester/api/artifact/{run_id}/coverage-check.webm" type="video/webm">
|
|
</video>
|
|
```
|
|
|
|
## Benefits
|
|
|
|
1. **Behavior-first filtering** - think like a user, not a developer
|
|
2. **Rapid manual testing** - quickly run specific scenarios
|
|
3. **Better debugging** - video captures show exactly what happened
|
|
4. **Gherkin alignment** - tests map to documented behaviors
|
|
5. **Context-aware** - filter by the variables that matter (role, stage, state)
|
|
6. **Full coverage** - backend + frontend in one place
|
|
7. **Quick smoke tests** - one-click preset filters
|
|
8. **Better UX** - keyboard shortcuts, session memory, live search
|
|
|
|
## Next Steps
|
|
|
|
1. ✅ Design approved
|
|
2. Implement Phase 1 (Gherkin integration)
|
|
3. Implement Phase 2 (Pulse variables)
|
|
4. Implement Phase 3 (Frontend tests)
|
|
5. Implement Phase 4 (New filter UI)
|
|
6. Implement Phase 5 (Rapid testing UX)
|