soleprint init commit
This commit is contained in:
buenosairesam
411
station/tools/tester/ENHANCEMENT_DESIGN.md
Normal file
411
station/tools/tester/ENHANCEMENT_DESIGN.md
Normal file
@@ -0,0 +1,411 @@
|
||||
# Tester Enhancement Design
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Current tester filter UI "sucks" because:
|
||||
1. **Code-centric filtering** - organizes by Python modules/classes, not user behavior
|
||||
2. **No Gherkin integration** - can't filter by scenarios or features
|
||||
3. **No pulse variables** - can't filter by:
|
||||
- User roles (VET, USER/petowner, ADMIN)
|
||||
- Flow stages (coverage check, service selection, payment, turno)
|
||||
- Data states (has_pets, has_coverage, needs_payment)
|
||||
- Service types, mock behaviors
|
||||
4. **Clunky manual testing** - checkbox-based selection, not "piano playing" rapid execution
|
||||
5. **Backend tests only** - no frontend (Playwright) test support
|
||||
6. **No video captures** - critical for frontend test debugging
|
||||
|
||||
## Solution Overview
|
||||
|
||||
Transform tester into a **Gherkin-driven, behavior-first test execution platform** with:
|
||||
|
||||
### 1. Gherkin-First Organization
|
||||
- Import/sync feature files from `album/book/gherkin-samples/`
|
||||
- Parse scenarios and tags
|
||||
- Map tests to Gherkin scenarios via metadata/decorators
|
||||
- Filter by feature, scenario, tags (@smoke, @critical, @payment-flow)
|
||||
|
||||
### 2. Pulse Variables (Amar-specific filters)
|
||||
Enable filtering by behavioral dimensions:
|
||||
|
||||
**User Context:**
|
||||
- Role: VET, USER, ADMIN, GUEST
|
||||
- State: new_user, returning_user, has_pets, has_coverage
|
||||
|
||||
**Flow Stage:**
|
||||
- coverage_check, service_selection, cart, payment, turno_confirmation
|
||||
|
||||
**Service Type:**
|
||||
- medical, grooming, vaccination, clinical
|
||||
|
||||
**Mock Behavior:**
|
||||
- success, failure, timeout, partial_failure
|
||||
|
||||
**Environment:**
|
||||
- local, demo, staging, production
|
||||
|
||||
### 3. Rapid Testing UX ("Piano Playing")
|
||||
- **Quick filters** - one-click presets (e.g., "All payment tests", "Smoke tests")
|
||||
- **Keyboard shortcuts** - run selected with Enter, navigate with arrows
|
||||
- **Test chains** - define sequences to run in order
|
||||
- **Session memory** - remember last filters and selections
|
||||
- **Live search** - instant filter as you type
|
||||
- **Batch actions** - run all visible, clear all, select by pattern
|
||||
|
||||
### 4. Frontend Test Support (Playwright)
|
||||
- Detect and run `.spec.ts` tests via Playwright
|
||||
- Capture video/screenshots automatically
|
||||
- Display videos inline (like jira vein attachments)
|
||||
- Attach artifacts to test results
|
||||
|
||||
### 5. Enhanced Test Results
|
||||
```python
|
||||
@dataclass
|
||||
class TestResult:
|
||||
test_id: str
|
||||
name: str
|
||||
status: TestStatus
|
||||
duration: float
|
||||
error_message: Optional[str] = None
|
||||
traceback: Optional[str] = None
|
||||
|
||||
# NEW FIELDS
|
||||
gherkin_feature: Optional[str] = None # "Reservar turno veterinario"
|
||||
gherkin_scenario: Optional[str] = None # "Verificar cobertura en zona"
|
||||
tags: list[str] = field(default_factory=list) # ["@smoke", "@coverage"]
|
||||
artifacts: list[TestArtifact] = field(default_factory=list) # videos, screenshots
|
||||
pulse_context: dict = field(default_factory=dict) # {role: "USER", stage: "coverage"}
|
||||
|
||||
@dataclass
|
||||
class TestArtifact:
|
||||
type: str # "video", "screenshot", "trace", "log"
|
||||
filename: str
|
||||
path: str
|
||||
size: int
|
||||
mimetype: str
|
||||
url: str # streaming endpoint
|
||||
```
|
||||
|
||||
## Architecture Changes
|
||||
|
||||
### Directory Structure
|
||||
```
|
||||
ward/tools/tester/
|
||||
├── core.py # Test discovery/execution (existing)
|
||||
├── api.py # FastAPI routes (existing)
|
||||
├── config.py # Configuration (existing)
|
||||
├── base.py # HTTP test base (existing)
|
||||
├── gherkin/ # NEW - Gherkin integration
|
||||
│ ├── parser.py # Parse .feature files
|
||||
│ ├── mapper.py # Map tests to scenarios
|
||||
│ └── sync.py # Sync from album/book
|
||||
├── pulse/ # NEW - Pulse variable system
|
||||
│ ├── context.py # Define pulse dimensions
|
||||
│ ├── filters.py # Pulse-based filtering
|
||||
│ └── presets.py # Quick filter presets
|
||||
├── playwright/ # NEW - Frontend test support
|
||||
│ ├── runner.py # Playwright test execution
|
||||
│ ├── discovery.py # Find .spec.ts tests
|
||||
│ └── artifacts.py # Handle videos/screenshots
|
||||
├── templates/
|
||||
│ ├── index.html # Runner UI (existing)
|
||||
│ ├── filters.html # Filter UI (existing - needs redesign)
|
||||
│ ├── filters_v2.html # NEW - Gherkin/pulse-based filters
|
||||
│ └── artifacts.html # NEW - Video/screenshot viewer
|
||||
├── tests/ # Synced backend tests (existing)
|
||||
├── features/ # NEW - Synced Gherkin features
|
||||
├── frontend-tests/ # NEW - Synced frontend tests
|
||||
└── artifacts/ # NEW - Test artifacts storage
|
||||
├── videos/
|
||||
├── screenshots/
|
||||
└── traces/
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
**1. Test Discovery:**
|
||||
```
|
||||
Backend tests (pytest) → TestInfo
|
||||
Frontend tests (playwright) → TestInfo
|
||||
Gherkin features → FeatureInfo + ScenarioInfo
|
||||
Map tests → scenarios via comments/decorators
|
||||
```
|
||||
|
||||
**2. Filtering:**
|
||||
```
|
||||
User selects filters (UI)
|
||||
↓
|
||||
Filter by Gherkin (feature/scenario/tags)
|
||||
↓
|
||||
Filter by pulse variables (role/stage/state)
|
||||
↓
|
||||
Filter by test type (backend/frontend)
|
||||
↓
|
||||
Return filtered TestInfo list
|
||||
```
|
||||
|
||||
**3. Execution:**
|
||||
```
|
||||
Start test run
|
||||
↓
|
||||
Backend tests: pytest runner (existing)
|
||||
Frontend tests: Playwright runner (new)
|
||||
↓
|
||||
Collect artifacts (videos, screenshots)
|
||||
↓
|
||||
Store in artifacts/
|
||||
↓
|
||||
Return results with artifact URLs
|
||||
```
|
||||
|
||||
**4. Results Display:**
|
||||
```
|
||||
Poll run status
|
||||
↓
|
||||
Show progress + current test
|
||||
↓
|
||||
Display results with:
|
||||
- Status (pass/fail)
|
||||
- Duration
|
||||
- Error details
|
||||
- Gherkin context
|
||||
- Artifacts (inline videos)
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Gherkin Integration
|
||||
1. Create `gherkin/parser.py` - parse .feature files using `gherkin-python`
|
||||
2. Create `gherkin/sync.py` - sync features from album/book
|
||||
3. Enhance `TestInfo` with gherkin metadata
|
||||
4. Add API endpoint `/api/features` to list features/scenarios
|
||||
5. Update test discovery to extract Gherkin metadata from docstrings/comments
|
||||
|
||||
### Phase 2: Pulse Variables
|
||||
1. Create `pulse/context.py` - define pulse dimensions (role, stage, state)
|
||||
2. Create `pulse/filters.py` - filtering logic
|
||||
3. Create `pulse/presets.py` - quick filter configurations
|
||||
4. Enhance `TestInfo` with pulse context
|
||||
5. Add API endpoints for pulse filtering
|
||||
|
||||
### Phase 3: Frontend Test Support
|
||||
1. Create `playwright/discovery.py` - find .spec.ts tests
|
||||
2. Create `playwright/runner.py` - execute Playwright tests
|
||||
3. Create `playwright/artifacts.py` - collect videos/screenshots
|
||||
4. Add artifact storage directory
|
||||
5. Add API endpoint `/api/artifact/{run_id}/{artifact_id}` for streaming
|
||||
6. Enhance `TestResult` with artifacts field
|
||||
|
||||
### Phase 4: Enhanced Filter UI
|
||||
1. Design new filter layout (filters_v2.html)
|
||||
2. Gherkin filter section (features, scenarios, tags)
|
||||
3. Pulse filter section (role, stage, state, service, behavior)
|
||||
4. Quick filter presets
|
||||
5. Live search
|
||||
6. Keyboard navigation
|
||||
|
||||
### Phase 5: Rapid Testing UX
|
||||
1. Keyboard shortcuts
|
||||
2. Test chains/sequences
|
||||
3. Session persistence (localStorage)
|
||||
4. Batch actions
|
||||
5. One-click presets
|
||||
6. Video artifact viewer
|
||||
|
||||
## Quick Filter Presets
|
||||
|
||||
```python
|
||||
PRESETS = {
|
||||
"smoke": {
|
||||
"tags": ["@smoke"],
|
||||
"description": "Critical smoke tests",
|
||||
},
|
||||
"payment_flow": {
|
||||
"features": ["Pago de turno"],
|
||||
"pulse": {"stage": "payment"},
|
||||
"description": "All payment-related tests",
|
||||
},
|
||||
"coverage_check": {
|
||||
"scenarios": ["Verificar cobertura"],
|
||||
"pulse": {"stage": "coverage_check"},
|
||||
"description": "Coverage verification tests",
|
||||
},
|
||||
"frontend_only": {
|
||||
"test_type": "frontend",
|
||||
"description": "All Playwright tests",
|
||||
},
|
||||
"vet_role": {
|
||||
"pulse": {"role": "VET"},
|
||||
"description": "Tests requiring VET user",
|
||||
},
|
||||
"turnero_complete": {
|
||||
"features": ["Reservar turno"],
|
||||
"test_type": "all",
|
||||
"description": "Complete turnero flow (backend + frontend)",
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Gherkin Metadata in Tests
|
||||
|
||||
### Backend (pytest)
|
||||
```python
|
||||
class TestCoverageCheck(ContractHTTPTestCase):
|
||||
"""
|
||||
Feature: Reservar turno veterinario
|
||||
Scenario: Verificar cobertura en zona disponible
|
||||
Tags: @smoke @coverage
|
||||
Pulse: role=GUEST, stage=coverage_check
|
||||
"""
|
||||
|
||||
def test_coverage_returns_boolean(self):
|
||||
"""When ingreso direccion 'Av Santa Fe 1234, CABA'"""
|
||||
# test implementation
|
||||
```
|
||||
|
||||
### Frontend (Playwright)
|
||||
```typescript
|
||||
/**
|
||||
* Feature: Reservar turno veterinario
|
||||
* Scenario: Verificar cobertura en zona disponible
|
||||
* Tags: @smoke @coverage @frontend
|
||||
* Pulse: role=GUEST, stage=coverage_check
|
||||
*/
|
||||
test('coverage check shows message for valid address', async ({ page }) => {
|
||||
// test implementation
|
||||
});
|
||||
```
|
||||
|
||||
## Pulse Context Examples
|
||||
|
||||
```python
|
||||
# Coverage check test
|
||||
pulse_context = {
|
||||
"role": "GUEST",
|
||||
"stage": "coverage_check",
|
||||
"state": "new_user",
|
||||
"service_type": None,
|
||||
"mock_behavior": "success",
|
||||
}
|
||||
|
||||
# Payment test
|
||||
pulse_context = {
|
||||
"role": "USER",
|
||||
"stage": "payment",
|
||||
"state": "has_pets",
|
||||
"service_type": "medical",
|
||||
"mock_behavior": "success",
|
||||
}
|
||||
|
||||
# VET acceptance test
|
||||
pulse_context = {
|
||||
"role": "VET",
|
||||
"stage": "request_acceptance",
|
||||
"state": "has_availability",
|
||||
"service_type": "all",
|
||||
"mock_behavior": "success",
|
||||
}
|
||||
```
|
||||
|
||||
## New Filter UI Design
|
||||
|
||||
### Layout
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Ward Tester - Gherkin-Driven Test Execution │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ [Quick Filters: Smoke | Payment | Coverage | Frontend] │
|
||||
│ │
|
||||
│ ┌─ Gherkin Filters ────────────────────────────────────┐ │
|
||||
│ │ Features: [All ▼] Reservar turno Pago Historial │ │
|
||||
│ │ Scenarios: [All ▼] Cobertura Servicios Contacto │ │
|
||||
│ │ Tags: [@smoke] [@critical] [@payment-flow] │ │
|
||||
│ └──────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─ Pulse Variables (Amar Context) ─────────────────────┐ │
|
||||
│ │ Role: [All] VET USER ADMIN GUEST │ │
|
||||
│ │ Stage: [All] coverage services cart payment │ │
|
||||
│ │ State: [All] new has_pets has_coverage │ │
|
||||
│ │ Service: [All] medical grooming vaccination │ │
|
||||
│ │ Behavior: [All] success failure timeout │ │
|
||||
│ └──────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─ Test Type ──────────────────────────────────────────┐ │
|
||||
│ │ [All] Backend (HTTP) Frontend (Playwright) │ │
|
||||
│ └──────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Search: [________________________] 🔍 [Clear Filters] │
|
||||
│ │
|
||||
│ ┌─ Tests (24 of 156) ──────────────────────────────────┐ │
|
||||
│ │ ☑ Verificar cobertura en zona disponible │ │
|
||||
│ │ Feature: Reservar turno [@smoke @coverage] │ │
|
||||
│ │ Backend + Frontend • Role: GUEST • Stage: cov │ │
|
||||
│ │ │ │
|
||||
│ │ ☑ Servicios filtrados por tipo de mascota │ │
|
||||
│ │ Feature: Reservar turno [@smoke @services] │ │
|
||||
│ │ Backend • Role: USER • Stage: services │ │
|
||||
│ │ │ │
|
||||
│ │ ... (more tests) │ │
|
||||
│ └──────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ [▶ Run Selected (24)] [Select All] [Deselect All] │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Keyboard Shortcuts
|
||||
- `Enter` - Run selected tests
|
||||
- `Ctrl+A` - Select all visible
|
||||
- `Ctrl+D` - Deselect all
|
||||
- `Ctrl+F` - Focus search
|
||||
- `Ctrl+1-9` - Quick filter presets
|
||||
- `Space` - Toggle test selection
|
||||
- `↑/↓` - Navigate tests
|
||||
|
||||
## Video Artifact Display
|
||||
|
||||
When a frontend test completes with video:
|
||||
|
||||
```
|
||||
┌─ Test Result: Verificar cobertura ─────────────────────┐
|
||||
│ Status: ✓ PASSED │
|
||||
│ Duration: 2.3s │
|
||||
│ │
|
||||
│ Artifacts: │
|
||||
│ ┌────────────────────────────────────────────────────┐ │
|
||||
│ │ 📹 coverage-check-chrome.webm (1.2 MB) │ │
|
||||
│ │ [▶ Play inline] [Download] [Full screen] │ │
|
||||
│ └────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────────────────────┐ │
|
||||
│ │ 📸 screenshot-before.png (234 KB) │ │
|
||||
│ │ [🖼 View] [Download] │ │
|
||||
│ └────────────────────────────────────────────────────┘ │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Inline video player (like jira vein):
|
||||
```html
|
||||
<video controls width="800">
|
||||
<source src="/tools/tester/api/artifact/{run_id}/coverage-check.webm" type="video/webm">
|
||||
</video>
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Behavior-first filtering** - think like a user, not a developer
|
||||
2. **Rapid manual testing** - quickly run specific scenarios
|
||||
3. **Better debugging** - video captures show exactly what happened
|
||||
4. **Gherkin alignment** - tests map to documented behaviors
|
||||
5. **Context-aware** - filter by the variables that matter (role, stage, state)
|
||||
6. **Full coverage** - backend + frontend in one place
|
||||
7. **Quick smoke tests** - one-click preset filters
|
||||
8. **Better UX** - keyboard shortcuts, session memory, live search
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ✅ Design approved
|
||||
2. Implement Phase 1 (Gherkin integration)
|
||||
3. Implement Phase 2 (Pulse variables)
|
||||
4. Implement Phase 3 (Frontend tests)
|
||||
5. Implement Phase 4 (New filter UI)
|
||||
6. Implement Phase 5 (Rapid testing UX)
|
||||
Reference in New Issue
Block a user