# Welvet · AAI · Tide — volatile online farm experiment

This note explains the stack, what the cloud soak was for, what the reports contain, and where the deep-research writeups plug in. It is the human-readable companion to the PDFs in `reports/` and the Welvet Feature Book (v1.1.1).

---

## 1. One-sentence pitch

**Welvet** is a Go AI engine with honest dtypes/quants and many training paths; **Lucy** measures whether a net can **learn while it still serves**; **Tide** runs huge live permute sweeps of those choices; **Ocean** consolidates many Tide peers; this experiment soaked **three cloud VMs at ~100% CPU for days/weeks** to see **stability and how far the tree got** — not to finish every cell.

---

## 2. Stack map

```
┌─────────────────────────────────────────────────────────────┐
│  Ocean (optional master)                                    │
│  polls many Tide boards · consolidates winners · master PDF │
└────────────────────────────▲────────────────────────────────┘
                             │ /api/board, /api/report
┌────────────────────────────┴────────────────────────────────┐
│  Tide (host: test51, live_mnist, layer sprint, …)           │
│  serve + train pulses · permute matrix · dash · checkpoints │
└────────────────────────────▲────────────────────────────────┘
                             │ metrics + TrainModes + cams
┌────────────────────────────┴────────────────────────────────┐
│  Welvet engine (+ Lucy measuring math)                      │
│  layers · 34 dtypes · 20 quants · SIMD/WebGPU · TrainModes  │
└────────────────────────────▲────────────────────────────────┘
                             │
┌────────────────────────────┴────────────────────────────────┐
│  AAI (private race tree)                                    │
│  Lucy benches (test41/48/50…), cameral sandwiches, races    │
│  consumes public Welvet; Tide re-exports Lucy via metrics   │
└─────────────────────────────────────────────────────────────┘
```

| Layer | Role |
|-------|------|
| **Welvet** | Engine: Dense MatVec microkernel, storage-truth dtypes/quants, 29 TrainModes, cameral/Parallel sandwiches, SIMD/WebGPU backends. Scorecarded to v1.1.1 (100/100). |
| **Lucy** | Shared measuring math: SoftAcc, Score, Availability, **LPD (Lucy Pareto density)** / “synthetic organism” density. Lives in Welvet; Tide displays it. |
| **AAI** | Private bench tree for Lucy races (cameral, credit sweeps, FP32 races). Same formulas Tide uses. |
| **Tide** | Dataset-agnostic **serve + train** framework: permute dtype×quant×mode×arch under live load, dashboard, PDF reports. |
| **Ocean** | Tide-of-tides: does **not** train; polls peers and stitches a master report. |

Public engine: [github.com/openfluke/welvet](https://github.com/openfluke/welvet). Tide package: `github.com/openfluke/tide`.

---

## 3. Welvet — what matters for this experiment

Welvet exists because earlier Loom-era code hit honesty walls (silent fallbacks, QAT morph, float32 masters). Non-negotiables:

1. No silent fallbacks — hard error  
2. No hardcoded float32 — `Tensor[T]`  
3. No QAT morph — **DType + Format = storage truth**  
4. Train keeps storage truth (in-dtype / re-Pack SGD)  
5. One feature → one folder; tests live in `w2a`, not engine packages  

**Why that matters here:** a permute sweep is only meaningful if a “float16 cell” is actually float16, not a fake cast. Lucy rankings are about **live-fit under real storage**, not offline float32 accuracy alone.

**TrainModes (29 named updates)** are credit / update paths (NormalBP, Tween, Mesh*, Split/Alt, FastProxy, Sparse, …) — not just “Adam vs SGD.” Different modes change how gradients/credit flow while serve continues.

**Cameral / Parallel:** multi-hemisphere nets (`cam=1..N`). Stem → Parallel → head Stack; hemispheres can use different TrainModes; CamSync can blend weights across cams.

---

## 4. Lucy — the metrics (read the PDFs with this)

Lucy Score is **live-fit**: can the net **learn while it still serves**, in a small box?

| Symbol | Formula / meaning |
|--------|-------------------|
| **Hard Acc** | Argmax accuracy % — the Acc pillar |
| **SoftAcc** | Serve-confidence from true-class softmax (diagnostic; **not** the Score Acc term) |
| **Throughput T** | Outputs / second while the sweep is live |
| **Availability** | `InferMs / (InferMs + TrainMs) × 100` — duty cycle; dies when train blocks serve |
| **Lucy Score** | `T × Availability × Acc / 10_000` |
| **Q** | Geomean of Acc/Thru/Avail keep vs learner peaks (learner = Acc keep ≥70% of Acc champ) |
| **LPD** | **Lucy Pareto density** — `Q × shrink` vs Acc-champ RAM; **0** unless RelAcc ≥70% (weeds Score/MiB traps). Memory intelligence on the live-fit Pareto edge: how much Acc/Thru/Avail keep you retain when you shrink the weight box. |
| **Gold / Gold-std / Trap** | Trifecta-in-small-box vs two-pillar vs false density |

**Pareto idea:** undominated edge of learning vs size vs live duty. Prefer **LPD** over MobileScore (`Score / WeightMiB`) for goldilocks — MobileScore rewards tiny chance-Acc models.

Tide dashboards expose live / Lucy / honesty / winners / boards; PDF = `/api/report.pdf`.

---

## 5. Tide & Ocean in practice

### Tide
- Host supplies a `Dataset` + model `Spec`.
- Runner does concurrent **infer + train** pulses.
- Matrix (SIMD): dtypes × quants × TrainModes × cameral arches (host-configured).
- Checkpoint/resume: done cell IDs skip on restart so adding modes does not replay finished work.
- This farm’s host: **test51** — “think-race” / tree sweep feeding Tide Lucy reports (`dense` layer focus, challenge trees, LR↑ × cams × grids).

### Ocean
- Bind locally (we used `:8090`), static peers = the three farm Tides.
- UI `/`, PDF `/api/report.pdf`, compare `/compare`.
- Local helper in this repo: `./bin/ocean` or `go run ./cmd/ocean` (writes under `reports/`).

---

## 6. The experiment — volatile online soak

### Intent (important)
This was **not** “train until 100% of cells complete.”

It was a **longevity / stability / progress-under-load** test:

- Run the full permute tree on **cheap cloud VMs**
- Hold **~100% CPU** for **days/weeks**
- Watch: process survival, checkpoint resume, dashboard liveliness, cells/hour, mid-run Lucy winners
- Extract Tide/Ocean PDFs as the artifact

Finishing ~1–2% of a 110k–122k leaf tree after weeks on small boxes is an **expected cost of a huge live matrix**, not by itself a failed experiment.

### Farm layout

| Peer | Host (at run time) | Tide | Tree dash | Role |
|------|--------------------|------|-----------|------|
| **c1** | `172.236.32.254` | `:8080` | `:5151` | test51 config slice 1 (TrainMode family A) |
| **c2** | `172.236.32.77` | `:8080` | `:5151` | config slice 2 (family B) |
| **c3** | `172.236.53.25` | `:8080` | `:5151` | config slice 3 (family C) |

Runtime: Docker image `test51:farm`, containers `test51-c1` / `c2` / `c3`, env flags like `TEST51_FULL=true`, `TEST51_RESUME=true`, `TEST51_AUTOSTART=true`, `TIDE_ADDR=0.0.0.0:8080`, layer `dense`.

Cell IDs look like:

```text
challenge|grid|layer|dtype|lr=…|cam=N|TrainMode
```

Examples seen: challenges **chase / flee / collect / shock**; grids `1x1x1`…`3x3x3`; modes such as `NormalBP`, `StepTweenSplit`, `MeshTweenAlt`.

### What “volatile online” means here
- Public cloud VMs (reboot, noisy neighbors, disk, network)
- Sustained CPU saturation
- HTTP dashboards competing with the training loop (c3 showed intermittent `/api/*` stalls while training logs continued)
- Resume-after-restart semantics matter as much as peak Score

---

## 7. Snapshot at extract (Sep 2026)

Use the PDFs as ground truth; JSON boards were mid-pull helpers.

| Peer | Progress (board) | Rough best Score cell (board) | Acc / Thru / Avail / Score |
|------|------------------|-------------------------------|----------------------------|
| c1 | ~2.2% (2676 / 122400) | `chase|…|float64|lr=2|cam=1|NormalBP` | ~62% / ~1359 / ~80% / ~679 |
| c2 | ~1.0% (1208 / 122400) | `chase|…|float64|lr=2|cam=1|StepTweenSplit` | ~77% / ~2105 / ~95% / ~1554 |
| c3 | early after restart (~55 / 110160) | `chase|…|float64|lr=2|cam=1|MeshTweenAlt` | ~78% / ~983 / ~49% / ~376 |

Ocean at pull: often **2/3 peers up** (c3 HTTP flaky); holistic lean **StepTweenSplit** / **float64** (confirm in `ocean-report.pdf`).

### Artifacts in this repo

```text
reports/
  c1-report.pdf      # Tide Lucy PDF — peer 1
  c2-report.pdf
  c3-report.pdf
  ocean-report.pdf   # consolidated master (peers that answered)
  c*-board.json      # compact /api/board dumps (optional)
  c*-meta.json

EXPERIMENT.md        # this file
notes.md             # ops cheat sheet (SSH / ocean commands)
cmd/ocean/           # local Ocean runner for the three peers
tide/                # Tide library checkout used for Ocean
```

Also attach externally: **Welvet Feature Book v1.1.1** (engine + Lucy + cameral/AAI chapters).

---

## 8. How to read the reports

1. **Cover / meta** — task name (`test51`), epoch, plan size.  
2. **Progress** — cells done vs plan; do not confuse with “model converged.”  
3. **Best Score / Acc / Thru / Avail** — live-fit pillars; check whether winners are early bias.  
4. **LPD (Lucy Pareto density) / gold / trap** — memory intelligence vs false density.  
5. **Winners / heat / mode tables** — which TrainModes and dtypes keep showing up.  
6. **Ocean PDF** — cross-machine consolidation; missing peer = that Tide was down or too slow to answer.

---

## 9. Industry framing (short)

| Practice | Typical goal | Contrast with this work |
|----------|--------------|-------------------------|
| PyTorch / JAX train loops | Offline loss ↓, val Acc ↑ | Here: Acc **and** serve duty cycle **during** train |
| MLPerf Training / Inference | Throughput / time-to-train / offline latency | Lucy Score mixes T × Avail × Acc under live load |
| Continual / test-time adaptation papers | Non-stationary data streams | Similar spirit; this stack also permutes dtype×mode×cam on SIMD |
| TinyML / edge benches | Size, latency, power | LPD/gold bands aim at “learn in a small box,” not only compress |
| NAS / HPO sweeps | Search arch/hyperparams offline | Tide searches under **concurrent serve+train**, storage-truth numeric paths |
| Chaos / soak tests | Uptime under stress | This farm is soak+science: stability **and** mid-run Lucy leaders |

**Claims this experiment supports:** long-run CPU soak behavior; progress rate on huge trees; dashboard/checkpoint behavior; mid-run ranking signals.  
**Claims it does not support alone:** SOTA on public NLP/CV leaderboards; finished full-matrix winners; multi-GPU fairness; statistical finality of early leaders.

---

## 10. Deep research — paste zone

Paste completed deep-research outputs below (or link files). Keep this section as the narrative glue.

### Deep research A — system & industry

*(paste or link)*

### Deep research B — report interpretation / soak findings

*(paste or link)*

### Combined takeaways (fill after reading both)

- Stability:  
- Progress under 100% CPU:  
- Strongest mid-run signals (mode / dtype / challenge):  
- Gaps vs industry:  
- Next experiment if any:  

---

## 11. Ops (optional)

Start Ocean against the three peers (if hosts still exist):

```bash
cd /home/openfluke/git/online_extract
./bin/ocean -addr 0.0.0.0:8090 -out reports
# UI  http://127.0.0.1:8090
# PDF http://127.0.0.1:8090/api/report.pdf
```

Per-host PDF while alive: `http://<ip>:8080/api/report.pdf`.

SSH / passwords: see `notes.md` (do not commit secrets to public remotes).

---

## 12. Glossary

| Term | Meaning |
|------|---------|
| **Live-fit** | Learn and serve at the same time; Lucy Score encodes that |
| **Permute / matrix cell** | One dtype×format×mode×arch×(challenge/LR/cam/…) trial |
| **Availability** | Fraction of time spent inferring vs infer+train |
| **LPD** | **Lucy Pareto density** — live-fit condensed into RAM (`Q × shrink` vs Acc champ); 0 if Acc keep &lt;70% |
| **Trap** | Tiny RAM + weak Acc keep — looks “dense,” doesn’t learn |
| **Cameral** | Multi-hemisphere Parallel sandwich |
| **Ocean peer** | One Tide origin Ocean polls |
| **Soak** | Long run under sustained load for stability, not epoch completeness |

---

*Generated for the online_extract archive of the test51 three-machine farm. Update §7 and §10 when new reports or deep-research drafts land.*