---
license: other
tags: [maritime, segmentation, thermal, lwir, flir-lepton, usv, water-segmentation, lraspp, mobilenetv3, cross-modal, lars, sailswarm, raspberry-pi, onnx]
library_name: pytorch
---

# SailSwarm LRASPP thermal student (water/sky/obstacle segmentation on a FLIR Lepton 3, 160×120)

`lraspp_mobilenet_v3_large` (torchvision, ~3.2 M params) trained to segment
**obstacle / water / sky** in long-wave infrared frames from a FLIR Lepton 3.0
(PureThermal, 160×120, ~57° HFOV) mounted on the SailSwarm autonomous-sailboat
obstacle-detection box (University of Konstanz, Lake Constance). Companion to the
RGB model [`NexusDwin/sailswarm-lraspp-student`](https://huggingface.co/NexusDwin/sailswarm-lraspp-student);
same architecture, same 3-class convention `{0 obstacle, 1 water, 2 sky}`.

The point of this model: thermal is the night-time primary sensor for the boat, and
no public maritime LWIR segmentation ground truth exists at this resolution. The
labels are **cross-modal**: the RGB student's masks on the co-mounted fisheye camera,
warped into the thermal frame (bearing-for-bearing under the measured linear thermal
bearing model + a horizon-fit vertical offset; the two lenses share a rigid mount
with a ~6 cm baseline, so parallax is <1° beyond 2.5 m).

## What is in this repo

| Folder | Generation | Data | Pinned holdout (under-pier clip 2026-07-08 16-44-56, n=752) |
|---|---|---|---|
| `v1_day_2026-08-22/` | JOINT recipe, day-only thermal | 33,467 gray-Konstanz fisheye frames (six missions) + gray LaRS + ~10k warped real thermal frames, 60 epochs | **acc 0.9090 / 0.9069 / 0.9080** (seeds 0/1/2), obstacle IoU 0.89, water IoU 0.46, sky 0.74 |
| `v2_night_2026-08-28/` | same JOINT recipe, **first dusk/night thermal** (2026-08-26 outing: pontoon → open water → harbour dusk → night) | 68,350 gray-Konstanz native frames (whole 2026 corpus) + gray LaRS + 114,501 total training pairs incl. ~50k warped thermal frames from 63 clips, 60 epochs | **acc 0.9136 / 0.9139 / 0.9131** (seeds 0/1/2 → 0.9135 ± 0.0003), obstacle IoU 0.90, water IoU 0.54–0.55, sky 0.69 |

Per seed: `thermal_joint_s<k>_best.pth` (torchvision state dict) and
`thermal_joint_s<k>_160x120.onnx` (fp32, opset 17, fixed 160×120, input `image`
NCHW, output `logits` 1×3×120×160). `train_all_seeds.log` is the full training log.
Seed variance ≤ ±0.001 in both generations — pick any; seed 0 is the one we evaluate against.
**Use `v2_night_2026-08-28/` unless you specifically need the day-only model**: it is +0.005
acc on the pinned holdout and is the only generation that has seen thermal frames captured
after civil dusk. Water IoU +0.10 / sky −0.06 vs v1 on that holdout — the night frames shift
where the model draws the water boundary under the pier; treat the sky drop as a known
property of this eval scene (almost no sky visible), not a regression measurement.
`warp_fit_report.json` is the per-clip fisheye→thermal warp fit used to make v2's labels.

## Why "JOINT" — the lesson this model encodes

Trained on thermal alone (~1k frames of one dock), this architecture memorised scene
*layout* (holdout 0.38–0.58 across seeds). Grayscale RGB from the same lake, trained
jointly in the same batches as the thermal frames, is a stronger structural prior than
more thermal data: 10k gray frames of one dock → 0.61 ± 0.02; 33k gray frames across six
missions → **0.908 ± 0.001**. Sequential fine-tuning from a gray-pretrained init does
not reach this; joint batches do. Augmentations: random polarity inversion (p 0.5 —
the Lepton's AGC flips warm/cold contrast between day and night) and ±12 px vertical
shift (breaks horizon-row memorisation).

## Preprocessing (must match)

Input is the **upright, colourised 8-bit Lepton stream collapsed to luminance and
replicated to 3 channels**, then ImageNet-normalised — exactly what the `--gray`
training flag did:

```python
import onnxruntime as ort, numpy as np, cv2
sess = ort.InferenceSession("v1_day_2026-08-22/thermal_joint_s0_160x120.onnx",
                            providers=["CPUExecutionProvider"])
frame = cv2.imread("thermal.png")                       # 160x120 BGR from the PureThermal stream
g = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32) / 255.0
x = np.stack([g, g, g], 0)                              # gray x3, CHW
x = (x - np.array([0.485, 0.456, 0.406])[:, None, None]) / np.array([0.229, 0.224, 0.225])[:, None, None]
cls = sess.run(None, {"image": x[None].astype(np.float32)})[0][0].argmax(0)   # 120x160 class map
```

Our capture rotates the thermal frame 180° at the box (the module is mounted
inverted); feed the model the *upright* frame. Repair dead sensor rows before
inference if your Lepton has them (ours has row 21 stuck hot).

## Limits — read before using

- **v1 is day-only; v2 has one dusk/night outing** (2026-08-26, ~4 h of Lepton footage,
  pseudo-labelled from the RGB model while the RGB model could still see). Frames after
  full dark are labelled from an RGB teacher operating near its own luminance floor —
  those labels are the weakest in the set. The pinned holdout is a daytime clip, so the
  numbers above do **not** measure night performance; audited night thermal ground truth
  does not exist yet.
- **Water IoU 0.46 on the holdout is a property of the holdout scene** (an under-pier
  clip with almost no visible water), not a measurement of open-water quality; there is
  no independent thermal ground truth yet — every label is a warped RGB pseudo-label,
  so systematic RGB-model errors (e.g. reflections read as obstacle) are inherited.
- **fp32 only.** Static int8 quantisation fails our pixel-agreement gate on this model
  (per-frame agreement min 40% on seed 0, worse on seeds 1–2) — recipe-independent
  fragility of weak thermal logits under quantisation. At 160×120 the fp32 ONNX is
  fast enough on a Raspberry Pi 4 CPU.
- Not validated for closed-loop navigation.

## Provenance / license

Backbone: torchvision `lraspp_mobilenet_v3_large` (BSD-3, ImageNet-pretrained).
Training data: **LaRS** (Žust et al., ICCV 2023; research/non-commercial — derived
weights inherit its terms) as grayscale, plus Lake-Constance fisheye and Lepton
footage captured by the SailSwarm project (University of Konstanz, Cluster of
Excellence in Collective Behaviour) with pseudo-labels from
[`NexusDwin/sailswarm-lraspp-student`](https://huggingface.co/NexusDwin/sailswarm-lraspp-student).
Trained with `scripts/gpu_seg/train_student.py` (`--gray --aug-polarity 0.5
--aug-vshift 12 --size 160x120`) from the SailSwarm obstacle-detection repository.

## v3_night_2026-09-02 (added 2026-09-09)

JOINT retrain of the thermal LRASPP student with the 2026-08-26 dusk/night outing
in the thermal set (three seeds, 60 epochs, same recipe as v2). Pinned under-pier
holdout (clip 2026-07-08 16-44-56, n=752): **pixel accuracy 0.9148 / 0.9149 / 0.9146
= 0.9148 ± 0.0001, obstacle IoU 0.902** (v2_night: 0.9135 ± 0.0003; v1_day: 0.908).
Seed 1 (`thermal_night_s1_160x120.onnx`, md5 d1add978…) is the copy deployed on the
box as `~/ewasr/thermal_student_night_v3_s1_160x120.fp32.onnx`: **25 fps at 160×120
on a Raspberry Pi 4** (2 threads, fp32; int8 fails the agreement gate for every thermal
generation, so fp32 is the deployment format). Not yet a voter in the live fusion
scorer; the shadow runtime consumes the fisheye student only (2026-09-09).
Training log: `night_ladder.log`.

## v4b_night_gated_2026-09-12 and v5_massmind_2026-09-12 (added 2026-09-15)

Both are warm-started from v3 (three seeds each, 60 epochs, same recipe) after the
2026-09-08 lake session, whose recording runs from daylight into true night. **Every
thermal pseudo-label is now gated on fisheye luminance** (mean luminance >= 25 per chunk):
after 20:15 the RGB teacher's frame is black and its mask reads 99.6 to 100 % obstacle,
so an ungated retrain (v4, discarded) merely learned to paint everything obstacle. No
night number can exist from cross-modal labels; the holdout below is the four darkest
chunks that still have a valid teacher (19:56 to 20:11, dusk).

| model | 19:56 | 20:01 | 20:06 | 20:11 | mean pixel acc |
|---|---|---|---|---|---|
| v3_night (never saw 2026-09-08) | 0.858 | 0.883 | 0.876 | 0.719 | 0.834 |
| **v4b_night_gated** | 0.869 | 0.905 | 0.900 | 0.728 | **0.851** |
| v5_massmind | | | | | 0.844 (within seed noise of v4b) |

- **v4b**: 31 gated 2026-09-08 thermal chunks + 31 gray-Konstanz + LaRS-gray. Modest,
  consistent dusk gain (water IoU 0.80 to 0.85 -> 0.90 to 0.95). Logs `thermal_train_0908_s*.log`.
- **v5**: the v4b mix plus the **MassMIND** LWIR training split (Boston-harbour USV,
  7 pixel classes mapped to sky/water/obstacle). Konstanz dusk 0.844 (no change) but
  MassMIND test **pixel acc 0.973 / obstacle IoU 0.85** (v3: 0.926 / 0.63): a broader
  maritime-LWIR model at no cost on our own domain. MassMIND is CC BY-NC-SA; derived
  weights inherit its terms.
- fp32 only (int8 fails the agreement gate on every thermal generation). None of these
  is deployed on the vessel; the box still runs v3 seed 1.
- The 20:11 chunk scores obstacle IoU about 0.04 for every model: the teacher there is
  already unreliable, which is the reason for the gate.

Record: `docs/history/2026-09-12_night_retrain_aime.md` sections 3 and 9 in the repository.