01 — Premise
An onboard ID is not a memory
The HuskyLens 2 is a genuinely good little vision module. It detects objects, tracks them frame to frame, assigns each one a numeric ID, and hands all of that to a microcontroller over I²C without the microcontroller doing any vision work at all.
But those IDs are tracking IDs, not identity. Look away from a chair and back at it and it gets a new number. Move to the other side of the room and the same chair is a stranger. For a robot that is supposed to build a persistent picture of a room — to know that the chair by the window is the same chair it saw twenty minutes ago — a tracking ID is not enough.
The fix is not a better camera. It is a second track that does re-identification properly, running somewhere with a GPU, on the same video the camera is already producing.
02 — Architecture
Two tracks, one camera
One HuskyLens 2 feeds both paths simultaneously. The camera's control protocol stays set to I²C while its Wi-Fi module streams video independently — those are separate subsystems on the module, and running both at once is the whole basis of this design.
The division of labour
| Track 1 — onboard | Track 2 — PC brain | |
|---|---|---|
| Latency | Milliseconds | Frame-rate bound, network bound |
| Answers | Something is at (x, y) right now | That something is CHAIR_3, seen before |
| Survives occlusion | No — new ID on reacquire | Yes — that is the entire point |
| Runs without the PC | Yes | No |
| Drives | Pan/tilt, immediate reactions | Scene understanding, dialogue |
HuskyLens 2 onboard IDs are treated as proposals only. Identity lives on the PC. That single decision is what keeps the two tracks from fighting each other — the microcontroller never has to be right about identity, only about position.
03 — Track one
I²C to the ESP32-C3
The C3 is the robot's reflexes. It reads detections over I²C, can close
a pan-tilt loop on them, and emits NDET / DT
proposal records over USB serial for the PC to consume.
-
Give the camera its own power
A regulated 5 V at 2 A minimum, or its own USB-C supply. Do not run it off the C3's 3.3 V regulator — a vision module with a display and a Wi-Fi radio is not a sensor-sized load, and browning it out produces I²C failures that look like wiring faults.
-
Wire the bus and share a ground
SDA → GPIO8,SCL → GPIO9, and a common ground between the camera and the C3. If the camera has a separate supply, that ground bond is what makes I²C work at all. -
Set the camera's protocol to I²C
In the camera's own menu. It then answers at address
0x50. Leave it on I²C even when you enable Wi-Fi streaming later — the two are independent. -
Share the bus deliberately
A PCA9685 servo driver at
0x40lives on the same two pins for pan/tilt. Different addresses, one bus, one pair of pull-ups. Both devices need to agree on the 3.3 V logic level. -
Flash the tracker
The sketch reads the camera at
0x50, selects object recognition, and emits proposal records. Pan/tilt boots limp and can stay physically unplugged — servos that snap to a default position on power-up break things.
If a scan reads SDA and SCL both high but gets no ACK, the bus wiring is probably fine and the device is not. In my case that meant the peripheral was unpowered at VCC. Check the rail before you rewrite the driver — a floating-high bus with no acknowledgement is the signature of a device that is not awake, not one that is misaddressed.
First flash has to be over USB
The stock partition layout on these boards is often huge_app,
which has no OTA slot — so there is nowhere for an over-the-air image to
land. The tracker builds against the OTA-capable min_spiffs
layout instead, which means the first install must go over USB
and every one after it can go over Wi-Fi. Wi-Fi and OTA passwords stay
out of source control.
04 — Track two
RTSP to the PC
The camera's Wi-Fi module publishes a standard RTSP stream. That is the unlock: full-resolution frames reach a machine with a GPU without the microcontroller ever touching image data.
-
Attach the Wi-Fi module and join the LAN
Same network as the PC. The camera's own screen shows the wireless IP it has been given.
-
Turn on RTSP
Under the camera's Video Streaming menu. The endpoint DFRobot documents is
rtsp://<wireless-ip>:8554/live. -
Confirm the stream before writing any code
Open it in any RTSP-capable player first. If the stream is not there, no amount of Python will find it, and debugging a perception pipeline against a dead socket wastes an afternoon.
-
Scale the I²C boxes onto the RTSP frame
The two tracks are at different resolutions. The bridge rescales the camera's I²C bounding boxes into the RTSP frame's coordinate space so the proposals line up with what the PC is actually looking at.
05 — The brain
How identity actually persists
The PC side is a hardware port of REMIND — RE-Identification with Memory for INDoor Navigation (arXiv:2607.09267). It is monocular RGB throughout: no depth sensor, no pose estimate, no SLAM. Just appearance and memory.
The split in responsibilities is clean. YOLO answers "what and where." DINOv3 plus the memory banks answer "which one."
| Stage | Implementation |
|---|---|
| Detect / segment | Ultralytics YOLO-seg (yolo11n-seg default), or HuskyLens 2 boxes |
| Appearance | Frozen DINOv3 ViT-S/16, one forward pass per frame |
| Object vector | Mask-weighted patch mean plus trimmed mean |
| Parts | K-means over object patches, Hungarian part-to-part matching |
| Background | Expanded-box ring taken off the mask |
| Memory | Working and stable prototype banks — EMA update, insert, promote |
| Neighbours | Class co-occurrence histogram plus pairwise distance moments |
| Assignment | Quality-weighted score with a lock, Hungarian with dummy "new object" columns |
| Guards | STRONG / AMBIGUOUS / WEAK, plus known-set distance rerank |
Why two memory banks
A single bank forces an impossible choice: update it eagerly and one bad frame poisons an object's identity forever; update it conservatively and the robot never learns that a chair looks different from a new angle.
Two banks resolve that. A working bank absorbs new observations quickly and cheaply. Observations that keep being confirmed get promoted into a stable bank that changes slowly. Bad frames land in the working bank and get discarded; genuine appearance changes accumulate until they earn promotion.
Why the Hungarian assignment has extra columns
A plain optimal assignment between detections and known objects is forced to match everything to something. That is wrong the moment a genuinely new object walks into frame. Adding dummy "new object" columns to the cost matrix gives the solver a legitimate way to say none of the above — the difference between a robot that discovers a new chair and one that insists the new chair is the old chair.
The gated DINOv3 weights and YOLO weights download on first real run, which means the pipeline has a network dependency at bootstrap. The paper authors' living-room sequence and the ScanNet++ conversions are not redistributed in my repo — licence and size — so reproducing their exact numbers requires sourcing those frames yourself and dropping them into the same folder layout.
06 — Operation
Running the whole stack
Install on the PC
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[brain,dev]"Meta's DINOv3 repository on Hugging Face is gated. Accept access for the ViT-S/16 pretrain model and authenticate locally before the first real frame. The token goes in your local credential store, never in the repo.
.venv/bin/hf auth loginTest the algorithms without any weights
This is the part I would keep if I rebuilt the project from scratch. The synthetic scene generates three same-class "chairs" that leave and return — exactly the case re-identification has to get right — so the memory, assignment and promotion logic can all be exercised with no GPU, no camera, and no model downloads.
pip install -e ".[dev]"
pytest -q
python -m remindbot synth --dest data/scenes --name synth_room
python -m remindbot assemble data/scenes/synth_room --fakeLive, both tracks
# terminal 1 — the PC brain
python -m remindbot serve --host 0.0.0.0 --port 8088
# terminal 2 — join RTSP video to the C3's I²C proposals
python -m remindbot bridge --husky-ip CAMERA_IP
# or start both local processes at once
python -m remindbot bridge --husky-ip CAMERA_IP --start-brain
# RTSP only, no microcontroller in the loop
python -m remindbot track --source rtsp://CAMERA_IP:8554/liveRun the bridge in a second terminal. The tracker dashboard and the bridge cannot both own the same serial port, and the failure mode is confusing — one process silently gets nothing rather than either of them erroring cleanly.
What the brain exposes
GET /scene— current objects with their stable labels, e.g.CHAIR_3POST /ingest— push a JPEG frame inPOST /chat— asks a local Ollama model with the current identities in the prompt
That last one is the reason the whole thing is worth building rather than just tracking boxes: a language model that knows which objects are actually in the room right now can answer questions about the room. And because Ollama runs on the same PC, none of it leaves the network.
07 — Status
Where it stands
The C3 vision path, the RTSP bridge, the full REMIND pipeline and the synthetic test harness all run. Pan/tilt is wired but boots limp by design. The wheel drive outputs boot detached and have to be explicitly armed.
Two things I got wrong first
- Continuous rotation is a hardware property. The wheel drive commands need continuous-rotation or properly modified servos. An ordinary 180° positional MG90S rotates to a commanded angle and stops — no firmware turns it into a wheel motor. I spent real time treating that as a software bug.
- Servos need their own rail. Power them from a separate regulated 5–6 V supply with a ground shared to the microcontroller. Servo current spikes on a logic rail cause resets that look like firmware crashes.
Still open
- Rear wheel GPIO assignments — deliberately absent from the code until the real pins are known, rather than guessed and left to rot
- Confirming the installed pan/tilt servo model; the firmware currently assumes MG996R
- Bulk capacitance across the servo rail near the connectors
