README.md

---
title: SAM 3 / 3.1 + Sapiens Pose ZeroGPU
emoji: 🎯
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.42.0
app_file: app.py
pinned: false
license: apache-2.0
hardware: zero-a10g
---

# sam3-zerogpu

HuggingFace ZeroGPU Space that hosts Meta's **SAM 3 / SAM 3.1** models
for Cadayn — **plus a bundled Sapiens-0.6B pose estimator** (see "Bundled
Sapiens pose" section below).

## Endpoints in this Space

**SAM 3 / SAM 3.1 (primary purpose):**

- `api_track_object` — SAM 3.1 multiplex video object tracking
- `api_annotate_image` — SAM 3 per-image text-prompted detection + optional masks
- `api_annotate_images_batch` — batched variant, up to 8 items per GPU slot

**Bundled Sapiens pose (added because of the 10 ZeroGPU Space cap):**

- `api_pose_image` — single image → 308 keypoints per detected person
- `api_pose_video_frames` — video URL + timestamps → per-frame multi-person
  poses with stable IDs across frames (ByteTrack)

**Bundled NeuFlow v2 optical flow (Phase 2 — same cap reasoning):**

- `api_optical_flow` — video URL + anchor timestamps; for each anchor
  computes flow between (t, t+pair_gap_s) and returns aggregated motion
  stats (mean / p95 / max magnitude px, dominant direction deg, direction
  consistency)

**Bundled Video Depth Anything (Phase 4 — same cap reasoning):**

- `api_video_depth` — video URL + anchor timestamps + optional per-frame
  bboxes; returns RELATIVE depth in [0, 1] (never metric), per-bbox
  p10/p50/p90 percentiles, and a near→far ordering of the bboxes per
  frame. Set `is_relative: true` flag is always present in the response
  so downstream code never promises metric distances.

The backend client lives in
`cadayn-backend/app/services/training/sam3p1_client.py`. Both halves are
versioned independently; the contract below documents what they must agree
on.

## API contract (v2 — Phase 3a W-3)

### `api_annotate_image` inputs

| Position | Name | Type | Notes |
|---|---|---|---|
| 0 | `image_url` | `str` | HTTPS URL; signed URLs ok |
| 1 | `text_prompts` | `str` | comma-separated class names |
| 2 | `confidence_threshold` | `float` | `0.0`–`1.0`, default `0.3` |
| 3 | `requested_masks` | `bool` | **v2 addition.** `True` = extract masks; `False` = boxes only |

`api_annotate_images_batch` has the same v2 flag in position 2
(after `items_json`, `confidence_threshold`).

### `detection.mask_rle.format`

Every returned detection includes a `mask_rle` dict with a `format`
field that callers must branch on:

- `uncompressed_rle_b64` — real SAM 3 segmentation from `processor.segment`
  or the `set_text_prompt` output dict. Decode with
  `pycocotools.mask.decode` after base64-decoding `counts`.
- `bbox_rle_b64` — rectangular fallback built from the detection's bbox.
  Used when the installed `sam3` release does not expose masks. Treat
  these as bbox-equivalent; do not use them for fine-grained overlays.

The mask section of the detection is absent when `requested_masks=False`.

## Deploy ordering with the backend

The Space and the backend are deployed from separate repos; the v2
contract takes 4 args on `api_annotate_image`. Mismatched versions break
differently depending on direction:

| Backend | Space | Result |
|---|---|---|
| v1 (3 args) | v1 (3 args) | OK — pre-Phase-3a baseline |
| v1 (3 args) | v2 (4 args) | OK — Gradio supplies `requested_masks=True` default |
| v2 (4 args) | v1 (3 args) | **BREAKS** — `ValueError: Expected 3 inputs, received 4`. Client retries once with 3 args and logs a warning |
| v2 (4 args) | v2 (4 args) | OK — steady state |

**Recommended ordering** when shipping a v2 change:

1. Deploy the Space first (`./deploy-phase-2-spaces.sh` or manual push).
2. Wait for the Space to reach `RUNNING`.
3. Deploy the backend — which now passes the 4th arg.

The client's one-shot retry is a belt-and-braces for the deploy window;
it is not a substitute for ordering.

## Env vars

- `SAM3_VIDEO_HF_SPACE_URL` — `api_track_object`
- `SAM3P1_IMAGE_HF_SPACE_URL` — `api_annotate_image` + batch
- `SAPIENS_HF_SPACE_URL` — `api_pose_image` + `api_pose_video_frames`
  (point at this same Space — the bundled endpoints share the container)
- `NEUFLOW_HF_SPACE_URL` — `api_optical_flow` (same Space; Phase 2)
- `VDA_HF_SPACE_URL` — `api_video_depth` (same Space; Phase 4)

All five env vars can point at the same Space URL — the bundled Space
registers all seven Gradio routes.

## Bundled Sapiens pose

Sapiens-0.6B + YOLOv8n person detection + supervision.ByteTrack share
this container because the magboola HF account is at the 10 ZeroGPU Space
cap and SAM was the most natural visual co-tenant.

The standalone copy lives at `huggingface-spaces/sapiens-zerogpu/` (kept in
the tree but **not deployed**) for future extraction once the cap unblocks.
If/when extracted, sync `app.py` between the two locations and split the
pose-related entries out of `requirements.txt`.

### Pose output schema

```json
{
  "ok": true,
  "frames": [
    {
      "timestamp_s": 2.0,
      "width": 1920,
      "height": 1080,
      "persons": [
        {
          "id": "p_3",
          "bbox": {"x1": 412, "y1": 88, "x2": 712, "y2": 920, "conf": 0.91},
          "keypoints": [
            {"name": "nose", "x": 562.3, "y": 185.7, "conf": 0.94, "occluded": false}
          ],
          "n_visible_keypoints": 287
        }
      ]
    }
  ],
  "model": "facebook/sapiens-pose-0.6b-torchscript",
  "max_persons_capped": false,
  "elapsed_s": 4.12
}
```

Pose limits: 32 frames per video call, 12 persons per frame.

`goliath_keypoints.txt` ships with the Space — populated from
`facebookresearch/sapiens` to give keypoints semantic names; falls back
to `kp_0..kp_307` if the file is empty.