py-feat/bs_to_au

BS → AU PLS

Licensing scope: See the license and provenance notice before relying on this card's license metadata for pretrained-weight redistribution or commercial use. Existing valid grants are preserved.

Predicts 20 FACS Action Unit intensities from 52 MediaPipe blendshapes via Cheong-style PLS regression. Lets MPDetector output AU columns comparable to Detector's xgb output.

BS -> AU PLS (v2)

Linear PLS regression mapping 52 MediaPipe blendshapes to 20 FACS Action Unit intensities. Used to give MPDetector an AU output stream comparable to Detector's xgb AU model output.

Training data

• 350,568 frames from ~10,000 CelebV-HQ celebrity videos
• Paired blendshapes (MPDetector mp_blendshapes head) + AU intensities (Detector with img2pose face + xgb AU on the same frames)
• Pose-filtered to |yaw| <= 40°, |pitch| <= 30° -> 347,897 retained
• 9,994 unique videos after filtering
• See /Storage/Projects/mp_blendshapes for the underlying training pipeline

Method

• PLSRegression(n_components=20, scale=True), Cheong / Py-Feat style
• 20 components = full rank (capped at min(n_features=52, n_targets=20))
• Linear features only — pairwise BS interactions were tested in nested CV (2026-05-05) and HURT out-of-sample R² (bs_only=0.236 vs bs_pairs=0.214, with 4-6x higher fold std)
• No pose covariates: kept pose-agnostic since MP blendshapes are designed to be pose-canonical
• No clipping at training (clip to [0,1] at inference if desired)

Performance (3-fold GroupKFold by video_id)

• Overall R² = 0.236 +/- 0.008 (variance-weighted across 20 AUs)
• Overall MAE = 0.171
• Strong on AU06/12/43 (~0.50)
• Moderate on AU01/02/09 (~0.29)
• Weak on AU11/15/28 (<0.10) — these are rare or visually subtle AUs

Citation context

• Cheong et al. 2023 (Py-Feat AU visualization model, tutorial 06 by E. Jolly): affine-aligned 68 dlib landmarks -> 20 AUs via PLS on EmotioNet/DISFA/BP4D (~13K class-balanced rows). Our model scales the recipe up to MP blendshapes on 10x larger wild-celebrity data.

Inference

The saved coef + intercept absorb PLSRegression's scale=True standardization, so inference is a single matmul:

au = blendshapes @ coef + intercept   # (n, 52) @ (52, 20) + (20,) = (n, 20)
au = np.clip(au, 0.0, 1.0)            # optional

File format

NPZ with:

• coef (52, 20) float32 — linear weights, rows match bs_columns
• intercept (20,) float32 — bias, matches au_columns
• bs_columns (52,) str — input feature order
• au_columns (20,) str — output AU order
• model_card () str — this markdown
• training_metadata () str — JSON dict with training context

Loader: np.load("bs_to_au_pls_v2.npz") — no extra deps needed.

License and provenance

This distribution retains its existing license: mit metadata. This PLS artifact was fit on CelebV-HQ frames using MediaPipe blendshapes and XGBoost AU predictions, with img2pose used for face/pose processing as described above.

The CelebV-HQ agreement and the licenses/permission scope of the exact teacher artifacts are separate from Py-Feat's original MIT code. img2pose is CC BY-NC 4.0; MediaPipe has its own source license. Source-code terms alone do not determine rights in model assets, predictions, or these fitted coefficients. See the component notice and dataset register.

This notice clarifies scope; it does not revoke existing valid grants or create a new license for third-party material. Dataset and teacher terms do not automatically relicense every trained artifact or inference output. A software license or model-card badge alone does not establish all checkpoint redistribution or commercial-use permissions.