File size: 3,340 Bytes
3f7bcb8 1c8bbe2 3f7bcb8 592578e 3f7bcb8 592578e 3f7bcb8 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 |
---
library_name: py-feat
pipeline_tag: image-feature-extraction
license: apache-2.0
language:
- en
---
# MP_Blendshapes
## Model Description
MP_Blendshapes has been ported to pytorch from Google's [mediapipe](https://github.com/google-ai-edge/mediapipe) library using Liam Schoneveld's github [repository](https://github.com/nlml/deconstruct-mediapipe). The inputs are 146/473 of mediapipe's FaceMeshV2 high density landmark model. The 52 blendshapes are similar to [ARKit Face Blendshapes](https://arkit-face-blendshapes.com/) and loosely correspond to Facial Action Unit Coding System.
See the mediapipe [model card](https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf) for more details.
## Model Details
- **Model Type**: MLP-Mixer (Keras)
- **Framework**: pytorch
## Model Sources
- **Repository**: [GitHub Repository](https://github.com/cosanlab/py-feat)
- **Model Card**: [Mediapipe blendshape model card](https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf)
- **Paper**: [Attention mesh: High-fidelity face mesh prediction in real-time](https://arxiv.org/abs/2006.10962)
## Citation
If you use the mp_blendshapes model in your research or application, please cite the following paper:
Grishchenko, I., Ablavatski, A., Kartynnik, Y., Raveendran, K., & Grundmann, M. (2020). Attention mesh: High-fidelity face mesh prediction in real-time. arXiv preprint arXiv:2006.10962.
```
@article{grishchenko2020attention,
title={Attention mesh: High-fidelity face mesh prediction in real-time},
author={Grishchenko, Ivan and Ablavatski, Artsiom and Kartynnik, Yury and Raveendran, Karthik and Grundmann, Matthias},
journal={arXiv preprint arXiv:2006.10962},
year={2020}
}
```
## Example Useage
```python
import torch
import pandas as pd
from huggingface_hub import hf_hub_download
from feat.au_detectors.MP_Blendshapes.MP_Blendshapes_test import MediaPipeBlendshapesMLPMixer
from feat.utils import MP_BLENDSHAPE_MODEL_LANDMARKS_SUBSET, MP_BLENDSHAPE_NAMES
device = 'cpu'
# Load model and weights
blendshape_detector = MediaPipeBlendshapesMLPMixer()
model_path = hf_hub_download(repo_id="py-feat/mp_blendshapes", filename="face_blendshapes.pth")
blendshape_model_file = hf_hub_download(repo_id='py-feat/resmasknet', filename="ResMaskNet_Z_resmasking_dropout1_rot30.pth")
blendshape_checkpoint = torch.load(blendshape_model_file, map_location=device)["net"]
blendshape_detector.load_state_dict(blendshape_checkpoint)
blendshape_detector.eval()
blendshape_detector.to(device)
# Test model
face_image = "path/to/your/test_image.jpg" # Replace with your extracted face image that is [224, 224]
# Extract Landmarks
landmark_detector = torch.load('/Users/lukechang/Dropbox/py-feat/mediapipe/model/face_landmarks_detector_Nx3x256x256_onnx.pth', weights_only=False)
landmark_detector.eval()
landmark_detector.to(device)
landmark_results = landmark_detector(torch.tensor(face_image).to(device))
# Blendshape Classification
landmarks = landmark_results[0].reshape(1,478,3)[:,:,:2]
img_size = torch.tensor((face_image_width, face_image_height)).unsqueeze(0).unsqueeze(0)
landmarks = landmarks * img_size
blendshapes = blendshape_detector(landmarks)
blendshape_results = pd.Series(blendshape_results.squeeze().detach().numpy(), index=BLENDSHAPE_NAMES)
``` |