README.md

---
license: apache-2.0
---

# DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiences

[Project Page](https://sled-group.github.io/driVLMe/) | [Paper](https://arxiv.org/abs/2406.03008) | [Video](https://youtu.be/Ep5fYLGkmsg)  | [Code](https://github.com/sled-group/driVLMe/tree/main)

Yidong Huang, Jacob Sansom, Ziqiao Ma, Felix Gervits, Joyce Chai  
University of Michigan, ARL  
IROS 2024


You can also download the pretrained checkpoints from [this link](https://www.dropbox.com/scl/fo/neqjdlhohygoa0wrv4uuy/AAjarkE6WY6sKt4LoAfyZ3c?rlkey=e0yvw6g1j8qqdp63vhgi0722d&st=tp2w6h3f&dl=0)

To run the open-loop evaluation, we can use the command 
```shell
python drivlme/single_video_inference_SDN.py --model-name  /nfs/turbo/coe-chaijy-unreplicated/pre-trained-weights/LLaVA/LLaVA-7B-Lightening-v1-1/ --projection_path ./DriVLMe_model_weights/bddx_pretrain_ckpt/mm_projector.bin --lora_path  ./DriVLMe_model_weights/DriVLMe/ --json_path datasets/SDN_test_actions.json --video_root videos/SDN_test_videos/ --out_path SDN_test_actions.json

python evaluation/physical_action_acc.py
```
for NfD task and 
```shell
python drivlme/single_video_inference_SDN.py --model-name  /nfs/turbo/coe-chaijy-unreplicated/pre-trained-weights/LLaVA/LLaVA-7B-Lightening-v1-1/ --projection_path ./DriVLMe_model_weights/bddx_pretrain_ckpt/mm_projector.bin --lora_path  ./DriVLMe_model_weights/DriVLMe/ --json_path datasets/SDN_test_conversations.json --video_root videos/SDN_test_videos/ --out_path SDN_test_conversations.json

python evaluation/diag_action_acc.py
```
for RfN task.