Instructions to use dphn/dolphin-vision-7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use dphn/dolphin-vision-7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="dphn/dolphin-vision-7b", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("dphn/dolphin-vision-7b", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use dphn/dolphin-vision-7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "dphn/dolphin-vision-7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-vision-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/dphn/dolphin-vision-7b

SGLang

How to use dphn/dolphin-vision-7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "dphn/dolphin-vision-7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-vision-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "dphn/dolphin-vision-7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dphn/dolphin-vision-7b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use dphn/dolphin-vision-7b with Docker Model Runner:
```
docker model run hf.co/dphn/dolphin-vision-7b
```

Transformers doesn't support it yet?

by mahiatlinux - opened Jul 15, 2024

Discussion

mahiatlinux

Jul 15, 2024

•

edited Jul 15, 2024

Getting this error while trying to make a Space for it.

ValueError: The checkpoint you are trying to load has model type `bunny-qwen` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

nicolollo

Jul 16, 2024

Files missing ? Bunny its from https://github.com/BAAI-DCAI/Bunny and also it should be bunny-qwen2 ? like in Bunny Qwen models but it is customized i guess so ....

saishf

Jul 17, 2024

•

edited Jul 17, 2024

It probably needs it's own version of these

],
"auto_map": {
"AutoConfig": "configuration_llava_qwen2.LlavaQwen2Config",
"AutoModelForCausalLM": "modeling_llava_qwen2.LlavaQwen2ForCausalLM"
},

Dolphin-Vision-72B has them and that seems to work fine

I added a copy to the model myself and it will now load into memory. It just won't inference

nicolollo

Jul 17, 2024

It probably needs it's own version of these

],
"auto_map": {
"AutoConfig": "configuration_llava_qwen2.LlavaQwen2Config",
"AutoModelForCausalLM": "modeling_llava_qwen2.LlavaQwen2ForCausalLM"
},

Dolphin-Vision-72B has them and that seems to work fine

I added a copy to the model myself and it will now load into memory. It just won't inference

It bunny not llava

SaisExperiments

Jul 17, 2024

•

edited Jul 17, 2024

It bunny not llava

It needs the same kind of files, just for bunny instead. These files just give transformers the info required to start without giving the unknown model type error. Which this model does, from what I think is due to the lack of similar files?

Something else interesting is that the llava config files only mention 7b while being in the 72b repo, it's confusing

Edit: it might be confusing, this is the account I use on mobile. I'm also @saishf

catworld1212

Jul 20, 2024

I'm getting the same error "The checkpoint you are trying to load has model type bunny-qwen but Transformers does not recognize this
architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date."

qnguyen3

Jul 23, 2024

i updated the files

mahiatlinux changed discussion status to closed Aug 13, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment