Instructions to use aws-neuron/SOLAR-10.7B-v1.0-neuron with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aws-neuron/SOLAR-10.7B-v1.0-neuron")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron")
model = AutoModelForCausalLM.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aws-neuron/SOLAR-10.7B-v1.0-neuron"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron

SGLang

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Docker Model Runner:
```
docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron
```

jburtoft commited on Jan 13, 2024

Commit

6e0ef47

verified ·

1 Parent(s): 375e334

672068cb6f8f3b30fb85a7f1a86373148199225903a05bef16376bd8cd708de5

Browse files

Files changed (21) hide show

checkpoint/pytorch_model.bin/p168.model.layers.18.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p169.model.layers.18.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p17.model.layers.1.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p170.model.layers.18.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p171.model.layers.18.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p172.model.layers.19.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p173.model.layers.19.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p174.model.layers.19.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p175.model.layers.19.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p176.model.layers.19.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p177.model.layers.19.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p178.model.layers.19.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p179.model.layers.19.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p18.model.layers.1.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p180.model.layers.19.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p181.model.layers.20.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p182.model.layers.20.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p183.model.layers.20.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p184.model.layers.20.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p185.model.layers.20.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p186.model.layers.20.mlp.up_proj.weight +3 -0

checkpoint/pytorch_model.bin/p168.model.layers.18.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c772d8fc9070027705cffd45fa615c9196fff7e02a1dac813ade3d34c6a628b5
+size 234881910

checkpoint/pytorch_model.bin/p169.model.layers.18.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0d24e3e72ec7ca7b7fa385db90281bc5eb5585efd449b448f1a50037a8240661
+size 234881916

checkpoint/pytorch_model.bin/p17.model.layers.1.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dba08d1069cf30ffb36cc9db7b9a7aa367b9bd0a50875dfac03da1140c91bc69
+size 17276

checkpoint/pytorch_model.bin/p170.model.layers.18.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c783ff3578f6ef4b1ee508098384bb463ac9ba640596e7b319c8336d69cdced3
+size 17282

checkpoint/pytorch_model.bin/p171.model.layers.18.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1df7bee5a99460b40aef61708918ba40ededba429ccacf3588ea520fde35d6d5
+size 17309

checkpoint/pytorch_model.bin/p172.model.layers.19.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2959a20c4ea5303dae6eeced757847afcd86de91b69622dfade1e37734ae1305
+size 67109765

checkpoint/pytorch_model.bin/p173.model.layers.19.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:18fc7c4b6febe85f73bb68e2928a25c6a97583eed12e293a48bdb83968ade504
+size 16778117

checkpoint/pytorch_model.bin/p174.model.layers.19.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:097127b65d021d02e575b966096a2a82eb69f566f26bddf2cd08265049eb4baf
+size 16778117

checkpoint/pytorch_model.bin/p175.model.layers.19.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d425c44b89a8f8ce9353f1d37fddc0b428ae69ca785321dc1ac1d60d9601efe9
+size 67109765

checkpoint/pytorch_model.bin/p176.model.layers.19.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ffda8d910ad811bcfa923ffe9bde0dfd38fcd69136676320d3403fdf55778720
+size 234881916

checkpoint/pytorch_model.bin/p177.model.layers.19.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:62c73310ac0d67ec1e6d0a84ef91623a71a7f46dac5266baaf24fbb8b27e84b3
+size 234881910

checkpoint/pytorch_model.bin/p178.model.layers.19.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b4ac7558931ade857874799206c31e1d0040b296150f4c85ac3d4b5a6cf393c6
+size 234881916

checkpoint/pytorch_model.bin/p179.model.layers.19.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f89b4769afecc42c28e3da82d5ae9e778dac6545884cf81a27ed974f57e62e87
+size 17282

checkpoint/pytorch_model.bin/p18.model.layers.1.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6b6d6142b827518cbe96c86c8b2837bed0430ede791838071b1bfc357ec9815e
+size 17303

checkpoint/pytorch_model.bin/p180.model.layers.19.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8f3a279c697fcfbc6070ca95afa4e6cac8c7da0a60e2fac15abf3513dc795cb7
+size 17309

checkpoint/pytorch_model.bin/p181.model.layers.20.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e7b139f29dd9c460ddf9068bf08c3bf4dfbbdaee7e8ec463bfb48a76cf5bda7d
+size 67109765

checkpoint/pytorch_model.bin/p182.model.layers.20.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:66a630befb1d803a14453aaff78b50b139ce659408ee7d9c8966fdfc2137570e
+size 16778117

checkpoint/pytorch_model.bin/p183.model.layers.20.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:242ca577a9c313d1daa738b20e1c2bd2fc6ab3c614f7e720ef875270450d550a
+size 16778117

checkpoint/pytorch_model.bin/p184.model.layers.20.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6bfe77053c66e0b89c3fa98730520b356968a95999920b433841cbac57c841d1
+size 67109765

checkpoint/pytorch_model.bin/p185.model.layers.20.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f3648adc0acb8846aece59a68c8a905c0e15f3c37e3c86f6312aa3e469d0f9af
+size 234881916

checkpoint/pytorch_model.bin/p186.model.layers.20.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1e72c776776521cf222284fed005cb032899d21ef4ed4135c8e6834a94433b0d
+size 234881910