Instructions to use aws-neuron/SOLAR-10.7B-v1.0-neuron with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="aws-neuron/SOLAR-10.7B-v1.0-neuron")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron")
model = AutoModelForCausalLM.from_pretrained("aws-neuron/SOLAR-10.7B-v1.0-neuron", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aws-neuron/SOLAR-10.7B-v1.0-neuron"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron

SGLang

How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aws-neuron/SOLAR-10.7B-v1.0-neuron" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aws-neuron/SOLAR-10.7B-v1.0-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use aws-neuron/SOLAR-10.7B-v1.0-neuron with Docker Model Runner:
```
docker model run hf.co/aws-neuron/SOLAR-10.7B-v1.0-neuron
```

jburtoft commited on Jan 13, 2024

Commit

31a4535

verified ·

1 Parent(s): 2fffa4f

995e9a82f940d57432ccb02fbd770ffaee43d9c169e0f411b0f090f692904d81

Browse files

Files changed (20) hide show

checkpoint/pytorch_model.bin/p34.model.layers.3.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p340.model.layers.37.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p341.model.layers.37.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p342.model.layers.37.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p343.model.layers.38.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p344.model.layers.38.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p345.model.layers.38.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p346.model.layers.38.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p347.model.layers.38.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p348.model.layers.38.mlp.up_proj.weight +3 -0
checkpoint/pytorch_model.bin/p349.model.layers.38.mlp.down_proj.weight +3 -0
checkpoint/pytorch_model.bin/p35.model.layers.3.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p350.model.layers.38.input_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p351.model.layers.38.post_attention_layernorm.weight +3 -0
checkpoint/pytorch_model.bin/p352.model.layers.39.self_attn.q_proj.weight +3 -0
checkpoint/pytorch_model.bin/p353.model.layers.39.self_attn.k_proj.weight +3 -0
checkpoint/pytorch_model.bin/p354.model.layers.39.self_attn.v_proj.weight +3 -0
checkpoint/pytorch_model.bin/p355.model.layers.39.self_attn.o_proj.weight +3 -0
checkpoint/pytorch_model.bin/p356.model.layers.39.mlp.gate_proj.weight +3 -0
checkpoint/pytorch_model.bin/p357.model.layers.39.mlp.up_proj.weight +3 -0

checkpoint/pytorch_model.bin/p34.model.layers.3.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1a036d25b9d5f1d68f5b896230804d1b682b70fb7eea5982974a40ec8c14d819
+size 234881910

checkpoint/pytorch_model.bin/p340.model.layers.37.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:559e3c9331367c1c58428754aaa124b1461e0e99ca99fa6d5b09f3056de82d4a
+size 234881916

checkpoint/pytorch_model.bin/p341.model.layers.37.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:43c53f52e90d872b1d73d9f36305f099efc3d0699c4377de1099172083777050
+size 17282

checkpoint/pytorch_model.bin/p342.model.layers.37.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1fc331e4005a335899cff5972a2ee6f51d6c36dcb45ff38ea8d13ef14dde5439
+size 17309

checkpoint/pytorch_model.bin/p343.model.layers.38.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:06c136c11d92e5a2d7d475837a988d6eb1489901e4301ccae7bf5d5db54b5122
+size 67109765

checkpoint/pytorch_model.bin/p344.model.layers.38.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:910017accc67b200c53a4370bd82739ac70fae2c2ddc9a72c0ff0319d5f2dc9e
+size 16778117

checkpoint/pytorch_model.bin/p345.model.layers.38.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c68437f601c4708fa04464d0a2977815f4515f7bb199f862ed0e382a321dc1ca
+size 16778117

checkpoint/pytorch_model.bin/p346.model.layers.38.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9e2e09186b2b74290422f8fbf0772efaab7c74501b50fa94846ac4829253ca5e
+size 67109765

checkpoint/pytorch_model.bin/p347.model.layers.38.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:250ccbecef578bfd967666a1ca0c46b201b7ca636e607243687dc3cb5a960bb9
+size 234881916

checkpoint/pytorch_model.bin/p348.model.layers.38.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a42f130c82511bc7392be7c1a76aad57fe90abcca2e46a020fc32fd696edef5e
+size 234881910

checkpoint/pytorch_model.bin/p349.model.layers.38.mlp.down_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:24214598f9fb72ef526ce1b83eadec377ff22c510aa222a21ec9f0a441843076
+size 234881916

checkpoint/pytorch_model.bin/p35.model.layers.3.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0bec347263e06ffb8d8fa29bda967c53ccdcac61b908d19ff82a509cbf209a75
+size 17276

checkpoint/pytorch_model.bin/p350.model.layers.38.input_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9c1713ba669429abb10d65307318ad8ffcf6eca72d233055a8b2257352b13f4d
+size 17282

checkpoint/pytorch_model.bin/p351.model.layers.38.post_attention_layernorm.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3a972d406089a15e9a10032c9ae300fd290bbbeca057796ca38ff8489e555cf0
+size 17309

checkpoint/pytorch_model.bin/p352.model.layers.39.self_attn.q_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cc06e60aa3e5fc26849b3305da9fd709a06a38291dc6b5a366f06d48fddd432c
+size 67109765

checkpoint/pytorch_model.bin/p353.model.layers.39.self_attn.k_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:005174d32cd8b9de2328575d432040fffb69ff8edc3ab5881ea21d59d28933fc
+size 16778117

checkpoint/pytorch_model.bin/p354.model.layers.39.self_attn.v_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ba21e6dfeaa05e899d915f95f399a9c12f1a2c4cb67b1652a8536ea14cd4360e
+size 16778117

checkpoint/pytorch_model.bin/p355.model.layers.39.self_attn.o_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9227497ba37f97e0f281120fb2c62fb1da6e99033ae5fafc2e48a1c29a68c524
+size 67109765

checkpoint/pytorch_model.bin/p356.model.layers.39.mlp.gate_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:14b32291140a4f32cb4b6c39acd34a0c2758bff07db653ee946869de94eed653
+size 234881916

checkpoint/pytorch_model.bin/p357.model.layers.39.mlp.up_proj.weight ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b53ab6146098cad2064a45e6bd919db8b15cd01a0a9a5f43165bb74ebffbc9a1
+size 234881910