Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+figures/step3_bmk.jpeg filter=lfs diff=lfs merge=lfs -text

.mv

@@ -0,0 +1 @@
+Revision:master,CreatedAt:1754151300

LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -0,0 +1,157 @@
+# Introduction
+
+**FlagOS** is a unified heterogeneous computing software stack for large models, co-developed with leading global chip manufacturers. With core technologies such as the **FlagScale** distributed training/inference framework, **FlagGems** universal operator library, **FlagCX** communication library, and **FlagTree** unified compiler, the **FlagRelease** platform leverages the FlagOS stack to automatically produce and release various combinations of <chip + open-source model>. This enables efficient and automated model migration across diverse chips, opening a new chapter for large model deployment and application.
+
+Based on this, the **step3-FlagOS** model is adapted for the Nvidia chip using the FlagOS software stack, enabling:
+
+### Integrated Deployment
+
+- Deep integration with the open-source [FlagScale framework](https://github.com/FlagOpen/FlagScale)
+- Out-of-the-box inference scripts with pre-configured hardware and software parameters
+- Released **FlagOS** container image supporting deployment within minutes
+
+### Consistency Validation
+
+- Rigorously evaluated through benchmark testing: Performance and results from the FlagOS software stack are compared against native stacks on multiple public.
+
+# Technical Overview
+
+## **FlagScale Distributed Training and Inference Framework**
+
+FlagScale is an end-to-end framework for large models across heterogeneous computing resources, maximizing computational efficiency and ensuring model validity through core technologies. Its key advantages include:
+
+- **Unified Deployment Interface:** Standardized command-line tools support one-click service deployment across multiple hardware platforms, significantly reducing adaptation costs in heterogeneous environments.
+- **Intelligent Parallel Optimization:** Automatically generates optimal distributed parallel strategies based on chip computing characteristics, achieving dynamic load balancing of computation/communication resources.
+- **Seamless Operator Switching:** Deep integration with the FlagGems operator library allows high-performance operators to be invoked via environment variables without modifying model code.
+
+## **FlagGems Universal Large-Model Operator Library**
+
+FlagGems is a Triton-based, cross-architecture operator library collaboratively developed with industry partners. Its core strengths include:
+
+- **Full-stack Coverage**: Over 100 operators, with a broader range of operator types than competing libraries.
+- **Ecosystem Compatibility**: Supports 7 accelerator backends. Ongoing optimizations have significantly improved performance.
+- **High Efficiency**: Employs unique code generation and runtime optimization techniques for faster secondary development and better runtime performance compared to alternatives.
+
+## **FlagEval Evaluation Framework**
+
+FlagEval (Libra)** is a comprehensive evaluation system and open platform for large models launched in 2023. It aims to establish scientific, fair, and open benchmarks, methodologies, and tools to help researchers assess model and training algorithm performance. It features:
+  - **Multi-dimensional Evaluation**: Supports 800+ model evaluations across NLP, CV, Audio, and Multimodal fields, covering 20+ downstream tasks including language understanding and image-text generation.
+  - **Industry-Grade Use Cases**: Has completed horizontal evaluations of mainstream large models, providing authoritative benchmarks for chip-model performance validation.
+
+# Evaluation Results
+
+## Benchmark Result 
+
+| Metrics                   | step3-H100-CUDA | step3-FlagOS |
+| ------------------------- | --------------------- | ------------------ |
+|CMMMU|65.69|65.24|
+|MMMU|71.3|70.86|
+|MMLU_Pro_standard|55.36|54.75|
+|MMLU_Pro_vision|48.73|48.77|
+|MM-Vet v2|76.65|76.08|
+|OCRBench|86.26|86.32|
+|MathVision|57.26|56.910|
+|CII-Bench|66.27|65.75|
+|Blink|62.70|62.37|
+
+
+# User Guide
+
+**Environment Setup**
+
+| Item | Version          | 
+| ------------- | ------------------------------------------------------------ |  
+| Docker Version                  | Docker version 28.1.0, build 4d8c241| 
+| Operating System                | Ubuntu 22.04.5 LTS    | 
+| FlagScale                       | Version: 0.8.0                        | 
+| FlagGems                        | Version: 3.0                          | 
+
+## Operation Steps
+
+### Download Open-source Model Weights
+
+```bash
+pip install modelscope
+modelscope download --model stepfun-ai/step3 --local_dir /share/step3
+
+```
+
+### Download FlagOS Image
+
+```bash
+docker pull harbor.baai.ac.cn/flagrelease-public/flagrelease_nvidia_step3
+```
+
+### Start the inference service
+
+```bash
+#Container Startup
+docker run --rm --init --detach   --net=host --uts=host --ipc=host   --security-opt=seccomp=unconfined   --privileged=true   --ulimit stack=67108864   --ulimit memlock=-1   --ulimit nofile=1048576:1048576   --shm-size=32G   -v /share:/share   --gpus all   --name flagos   harbor.baai.ac.cn/flagrelease-public/flagrelease_nvidia_step3   sleep infinity
+```
+
+### Serve
+
+```bash
+flagscale serve step3
+
+```
+
+
+## Service Invocation
+
+### API-based Invocation Script
+
+```bash
+import openai
+openai.api_key = "EMPTY"
+openai.base_url = "http://<server_ip>:9010/v1/"
+model = "step3-nvidia-flagos"
+messages = [
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": "What's the weather like today?"}
+]
+response = openai.chat.completions.create(
+    model=model,
+    messages=messages,
+    stream=False,
+)
+for item in response:
+    print(item)
+
+```
+
+### AnythingLLM Integration Guide
+
+#### 1. Download & Install
+
+- Visit the official site: https://anythingllm.com/
+- Choose the appropriate version for your OS (Windows/macOS/Linux)
+- Follow the installation wizard to complete the setup
+
+#### 2. Configuration
+
+- Launch AnythingLLM
+- Open settings (bottom left, fourth tab)
+- Configure core LLM parameters
+- Click "Save Settings" to apply changes
+
+#### 3. Model Interaction
+
+- After model loading is complete:
+  - Click **"New Conversation"**
+  - Enter your question (e.g., “Explain the basics of quantum computing”)
+  - Click the send button to get a response
+
+# Contributing
+
+We warmly welcome global developers to join us:
+
+1. Submit Issues to report problems
+2. Create Pull Requests to contribute code
+3. Improve technical documentation
+4. Expand hardware adaptation support
+
+
+# License
+
+本模型的权重来源于stepfun-ai/step3，以apache2.0协议https://www.apache.org/licenses/LICENSE-2.0.txt开源。

chat_template.json

@@ -0,0 +1,3 @@
+{
+    "chat_template": "{% macro render_content(content) %} {% if content is string %}{{- content }}{% elif content is mapping %}{{- content['value'] if 'value' in content else content['text'] }}{% elif content is iterable %}{% for item in content %}{% if item.type == 'text' %}{{- item['value'] if 'value' in item else item['text'] }}{% elif item.type == 'image' %}<im_patch>{% endif %}{% endfor %}{% endif %} {% endmacro %}{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{{ bos_token }}{% for message in messages %}{% if message.role == 'system' %}{{ render_content(message['content']) }}{% endif %}{% endfor %}{% if tools is defined and tools %}{% set ns = namespace(data='') %}{% for tool in tools %}{% set ns.data = ns.data + (tool | tojson(ensure_ascii=False)) + '\n' %}{% endfor %}{% set tool_schemas_var = ns.data %}# Tools \nYou may call one or more tools to assist with the user query. You are provided with tool schemas within <tools></tools> XML tags: <tools>{{ tool_schemas_var }}</tools> When making tool calls, use XML format to invoke tools and pass parameters: <｜tool_calls_begin｜>\n<｜tool_call_begin｜>\nfunction<｜tool_sep｜><steptml:invoke name=\"tool_name0\"><steptml:parameter name=\"parameter_name0\">[parameter value]</steptml:parameter>...</steptml:invoke><｜tool_call_end｜>\n<｜tool_call_begin｜>\nfunction<｜tool_sep｜><steptml:invoke name=\"tool_name1\"><steptml:parameter name=\"parameter_name1\">[parameter value]</steptml:parameter>...</steptml:invoke><｜tool_call_end｜>\n<｜tool_calls_end｜>\nNote: * You can invoke one or more tools in parallel. * Each tool call must be complete and self-contained within a single <steptml:toolcall></steptml:toolcall> block. {% endif %}{% for message in messages %}{% if message.role == 'tool_description' %}{{ render_content(message['content']) }}{% elif message.role == 'user' %}{{- '<|BOT|>' + message.role + '\\n' + render_content(message['content']) }}{{- '<|EOT|>' }}{% elif message.role == 'tool_response' %}<｜tool_outputs_begin｜>\n{% for tool_output in message['content'] %}<｜tool_output_begin｜>\n{{ render_content(tool_output) }}<｜tool_output_end｜>{% endfor %}\n<｜tool_outputs_end｜>\n{% else %}{{- '<|BOT|>' + message.role + '\n' }}{% if message['content'] is defined %}{{- render_content(message['content']) }}{% endif %}{% if message.tool_calls is defined %}<｜tool_calls_begin｜>\n{% for tool in message.tool_calls %}<｜tool_call_begin>｜>\n{{ tool['type'] }}<｜tool_sep｜>{{- '<steptml:invoke name=\"' + tool['function']['name'] + '\">' }}{% for name, param in tool['function']['arguments'].items() %} {{- '<steptml:parameter name=\"' + name + '\">' + param | string + '</steptml:parameter>' }}{% endfor %}</steptml:invoke><｜tool_call_end｜>\n{% endfor %}<｜tool_calls_end｜>\n{% endif %}<|EOT|>{% endif %}{% endfor %}{% if add_generation_prompt %}{{- '<|BOT|>assistant\n<think>\n' }}{% endif %}"
+}

config.json

@@ -0,0 +1,53 @@
+{
+  "architectures": [
+    "Step3VLForConditionalGeneration"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_step3.Step3VLConfig",
+    "AutoModelForCausalLM": "modeling_step3.Step3vForConditionalGeneration"
+  },
+  "model_type": "step3_vl",
+  "im_end_token": "<im_end>",
+  "im_patch_token": "<im_patch>",
+  "im_start_token": "<im_start>",
+  "image_token_len": 169,
+  "patch_token_len": 81,
+  "understand_projector_stride": 2,
+  "projector_bias": false,
+  "image_token_id": 128001,
+  "bos_token_id": 0,
+  "eos_token_id": 128805,
+  "text_config": {
+    "architectures": [
+      "Step3TextForCausalLM"
+    ],
+    "model_type": "step3_text",
+    "hidden_size": 7168,
+    "intermediate_size": 18432,
+    "num_hidden_layers": 61,
+    "max_seq_len": 65536,
+    "max_position_embedding": 65536,
+    "vocab_size": 128815,
+    "torch_dtype": "bfloat16",
+    "moe_layers_enum": "4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59",
+    "num_attention_heads": 64,
+    "num_attention_groups": 1,
+    "head_dim": 256,
+    "share_q_dim": 2048,
+    "moe_num_experts": 48,
+    "moe_top_k": 3,
+    "moe_intermediate_size": 5120,
+    "share_expert_dim": 5120,
+    "norm_expert_weight": false,
+    "rope_theta": 500000
+  },
+  "vision_config": {
+    "hidden_size": 1792,
+    "output_hidden_size": 4096,
+    "image_size": 728,
+    "intermediate_size": 15360,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 63,
+    "patch_size": 14
+  }
+}

configuration.json

@@ -0,0 +1 @@
+{"framework":"Pytorch","task":"image-text-to-text"}

configuration_step3.py

@@ -0,0 +1,120 @@
+from typing import Any, Optional, Union
+
+from transformers.configuration_utils import PretrainedConfig
+
+
+class Step3VisionEncoderConfig(PretrainedConfig):
+    model_type = "step3_vision_encoder"
+
+    def __init__(
+        self,
+        hidden_size=1792,
+        intermediate_size=3072,
+        output_hidden_size=4096,
+        num_hidden_layers=63,
+        num_attention_heads=16,
+        num_channels=3,
+        image_size=728,
+        patch_size=14,
+        hidden_act="quick_gelu",
+        layer_norm_eps=1e-5,
+        **kwargs,
+    ):
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.output_hidden_size = output_hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_channels = num_channels
+        self.patch_size = patch_size
+        self.image_size = image_size
+        self.layer_norm_eps = layer_norm_eps
+        self.hidden_act = hidden_act
+        super().__init__(**kwargs)
+
+
+class Step3TextConfig(PretrainedConfig):
+    model_type = "step3_text"
+    architectures = ["Step3TextForCausalLM"]
+
+    def __init__(
+        self,
+        hidden_size: int = 7168,
+        intermediate_size: int = 18432,
+        num_attention_heads: int = 64,
+        num_attention_groups: int = 1,
+        num_hidden_layers: int = 61,
+        max_seq_len: int = 65536,
+        vocab_size: int = 128815,
+        rms_norm_eps: float = 1e-5,
+        moe_intermediate_size: int = 5120,
+        moe_num_experts: int = 48,
+        moe_top_k: int = 3,
+        rope_theta: float = 500000,
+        rope_scaling: Optional[dict[str, Any]] = None,
+        max_position_embedding: int = 65536,
+        share_expert_dim: int = 5120,
+        share_q_dim: int = 2048,
+        head_dim: int = 256,
+        norm_expert_weight: bool = False,
+        moe_layers_enum: tuple[int] = (4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
+                                       15, 16, 17, 18, 19, 20, 21, 22, 23, 24,
+                                       25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
+                                       35, 36, 37, 38, 39, 40, 41, 42, 43, 44,
+                                       45, 46, 47, 48, 49, 50, 51, 52, 53, 54,
+                                       55, 56, 57, 58, 59),
+        **kwargs,
+    ) -> None:
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_attention_heads = num_attention_heads
+        self.num_attention_groups = num_attention_groups
+        self.num_hidden_layers = num_hidden_layers
+        self.max_seq_len = max_seq_len
+        self.vocab_size = vocab_size
+        self.rms_norm_eps = rms_norm_eps
+        self.moe_intermediate_size = moe_intermediate_size
+        self.moe_num_experts = moe_num_experts
+        self.moe_top_k = moe_top_k
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.max_position_embedding = max_position_embedding
+        self.share_expert_dim = share_expert_dim
+        self.share_q_dim = share_q_dim
+        self.head_dim = head_dim
+        self.norm_expert_weight = norm_expert_weight
+        self.moe_layers_enum = moe_layers_enum
+
+        super().__init__(**kwargs)
+
+
+class Step3VLConfig(PretrainedConfig):
+    model_type = "step3_vl"
+
+    def __init__(
+        self,
+        vision_config: Optional[Union[dict, Step3VisionEncoderConfig]] = None,
+        text_config: Optional[Union[dict, Step3TextConfig]] = None,
+        understand_projector_stride: int = 1,
+        projector_bias: bool = True,
+        image_token_id: int = 128001,
+        **kwargs,
+    ) -> None:
+        if vision_config is None:
+            vision_config = Step3VisionEncoderConfig()
+        elif isinstance(vision_config, dict):
+            vision_config = Step3VisionEncoderConfig(**vision_config)
+        self.vision_config = vision_config
+
+        if text_config is None:
+            text_config = Step3TextConfig()
+        elif isinstance(text_config, dict):
+            text_config = Step3TextConfig(**text_config)
+        self.text_config = text_config
+
+        self.understand_projector_stride = understand_projector_stride
+        self.projector_bias = projector_bias
+        self.hidden_size = text_config.hidden_size
+        self.image_token_id = image_token_id
+
+        super().__init__(**kwargs)

docs/deploy_guidance.md

@@ -0,0 +1,226 @@
+# Step3 Model Deployment Guide
+
+This document provides deployment guidance for Step3 model.
+
+Currently, our open-source deployment guide only includes TP and DP+TP deployment methods. The AFD (Attn-FFN Disaggregated) approach mentioned in our [paper](https://arxiv.org/abs/2507.19427) is still under joint development with the open-source community to achieve optimal performance. Please stay tuned for updates on our open-source progress.
+
+## Overview
+
+Step3 is a 321B-parameter VLM with hardware-aware model-system co-design optimized for minimizing decoding costs. 
+
+For out fp8 version, about 326G memory is required.
+The smallest deployment unit for this version is 8xH20 with either Tensor Parallel (TP) or Data Parallel + Tensor Parallel (DP+TP).
+
+For out bf16 version, about 642G memory is required.
+The smallest deployment unit for this version is 16xH20 with either Tensor Parallel (TP) or Data Parallel + Tensor Parallel (DP+TP).
+
+## Deployment Options
+
+### vLLM Deployment
+
+Please make sure to use nightly version of vllm after this [PR](https://github.com/vllm-project/vllm/pull/21998) is merged. For details, please refer to [vllm nightly installation doc](https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#pre-built-wheels).
+```bash
+uv pip install -U vllm \
+    --torch-backend=auto \
+    --extra-index-url https://wheels.vllm.ai/nightly
+```
+
+We recommend to use the following command to deploy the model:
+
+**`max_num_batched_tokens` should be larger than 4096. If not set, the default value is 8192.**
+
+#### BF16 Model
+##### Tensor Parallelism(Serving on 16xH20):
+
+```bash
+# start ray on node 0 and node 1
+
+# node 0:
+vllm serve /path/to/step3 \
+    --tensor-parallel-size 16 \
+    --reasoning-parser step3 \
+    --enable-auto-tool-choice \
+    --tool-call-parser step3 \
+    --trust-remote-code \
+    --max-num-batched-tokens 4096 \
+    --port $PORT_SERVING
+```
+
+###### Data Parallelism + Tensor Parallelism(Serving on 16xH20):
+Step3 only has single kv head, so attention data parallelism can be adopted to reduce the kv cache memory usage.
+
+```bash
+# start ray on node 0 and node 1
+
+# node 0:
+vllm serve /path/to/step3 \
+    --data-parallel-size 16 \
+    --tensor-parallel-size 1 \
+    --reasoning-parser step3 \
+    --enable-auto-tool-choice \
+    --tool-call-parser step3 \
+    --max-num-batched-tokens 4096 \
+    --trust-remote-code \
+```
+
+#### FP8 Model
+##### Tensor Parallelism(Serving on 8xH20):
+
+```bash
+vllm serve /path/to/step3-fp8 \
+    --tensor-parallel-size 8 \
+    --reasoning-parser step3 \
+    --enable-auto-tool-choice \
+    --tool-call-parser step3 \
+    --gpu-memory-utilization 0.85 \
+    --max-num-batched-tokens 4096 \
+    --trust-remote-code \
+```
+
+###### Data Parallelism + Tensor Parallelism(Serving on 8xH20):
+
+```bash
+vllm serve /path/to/step3-fp8 \
+    --data-parallel-size 8 \
+    --tensor-parallel-size 1 \
+    --reasoning-parser step3 \
+    --enable-auto-tool-choice \
+    --tool-call-parser step3 \
+    --max-num-batched-tokens 4096 \
+    --trust-remote-code \
+```
+
+
+##### Key parameter notes:
+
+* `reasoning-parser`: If enabled, reasoning content in the response will be parsed into a structured format.
+* `tool-call-parser`: If enabled, tool call content in the response will be parsed into a structured format.
+
+### SGLang Deployment
+
+0.4.10 or later is needed for SGLang.
+
+```
+pip3 install "sglang[all]>=0.4.10"
+```
+
+#### BF16 Model
+##### Tensor Parallelism(Serving on 16xH20):
+
+```bash
+# node 1
+python -m sglang.launch_server \
+ --model-path stepfun-ai/step3 \
+ --dist-init-addr master_ip:5000 \
+ --trust-remote-code \
+ --tool-call-parser step3 \
+ --reasoning-parser step3 \
+ --tp 16 \
+ --nnodes 2 \
+ --node-rank 0
+
+# node 2
+python -m sglang.launch_server \
+ --model-path stepfun-ai/step3 \
+ --dist-init-addr master_ip:5000 \
+ --trust-remote-code \
+ --tool-call-parser step3 \
+ --reasoning-parser step3 \
+ --tp 16 \
+ --nnodes 2 \
+ --node-rank 1
+```
+
+#### FP8 Model
+##### Tensor Parallelism(Serving on 8xH20):
+
+```bash
+python -m sglang.launch_server \
+    --model-path /path/to/step3-fp8 \
+    --trust-remote-code \
+    --tool-call-parser step3 \
+    --reasoning-parser step3 \
+    --tp 8
+```
+
+
+### TensorRT-LLM Deployment
+
+[Coming soon...]
+
+
+## Client Request Examples
+
+Then you can use the chat API as below:
+```python
+from openai import OpenAI
+
+# Set OpenAI's API key and API base to use vLLM's API server.
+openai_api_key = "EMPTY"
+openai_api_base = "http://localhost:8000/v1"
+
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
+
+chat_response = client.chat.completions.create(
+    model="step3",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": "https://xxxxx.png"
+                    },
+                },
+                {"type": "text", "text": "Please describe the image."},
+            ],
+        },
+    ],
+)
+print("Chat response:", chat_response)
+```
+You can also upload base64-encoded local images:
+
+```python
+import base64
+from openai import OpenAI
+# Set OpenAI's API key and API base to use vLLM's API server.
+openai_api_key = "EMPTY"
+openai_api_base = "http://localhost:8000/v1"
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
+image_path = "/path/to/local/image.png"
+with open(image_path, "rb") as f:
+    encoded_image = base64.b64encode(f.read())
+encoded_image_text = encoded_image.decode("utf-8")
+base64_step = f"data:image;base64,{encoded_image_text}"
+chat_response = client.chat.completions.create(
+    model="step3",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {
+            "role": "user",
+            "content": [
+                {
+                    "type": "image_url",
+                    "image_url": {
+                        "url": base64_step
+                    },
+                },
+                {"type": "text", "text": "Please describe the image."},
+            ],
+        },
+    ],
+)
+print("Chat response:", chat_response)
+
+```
+
+Note: In our image preprocessing pipeline, we implement a multi-patch mechanism to handle large images. If the input image exceeds 728x728 pixels, the system will automatically apply image cropping logic to get patches of the image.

generation_config.json

@@ -0,0 +1,8 @@
+{
+    "bos_token_id": 0,
+    "eos_token_id": 128805,
+    "temperature": 0.7,
+    "top_p": 0.95,
+    "do_sample": true,
+    "transformers_version": "4.54.0"
+}

model-00001.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d116ae88ed10f56742606e2da7818e1533988297a3006cd6791182059a9b3226
+size 9955092120

model-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05e8bc428a78e1781d7eeaef0cf32a547d7a4189178f4e84675e3f7aa9c9790c
+size 9957541368

model-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35e2a0d26a1131a46a6df4deaf7337d8dbdbd9002809728b55c978b28960fadb
+size 9966049896

model-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a94ccd8c0c564e9df04a693208435f9b40736ef0e00d3d4c2891e2c7fe50c65e
+size 9975131928

model-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f2012990f05f356635c3856aee2dbf1fa383ce641a9e0c1a547bb37cc4674d2
+size 9921772592

model-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44a5a3f511975156d039cf531e42cfbd5ecbdffb8ff0d29b29d8c7d4ca6cf05b
+size 7310672224

model-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f9242859c3de930b9d2535807edb1a253c6b4759f1160bf35b717d33ee66869
+size 7046430952

model-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9197980060dd760b08efde286c40e165ed0bdd8f4c9cfea7152b1c38260ec194
+size 7046430952

model-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:85e6923a6705304f11c9dd3c11255f1d688e01a4f51514189404463637a6b021
+size 7046430960

model-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09c6aeba446a0f4a0c0fd9c47f712d60f9805ab251cb2790c0288d4275141145
+size 7046430952

model-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cebf56573462ed491301b008bc57d3b25f2665d9a7aaf6f6a2265bb108992244
+size 7046430952

model-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e839654a313b7d78cae825323f9a06f6d6c394fabfea353fa9e76c799339269
+size 7046430960

model-00013.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c791af2fe41f79cda4e278dc891bdc2a0753b14b9afd475f27ff810d0f3965a4
+size 7046430952

model-00014.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3380f3e2160f3733feddfdd94b94f75e9e85131a5797ba0e1bf4f2065c54a25f
+size 7046430952

model-00015.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8f522596a3bea36902c309ffaf864cd640cc275978f10c006a78a816bbbfa49
+size 7046430960

model-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81cbf3a31597fc070cf9f13419a7df1f18949e0fc8086780fad4de390f0c5d32
+size 7046430960

model-00017.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1bd37184a4d54b3e303eb0ea3b8d3f3e396adaffb993d49097edd5f2818912a6
+size 7046430960

model-00018.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:543462296f44f57710405ee5df48dccdc7a28ec75170b6111b841d17a5fcdfd9
+size 7046430960

model-00019.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03bdba8a5242c255127bc887c2159ae76e947e8fa9eb9474736be166c4e98b6c
+size 7046430960

model-00020.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f07cc863fc86cc9eb5d1bab06f593de41c5cd36a9231f4edc133c07ce1aefd74
+size 7046430960

model-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8cac9d71d0ac36a460f3bf39da1f95d4335c02d9b6e9332e3042adbbfeba1a1
+size 7046430960

model-00022.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5db4f8395f0a5c71b26a227b04ad5a663b31e178a6da1ede1a243fc8c418e25
+size 7046430960

model-00023.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:229ba45a9cd8b8792a5b83b7148ca13f8c4cb3f2443546a3aa4e4d3f53bb6291
+size 7046430960

model-00024.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:095e4450b4c83f545ad450fc39b9c7d995f7f842b8c7802dc7485e4817ac46a0
+size 7046430960

model-00025.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef8050270d6c6a289cd3889a7b6aaed62fe50380ada7b887cbd3b4a7bed0d55e
+size 7046430960

model-00026.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f49a63250dc25e5d3f25dd1d002722f203b8a7e5d4c7742e3bf0690a1c1632e4
+size 7046430960

model-00027.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45dcbc3771b950d73ac97ce54d20b22090551fea9a4cd37bc4d953261c5cd5dd
+size 7046430960

model-00028.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f76ce1a53cfb59a0afb3bfd6ce5deabf6bbc3a84aaa0379e29d93d09e1a0a326
+size 7046430960

model-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0add3917e46cc0be92ac9230a413ff010a4437695ccc4df7ac056f085984788d
+size 7046430960

model-00030.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd1a4e9becd74c663a24dcb043bfda7797c4127368dd58eefe85ba09610dd73e
+size 7046430960

model-00031.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ff8d05fd4803bbbbd685ada90bda51c6b99bb2b12a8dbc36910eb0ef9dc7803
+size 7046430960

model-00032.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb58c7424bb3fb0c0d5c84f4cb7b220deb73cd677a5bb509ab782567a1f8a687
+size 7046430960

model-00033.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eac34581b2fb405be199b99b71356b60caf1f2c64ad5fe92dd14a3aa6b156ba3
+size 7046430960

model-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f9fc7d48f75050201c785b5973c8517fd4c82a95787772ec9e48fad400e4233
+size 7046430960

model-00035.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d32bffd85be89f836ca36ed7201aaa676b93a4db9ca9be4fd36094dc93d16d3b
+size 7046430960

model-00036.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:beb6e50876d8477768af98c35fb7268dff1897e037d7506b3d295547d75bcfcc
+size 7046430960

model-00037.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:756125d247a87695e01627d274078d8e20b4f7e45a2455021c4c287e613062ff
+size 7046430960