First commit

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+src/mteb_res_v2.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,148 @@
 ---
+tags:
+- mteb
+- sentence-transformers
+- transformers
+- sentence-similarity
+language:
+- en
+- zh
 license: apache-2.0
 ---
+
+# Conan-Embedding-v2
+
+## What's New?
+
+- **Performance**
+  
+  Conan-Embedding-v2 has now achieved SOTA performance on the MTEB leaderboard for both Chinese and English.
+  
+- **Cross-lingual Retrieval between Chinese and English**
+  
+  Conan-Embedding-v2 supports cross-lingual retrieval between Chinese and English samples.
+  
+- **Longer Context Support**
+  
+  Conan-Embedding-v2 now supports a context length of 32,768 tokens.
+  
+- **Conan 1.4B Large Model Trained from Scratch**
+  
+  A vocabulary and large language model trained from scratch, with a pre-trained model and vocabulary more tailored to the Embedding scenario, delivering stronger performance. 
+  
+  The Conan-1.4B base model will be open-sourced. Community workers can train their own Embedding models based on the Conan-1.4B base model.
+  
+
+## Performance
+
+Performance of Conan-Embedding-v2 on MTEB for Chinese and English
+
+![MTEB Result](./src/mteb_res_v2.png)
+
+**English**
+
+| Embedding TaskMertric   | Class. Acc. (12) | Clust V-Meas. (11) | PairClass AP (3) | Rerank MAP (4) | Retri nDCG @ 10 (15) | STS Spear. (12) | SummSpear. (1) | Avg.(56)  |
+|:-----------------------:|:----------------:|:------------------:|:----------------:|:--------------:|:--------------------:|:---------------:|:--------------:|:---------:|
+| bge-multilingual-gemma2 | 88.08            | 54.65              | 85.97            | 59.72          | 59.24                | 83.88           | 31.20          | 69.88     |
+| e5-mistral-7b-instruct  | 79.89            | 51.44              | 88.42            | 49.78          | 57.62                | 84.32           | **36.57**      | 67.98     |
+| gte-Qwen2-7B-instruct   | 86.58            | 56.92              | 85.90            | **61.42**      | 59.11                | 83.06           | 31.35          | 69.95     |
+| stella-en-1.5B-v5       | 87.63            | 57.69              | 88.07            | 61.21          | 61.01                | 84.51           | 31.49          | 71.19     |
+| bge-en-icl              | 88.95            | 57.89              | 88.14            | 59.86          | 62.16                | 84.24           | 30.77          | 71.67     |
+| NV-Embed-v2             | **90.37**        | 58.46              | 88.67            | 60.65          | 62.65                | 84.31           | 30.70          | 72.31     |
+| **Conan-embedding-v2**  | 90.15            | **60.86**          | **93.47**        | 60.89          | **66.40**            | **85.73**       | 28.08          | **74.22** |
+
+**Chinese**
+
+| Embedding TaskMertric   | Class.Acc. (9) | ClustV-Meas. (4) | PairClassAP (2) | RerankMAP (4) | RetrinDCG @ 10 (8) | STSSpear. (8) | Avg.(35)  |
+|:-----------------------:|:--------------:|:----------------:|:---------------:|:-------------:|:------------------:|:-------------:|:---------:|
+| e5-mistral-7b-instruct  | 72.96          | 52.30            | 72.19           | 61.86         | 61.75              | 48.34         | 59.92     |
+| gte-Qwen2-1.5B-instruct | 72.53          | 54.61            | 86.91           | 68.21         | 71.86              | 60.05         | 67.12     |
+| bge-multilingual-gemma2 | 75.31          | 59.30            | 86.67           | 68.28         | 73.73              | 55.19         | 67.64     |
+| gte-Qwen2-7B-instruct   | 75.77          | 66.06            | 87.48           | 68.92         | 75.71              | 65.20         | 71.62     |
+| xiaobu-embedding-v2     | 76.53          | 65.17            | 91.87           | 72.58         | 76.50              | 64.18         | 72.36     |
+| Conan-embedding-v1      | **76.77**      | 66.33            | 91.66           | 72.76         | 76.67              | 63.67         | 72.50     |
+| **Conan-embedding-v2**  | 76.47          | **68.84**        | **92.44**       | **74.41**     | **78.31**          | **65.48**     | **74.24** |
+
+
+## Model Detail
+
+### Model Structure
+
+**Conan-Embedding-v2 Structure:**
+
+```
+SentenceTransformer(  
+    (0): Transformer({
+        'max_seq_length': 32768, 
+        'do_lower_case': False
+        }) with Transformer model: ConanEmbedModel,
+    (1): Pooling({
+        'word_embedding_dimension': 3584, 
+        'pooling_mode_cls_token': False, 
+        'pooling_mode_mean_tokens': True, 
+        'pooling_mode_max_tokens': False, 
+        'pooling_mode_mean_sqrt_len_tokens': False, 
+        'pooling_mode_weightedmean_tokens': False, 
+        'pooling_mode_lasttoken': False, 
+        'include_prompt': True
+        }),
+    (2): Dense({
+        'in_features': 3584, 
+        'out_features': 3584, 
+        'bias': True, 
+        'activation_function': 'torch.nn.modules.linear.Identity'
+        })
+)
+```
+
+**Key Specifications of Conan-1.4B (Transformer):**
+
+- Number of Parameters (Non-Dense-Layer): 1.48B 
+  
+- Vocabulary Size: 150,000
+  
+- Number of Layers: 8
+  
+- Hidden Layer Dimension: 3584
+  
+- Number of Attention Heads (GOA): 32 for Q and 8 for KV
+  
+- Intermediate Dimension of FFN Layer: 8192
+  
+- Maximum Context Window: 32,768 Tokens
+
+For more model details, please refer to ```model/modeling_conan.py``` and ```config.json```, or stay tuned for the upcoming open-source release of Conan-1.4B Base Model.
+
+### Tokenizer
+
+We trained the Tokenizer on a large-scale multilingual dataset to build a standard BBPE(Byte-level Byte Pair Encoding) tokenizer with a vocabulary size of 150,000.
+
+## Technical Report
+
+We will soon release our technical report.
+
+## Using Conan-Embedding-v2
+
+Use ```/model/conan_api_client.py``` to access our test API. A sample call is as follows:
+
+```
+from modeling_conan import ConanClient
+
+AK = os.getenv("CONAN_AK")
+SK = os.getenv("CONAN_SK")
+client = ConanClient(ak=AK, sk=SK, url="https://ai.om.qq.com/api/conan/v2")
+res = client.embed("Hello!")
+print(res)
+
+```
+
+This is a temporary calling solution. Please contact us to obtain an access token.
+
+In the future, we will provide high-performance, cost-effective, and reliable Embedding services on Tencent Cloud.
+
+
+---
+
+**About**
+
+Created by the Tencent BAC Group. All rights reserved.

model/conan_api_client.py

@@ -0,0 +1,66 @@
+import os
+import hashlib
+import random
+import string
+import time
+import requests
+import json
+
+class ConanClient:
+    def __init__(self, ak, sk, url, timeout=30):
+        """
+        Initialize the Client
+
+        Args:
+            ak: Access Key
+            sk: Secret Key
+            url: Server URL
+            timeout: Request timeout in seconds (default: 30)
+        """
+        self.ak = ak
+        self.sk = sk
+        self.url = url
+        self.timeout = timeout
+
+    def __random_password(self, size=40, chars=None):
+        if chars is None:
+            chars = string.ascii_uppercase + string.ascii_lowercase + string.digits
+        random_chars = random.SystemRandom().choice
+        return "".join(random_chars(chars) for _ in range(size))
+
+    def __signature(self, random_str, time_stamp):
+        params_str = "%s:%d:%s:%s" % (self.ak, time_stamp, random_str, self.sk)
+        encoded_params_str = params_str.encode("utf-8")
+        return hashlib.md5(encoded_params_str).hexdigest()
+
+    def get_signature(self):
+        timestamp = int(time.time())
+        random_str = self.__random_password(20)
+        sig = self.__signature(random_str, timestamp)
+        params = {
+            "timestamp": timestamp,
+            "random": random_str,
+            "app_id": self.ak,
+            "sign": sig,
+        }
+        return params
+
+    def embed(self, text):
+        """
+        Embed text using the server
+
+        Args:
+            text: The input text to embed
+
+        Returns:
+            requests.Response object
+        """
+        params = self.get_signature()
+        params["body"] = text
+        params["content_id"] = f"test_{int(time.time())}"
+
+        headers = {"Content-Type": "application/json"}
+
+        rsp = requests.post(self.url, data=json.dumps(params), timeout=self.timeout, headers=headers)
+        result = json.loads(rsp.text)
+        return result

model/config.json

@@ -0,0 +1,40 @@
+{
+  "_name_or_path": "Conan-embedding-v2",
+  "add_eos": true,
+  "add_pad_token": true,
+  "architectures": [
+    "ConanEmbedModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_conan.ConanEmbedConfig",
+    "AutoModel": "modeling_conan.ConanEmbedModel"
+  },
+  "bos_token_id": 0,
+  "do_dir": true,
+  "eos_token_id": 1,
+  "hidden_act": "silu",
+  "hidden_size": 3584,
+  "initializer_range": 0.02,
+  "intermediate_size": 8192,
+  "is_mask_instruction": true,
+  "mask_type": "b",
+  "max_position_embeddings": 32768,
+  "mlp_bias": false,
+  "model_type": "Conan-embedding-v2",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 8,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "padding_side": "right",
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "sentence_pooling_method": "mean",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.41.2",
+  "use_cache": true,
+  "vocab_size": 150000
+}

model/modeling_conan.py

@@ -0,0 +1,435 @@
+from typing import Union, Mapping, Optional, Tuple, TypedDict, Dict, List
+from functools import partial
+
+import torch
+import numpy as np
+from transformers import AutoModel, PreTrainedTokenizerFast, BatchEncoding, DataCollatorWithPadding
+from transformers.models.auto import AutoTokenizer
+from transformers.models.llama.modeling_llama import LLAMA_INPUTS_DOCSTRING
+from transformers.modeling_outputs import BaseModelOutputWithPast
+from transformers.modeling_attn_mask_utils import _prepare_4d_attention_mask, _prepare_4d_attention_mask_for_sdpa
+from transformers import LlamaModel
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.utils import (
+    add_start_docstrings_to_model_forward,
+    logging,
+)
+from tqdm.auto import tqdm
+from datasets import Dataset
+from torch.utils.data import DataLoader
+from .configuration_conan import ConanEmbedConfig
+
+logger = logging.get_logger(__name__)
+
+
+class ConanEmbedFeatures(TypedDict):
+    input_dict: torch.Tensor
+    attention_mask: torch.Tensor
+    pool_mask: torch.Tensor
+
+
+def _move_to_device(maybe_tensor, device: torch.device):
+    if torch.is_tensor(maybe_tensor):
+        return maybe_tensor.to(device, non_blocking=device.type == "cuda")
+    elif isinstance(maybe_tensor, dict):
+        return {key: _move_to_device(value, device) for key, value in maybe_tensor.items()}
+    elif isinstance(maybe_tensor, list):
+        return [_move_to_device(x, device) for x in maybe_tensor]
+    elif isinstance(maybe_tensor, tuple):
+        return tuple([_move_to_device(x, device) for x in maybe_tensor])
+    elif isinstance(maybe_tensor, Mapping):
+        return type(maybe_tensor)({k: _move_to_device(v, device) for k, v in maybe_tensor.items()})
+    else:
+        return maybe_tensor
+
+
+def move_to_device(sample, device: torch.device):
+    if device.type == "cpu":
+        return sample
+
+    if len(sample) == 0:
+        return {}
+    return _move_to_device(sample, device)
+
+
+def input_transform_func(
+    tokenizer: PreTrainedTokenizerFast,
+    examples: Dict[str, List],
+    always_add_eos: bool,
+    max_length: int,
+    instruction: str,
+) -> BatchEncoding:
+    if always_add_eos:
+        examples["input_texts"] = [
+            instruction + input_example + tokenizer.eos_token for input_example in examples["input_texts"]
+        ]
+    print(examples["input_texts"])
+    batch_dict = tokenizer(
+        examples["input_texts"],
+        max_length=max_length,
+        padding=True,
+        return_token_type_ids=False,
+        return_tensors="pt",
+        truncation=True,
+    )
+    print(examples["input_texts"])
+    return batch_dict
+
+
+class ConanEmbedModel(LlamaModel):
+    config_class = ConanEmbedConfig
+
+    def __init__(self, config: ConanEmbedConfig) -> None:
+        """
+        Initialize the model with a given configuration.
+
+        Args:
+            config (ConanEmbedConfig): The configuration for the model.
+        """
+        super().__init__(config)
+        for layer in self.layers:
+            layer.self_attn.is_causal = not config.do_dir
+        self._attn_implementation = "eager"
+        self.tokenizer = AutoTokenizer.from_pretrained(config._name_or_path)
+        self.padding_side = config.padding_side
+        self.is_mask_instruction = config.is_mask_instruction
+        self.add_eos = config.add_eos
+        self.mask_type = config.mask_type
+        self.sentence_pooling_method = config.sentence_pooling_method
+        if config.add_pad_token and self.tokenizer is not None:
+            self.add_pad_token()
+
+    def add_pad_token(self):
+        self.tokenizer.pad_token = self.tokenizer.eos_token
+        self.tokenizer.padding_side = self.padding_side
+
+    def _sentence_embedding(self, last_hidden_state, attention_mask=None):
+        """Use the pooling method to get the sentence embedding.
+
+        Args:
+            last_hidden_state (torch.Tensor): The model output's last hidden state.
+            attention_mask (torch.Tensor): Mask out padding tokens during pooling.
+
+        Raises:
+            NotImplementedError: Specified pooling method not implemented.
+
+        Returns:
+            torch.Tensor: The sentence embeddings.
+        """
+        if self.sentence_pooling_method == "cls":
+            return last_hidden_state[:, 0]
+        elif self.sentence_pooling_method == "mean":
+            s = torch.sum(last_hidden_state, dim=1)
+            # d = attention_mask.sum(dim=1, keepdim=True).float()
+            return s
+        elif self.sentence_pooling_method == "last_token":
+            left_padding = attention_mask[:, -1].sum() == attention_mask.shape[0]
+            if left_padding:
+                return last_hidden_state[:, -1]
+            else:
+                sequence_lengths = attention_mask.sum(dim=1) - 1
+                batch_size = last_hidden_state.shape[0]
+                return last_hidden_state[
+                    torch.arange(batch_size, device=last_hidden_state.device),
+                    sequence_lengths,
+                ]
+        else:
+            raise NotImplementedError(f"pooling method {self.sentence_pooling_method} not implemented")
+
+    def prepare_kwargs_from_batch(
+        self,
+        batch_dict: Dict[str, torch.Tensor],
+        instruction_lens: int,
+        device: torch.device,
+    ) -> ConanEmbedFeatures:
+        """
+        Prepare the batch dictionary for encoding.
+
+        Args:
+            batch_dict: A dictionary containing the input_ids and attention_mask.
+            instruction_lens: The length of the instruction.
+            device: The device to move the data to.
+
+        Returns:
+            A ConanEmbedFeatures object with the prepared input_ids and attention_mask.
+        """
+        batch_dict = move_to_device(batch_dict, device)
+        attention_mask = batch_dict["attention_mask"].clone() if "attention_mask" in batch_dict else None
+        if (
+            attention_mask is not None
+            and self.padding_side == "right"
+            and self.is_mask_instruction
+            and instruction_lens > 0
+        ):
+            # Mask out the instruction tokens for mean-pooling
+            attention_mask[:, :instruction_lens] = 0
+        features: ConanEmbedFeatures = {
+            "input_ids": torch.tensor(batch_dict.get("input_ids").to(batch_dict.get("input_ids")).long()),
+            "attention_mask": batch_dict["attention_mask"],
+        }
+        return features
+
+    @torch.no_grad()
+    def _do_encode(
+        self,
+        prompts: List[str],
+        batch_size: int = 1,
+        instruction: str = "",
+        max_length: int = 4096,
+        num_workers: int = 32,
+        return_numpy: bool = False,
+    ) -> Union[torch.FloatTensor, np.ndarray]:
+        """
+        Encode a list of prompts using the model.
+
+        Args:
+            prompts: A list of prompts to encode.
+            batch_size: The batch size to use for encoding. Defaults to 1.
+            instruction: An instruction to prepend to the prompts. Defaults to "".
+            max_length: The maximum length of the input_ids. Defaults to 4096.
+            num_workers: The number of workers to use for encoding. Defaults to 32.
+            return_numpy: Whether to return the output as a numpy array or a torch tensor. Defaults to False.
+
+        Returns:
+            A tensor or numpy array of shape (len(prompts), hidden_size) containing the encoded prompts.
+        """
+        dataset: Dataset = Dataset.from_dict({"input_texts": prompts})
+        dataset.set_transform(
+            partial(
+                input_transform_func,
+                self.tokenizer,
+                always_add_eos=True,
+                max_length=max_length,
+                instruction=instruction,
+            )
+        )
+
+        data_collator = DataCollatorWithPadding(self.tokenizer)
+        data_loader = DataLoader(
+            dataset,
+            batch_size=batch_size,
+            shuffle=False,
+            drop_last=False,
+            num_workers=num_workers,
+            collate_fn=data_collator,
+            pin_memory=True,
+        )
+
+        if self.padding_side == "right" and self.is_mask_instruction and len(instruction) > 0:
+            instruction_lens = len(self.tokenizer.tokenize(instruction))
+        else:
+            instruction_lens = 0
+
+        encoded_embeds: List[torch.Tensor] = []
+        device = next(self.parameters()).device
+        for batch_dict in tqdm(data_loader, desc="encoding", mininterval=10):
+            features = self.prepare_kwargs_from_batch(batch_dict, instruction_lens, device=device)
+            embeds = self(**features)["sentence_embeddings"].squeeze(1)
+            encoded_embeds.append(embeds)
+        encoded_embeds = torch.cat(encoded_embeds, axis=0)
+        if return_numpy:
+            encoded_embeds = encoded_embeds.cpu().detach().numpy()
+        return encoded_embeds
+
+    @add_start_docstrings_to_model_forward(LLAMA_INPUTS_DOCSTRING)
+    def forward(
+        self,
+        input_ids: torch.LongTensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        token_type_ids: Optional[torch.LongTensor] = None,
+    ) -> Union[Tuple[torch.Tensor, ...], BaseModelOutputWithPast]:
+        """
+        Args:
+            input_ids: a tensor of shape (batch_size, sequence_length)
+            attention_mask: a tensor of shape (batch_size, sequence_length)
+            position_ids: a tensor of shape (batch_size, sequence_length)
+            past_key_values: a list of tensors of shape (batch_size, key_length, hidden_size)
+            inputs_embeds: a tensor of shape (batch_size, sequence_length, hidden_size)
+            use_cache: a boolean indicating whether to use the cache
+            output_attentions: a boolean indicating whether to output the attention weights
+            output_hidden_states: a boolean indicating whether to output the hidden states
+            return_dict: a boolean indicating whether to return a dictionary
+
+        Returns:
+            a tuple of length 4 containing the last hidden state, the cache, the hidden states,
+                and the attention weights
+            or a BaseModelOutputWithPast object
+        """
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        # retrieve input_ids and inputs_embeds
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time")
+        elif input_ids is not None:
+            batch_size, seq_length = input_ids.shape
+        elif inputs_embeds is not None:
+            batch_size, seq_length, _ = inputs_embeds.shape
+        else:
+            raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds")
+
+        if self.gradient_checkpointing and self.training:
+            if use_cache:
+                logger.warning_once(
+                    "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
+                )
+                use_cache = False
+
+        past_key_values_length = 0
+
+        if use_cache:
+            use_legacy_cache = not isinstance(past_key_values, Cache)
+            if use_legacy_cache:
+                past_key_values = DynamicCache.from_legacy_cache(past_key_values)
+            past_key_values_length = past_key_values.get_usable_length(seq_length)
+
+        if position_ids is None:
+            device = input_ids.device if input_ids is not None else inputs_embeds.device
+            position_ids = torch.arange(
+                past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device
+            )
+            position_ids = position_ids.unsqueeze(0).view(-1, seq_length)
+        else:
+            position_ids = position_ids.view(-1, seq_length).long()
+
+        if inputs_embeds is None:
+            inputs_embeds = self.embed_tokens(input_ids)
+
+        if attention_mask is not None and self._attn_implementation == "flash_attention_2" and use_cache:
+            is_padding_right = attention_mask[:, -1].sum().item() != batch_size
+            if is_padding_right:
+                raise ValueError(
+                    "You are attempting to perform batched generation with padding_side='right'"
+                    " this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to "
+                    " call `tokenizer.padding_side  = 'left'` before tokenizing the input. "
+                )
+
+        if self._attn_implementation == "flash_attention_2":
+            # 2d mask is passed through the layers
+            attention_mask = attention_mask if (attention_mask is not None and 0 in attention_mask) else None
+        elif self._attn_implementation == "sdpa" and not output_attentions:
+            # output_attentions=True can not be supported when using SDPA, and we fall back on
+            # the manual implementation that requires a 4D causal mask in all cases.
+            attention_mask = _prepare_4d_attention_mask_for_sdpa(attention_mask, inputs_embeds.dtype)
+        else:
+            # 4d mask is passed through the layers
+            attention_mask = _prepare_4d_attention_mask(
+                attention_mask,
+                inputs_embeds.dtype,
+            )
+
+        hidden_states = inputs_embeds
+
+        # decoder layers
+        all_hidden_states = () if output_hidden_states else None
+        all_self_attns = () if output_attentions else None
+        next_decoder_cache = None
+
+        for decoder_layer in self.layers:
+            if output_hidden_states:
+                all_hidden_states += (hidden_states,)
+
+            if self.gradient_checkpointing and self.training:
+                layer_outputs = self._gradient_checkpointing_func(
+                    decoder_layer.__call__,
+                    hidden_states,
+                    attention_mask,
+                    position_ids,
+                    past_key_values,
+                    output_attentions,
+                    use_cache,
+                )
+            else:
+                layer_outputs = decoder_layer(
+                    hidden_states,
+                    attention_mask=attention_mask,
+                    position_ids=position_ids,
+                    past_key_value=past_key_values,
+                    output_attentions=output_attentions,
+                    use_cache=use_cache,
+                )
+
+            hidden_states = layer_outputs[0]
+
+            if use_cache:
+                next_decoder_cache = layer_outputs[2 if output_attentions else 1]
+
+            if output_attentions:
+                all_self_attns += (layer_outputs[1],)
+
+        hidden_states = self.norm(hidden_states)
+
+        # add hidden states from the last decoder layer
+        if output_hidden_states:
+            all_hidden_states += (hidden_states,)
+
+        next_cache = None
+        if use_cache:
+            next_cache = next_decoder_cache.to_legacy_cache() if use_legacy_cache else next_decoder_cache
+
+        if not return_dict:
+            return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)
+
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=next_cache,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attns,
+        )
+
+    @torch.no_grad()
+    def encode(
+        self,
+        prompts: List[str],
+        instruction: str = "",
+        max_length: int = 4096,
+    ) -> Dict[str, torch.Tensor]:
+        """
+        Encode a list of prompts and an instruction using the model.
+
+        Args:
+            prompts: A list of prompts to encode.
+            instruction: An instruction to prepend to the prompts. Defaults to "".
+            max_length: The maximum length of the input_ids. Defaults to 4096.
+
+        Returns:
+            A dictionary containing the sentence embeddings with key "sentence_embeddings".
+        """
+        if self.padding_side == "right" and self.is_mask_instruction and len(instruction) > 0:
+            instruction_lens = len(self.tokenizer.tokenize(instruction))
+        else:
+            instruction_lens = 0
+
+        device = next(self.parameters()).device
+        batch_dict = input_transform_func(
+            self.tokenizer,
+            {"input_texts": [prompt for prompt in prompts]},
+            always_add_eos=False,
+            max_length=max_length,
+            instruction=instruction,
+        )
+
+        features: ConanEmbedFeatures = self.prepare_kwargs_from_batch(batch_dict, instruction_lens, device=device)
+        outputs = self(**features)
+
+        embeds = self._sentence_embedding(outputs.last_hidden_state)
+        return {"sentence_embeddings": embeds}
+
+
+# AutoModel Register
+AutoModel.register(ConanEmbedConfig, ConanEmbedModel)
+
+# Register for auto class
+ConanEmbedModel.register_for_auto_class("AutoModel")