README.md

---
license: mit
language:
- en
license_link: https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE
base_model: 
  - BAAI/bge-reranker-base
---
# bge-reranker-base-int8-ov

> [!WARNING]
> **Disclaimer**: This model is provided for evaluation purposes only. Performance, accuracy, and stability may vary. Use at your own discretion.

 * Model creator: [BAAI](https://huggingface.co/BAAI)
 * Original model: [bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base)

## Description
This is [bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) model converted to the [OpenVINO™ IR](https://docs.openvino.ai/2025/documentation/openvino-ir-format.html) (Intermediate Representation) format with quantization to INT8 by [NNCF](https://github.com/openvinotoolkit/nncf).


## Quantization Parameters

The quantization was performed using the next code:

```
from functools import partial

from transformers import AutoTokenizer

from optimum.intel import OVConfig, OVModelForSequenceClassification, OVQuantizationConfig, OVQuantizer


MODEL_ID = "OpenVINO/bge-reranker-base-fp16-ov"
base_model_path = "bge-reranker-base-fp16-ov"
int8_ptq_model_path = "bge-reranker-base-int8"

model = OVModelForSequenceClassification.from_pretrained(MODEL_ID)
model.save_pretrained(base_model_path)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.save_pretrained(base_model_path)


quantizer = OVQuantizer.from_pretrained(model)

def preprocess_function(examples, tokenizer):
    return tokenizer(examples["sentence"], padding="max_length", max_length=384, truncation=True)


calibration_dataset = quantizer.get_calibration_dataset(
    "glue",
    dataset_config_name="sst2",
    preprocess_function=partial(preprocess_function, tokenizer=tokenizer),
    num_samples=300,
    dataset_split="train",
)

ov_config = OVConfig(quantization_config=OVQuantizationConfig())

quantizer.quantize(ov_config=ov_config, calibration_dataset=calibration_dataset, save_directory=int8_ptq_model_path)
tokenizer.save_pretrained(int8_ptq_model_path)
```

For more information on quantization, check the [OpenVINO model optimization guide](https://docs.openvino.ai/2025/openvino-workflow/model-optimization-guide/quantizing-models-post-training.html).


## Compatibility

The provided OpenVINO™ IR model is compatible with:

* OpenVINO version 2025.1.0 and higher
* Optimum Intel 1.24.0 and higher


## Running Model Inference with [Optimum Intel](https://huggingface.co/docs/optimum/intel/index)

1. Install packages required for using [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) integration with the OpenVINO backend:

```
pip install optimum[openvino]
```

2. Run model inference:

```
from transformers import AutoTokenizer
from optimum.intel import OVModelForSequenceClassification


tokenizer = AutoTokenizer.from_pretrained('OpenVINO/bge-reranker-base-int8-ov')
model = OVModelForSequenceClassification.from_pretrained('OpenVINO/bge-reranker-base-int8-ov')

pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]

inputs = tokenizer(pairs, padding=True, truncation=True, return_tensors='pt', max_length=512)
scores = model(**inputs, return_dict=True).logits.view(-1, ).float()
print(scores)
```

For more examples and possible optimizations, refer to the [Inference with Optimum Intel](https://docs.openvino.ai/2025/openvino-workflow-generative/inference-with-optimum-intel.html).

You can find more detailed usage examples in OpenVINO Notebooks:

- [RAG text generation](https://openvinotoolkit.github.io/openvino_notebooks/?search=RAG+system)

## Limitations

Check the original [model card](https://huggingface.co/BAAI/bge-reranker-base) for limitations.

## Legal information

The original model is distributed under [MIT](https://github.com/FlagOpen/FlagEmbedding/blob/master/LICENSE) license. More details can be found in [bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base).

## Disclaimer

Intel is committed to respecting human rights and avoiding causing or contributing to adverse impacts on human rights. See [Intel’s Global Human Rights Principles](https://www.intel.com/content/dam/www/central-libraries/us/en/documents/policy-human-rights.pdf). Intel’s products and software are intended only to be used in applications that do not cause or contribute to adverse impacts on human rights.