Skip to content

Adding a Model

Adding a model to the Leaderboard

The MTEB Leaderboard is available here. To submit to it:

  1. Add the model meta to mteb
  2. Evaluate the desired model using mteb on the benchmarks
  3. Push the results to the results repository via a PR. Once merged they will appear on the leaderboard after a day.

Requesting an evaluation

If you want a model to be evaluated but are not submitting the results yourself, open an Evaluation Request issue instead and fill out the required information.

Maintainers prioritize evaluation requests for private subsets or cases where community contributors cannot run the benchmark directly. We aim to run a private evaluation within 7 working days of the request. For public benchmarks, the fastest path is still to run the evaluation locally and submit the results PR.

Info

This section contains info on how to submit a model implementation. If you wish to submit a model results see submit results.

Adding a model implementation

Adding a model implementation to mteb is quite straightforward. Typically, it only requires that you fill in metadata about the model and add it to the model directory:

Adding a ModelMeta object 
from mteb.models import ModelMeta, SentenceTransformerEncoderWrapper
from mteb.types import OutputDType

my_model = ModelMeta(
    name="model_name",
    loader=SentenceTransformerEncoderWrapper,
    languages=["eng-Latn"],  # follows ISO 639-3 and BCP-47
    open_weights=True,
    revision="5617a9f61b028005a4858fdac845db406aefb181",
    release_date="2025-01-01",
    n_parameters=568_000_000,
    memory_usage_mb=2167,
    embed_dim=4096,
    license="mit",
    max_tokens=8194,
    reference="https://huggingface.co/user-or-org/model-name",
    similarity_fn_name="cosine",
    framework=["Sentence Transformers", "PyTorch"],
    use_instructions=False,
    public_training_code="https://github.com/user-or-org/my-training-code",
    public_training_data="https://huggingface.co/datasets/user-or-org/full-dataset",
    training_datasets={"MSMARCO"},  # if you trained on the MSMARCO training set
    output_dtypes=[
        OutputDType.INT8,
        OutputDType.BINARY,
    ],  # Alternative output types supported by the model
)

This works for all Sentence Transformers compatible models. Once filled out, you can submit your model to mteb by submitting a PR.

You can generate it automatically by using:

from mteb.models import ModelMeta

meta = ModelMeta.from_hub("Qwen/Qwen3-Embedding-0.6B")
print(meta.to_python())

from mteb.models import ModelMeta
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B", device="cpu")
meta = ModelMeta.from_sentence_transformer_model(model)
print(meta.to_python())

from mteb.models import ModelMeta
from sentence_transformers import CrossEncoder

model = SentenceTransformer("Qwen/Qwen3-Reranker-0.6B", device="cpu")
meta = ModelMeta.from_cross_encoder(model)
print(meta.to_python())

Calculating the Memory Usage

To calculate memory_usage_mb, run:

model_meta = mteb.get_model_meta("model_name")
model_meta.calculate_memory_usage_mb()

Adding instruction models

Some models, such as the E5 models, use instructions or prefixes. You can directly add the prompts when saving and uploading your model to the Hub. Refer to this configuration file as an example.

However, you can also add these directly to the model configuration:

model = ModelMeta(
    loader=SentenceTransformerEncoderWrapper,
    loader_kwargs=dict(
        model_prompts={
           "query": "query: ",
           "passage": "passage: ",
        },
    ),
    ...
)

Using a custom Implementation

If you need to use a custom implementation, you can specify the loader parameter in the ModelMeta class. It should implement one of the following protocols: Encoder, CrossEncoder, or Search.

Custom Model Implementation 
from mteb.types import PromptType, Array
import numpy as np


class CustomModel:
    def __init__(self, model_name: str, revision: str, **kwargs):
        pass  # your initialization of model here

    def encode(
        self,
        inputs: DataLoader[BatchedInput],
        *,
        task_metadata: TaskMetadata,
        hf_split: str,
        hf_subset: str,
        prompt_type: PromptType | None = None,
        **kwargs,
    ) -> Array:

        arrays = []
        for batch in inputs:
            documents = batch["text"]
            # embed documents:
            embed_dim = 100
            embedding = np.zeros((len(documents), embed_dim))

        embeddings = np.concat(arrays)
        return embeddings

Then you can specify the loader parameter in the ModelMeta class:

your_model = ModelMeta(
    loader=CustomModel,
    loader_kwargs={...},
    ...
)

Loading dependencies only when needed

mteb imports every model implementation to list models and read their metadata, for example inmteb.get_model_metas(), the leaderboard, or when loading results. To keep this fast, a model file should only load the packages it depends on (torch, transformers, sentence_transformers or any model-specific package) when the model is actually used, not when the file is imported.

This means nothing that runs at import time may use them: top-level imports, decorators, default arguments, base classes and values inside ModelMeta(...). The patterns below cover the common cases.

Import packages inside the functions that use them. Imports needed only for type hints go under TYPE_CHECKING:

from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    import torch  # only used in type hints


class MyModel(AbsEncoder):
    def __init__(self, model_name: str, revision: str, **kwargs):
        from transformers import AutoModel  # not at the top of the file

        self.model = AutoModel.from_pretrained(model_name, revision=revision)

    def encode(self, inputs, **kwargs) -> torch.Tensor: ...

For optional packages, also add them as a requirement group, see Adding model dependencies.

Use torch.no_grad() or torch.inference_mode() as a context manager instead of a decorator, since a decorator runs when the file is imported:

def encode(self, inputs, **kwargs):
    import torch

    with torch.inference_mode():  # instead of @torch.inference_mode()
        ...

Default device to None and pick one inside the function, since a default argument runs when the file is imported. A device passed by the user is always used as is:

def __init__(self, model_name: str, revision: str, device: str | None = None, **kwargs):
    import torch

    if device is None:
        device = "cuda" if torch.cuda.is_available() else "cpu"

To default to MPS on Apple Silicon, check for it explicitly:

if device is None:
    if torch.cuda.is_available():
        device = "cuda"
    elif torch.backends.mps.is_available():
        device = "mps"
    else:
        device = "cpu"

ModelMeta entries are created when the file is imported, so name dtypes with OutputDType instead of passing a torch.dtype:

model = ModelMeta(
    loader=MyWrapper,
    loader_kwargs=dict(torch_dtype=OutputDType.BF16),  # not torch.bfloat16
    ...
)

OutputDType is a string enum, so transformers accepts it directly, and OutputDType.get_dtype() returns the matching torch.dtype.

A class that subclasses torch.nn.Module needs torch when it is defined, so define it inside the function that uses it:

class MyModel(AbsEncoder):
    def __init__(self, model_name: str, revision: str, **kwargs):
        import torch

        class Pooler(torch.nn.Module):
            def forward(self, hidden_states):
                return hidden_states.mean(dim=1)

        self.pooler = Pooler()

Adding model dependencies

If you are adding a model that requires additional dependencies, you can add them to the pyproject.toml file, under optional dependencies:

voyageai = ["voyageai>=1.0.0,<2.0.0"]

This ensures that the implementation does not break if a package is updated.

As it is an optional dependency, you can't use top-level dependencies, but will instead have to use import inside the wrapper scope:

Adding optional dependencies 
class VoyageAIModel:
    def __init__(self, model_name: str, revision: str, **kwargs) -> None:
        import voyageai
        ...

# in the model meta specify the requirement group:
voyage_model = ModelMeta(
    model_name = "...",
    extra_requirements_groups=["voyageai"],
    ...
)

Local Model Verification using mock tasks

Before submitting your model implementation in a Pull Request, you must verify that the model integrates correctly with the benchmarking pipeline using the local verification helper mock_run. Mock tasks run locally, execute very quickly, and do not download large datasets.

To run the local verification:

import mteb

# Load your new model
model = mteb.get_model("your_model_name")

# This will run the mock test tasks compatible with your model's modalities and protocols to verify your new model implementation
results = mteb.mock_run(model)

print(results.all_passed)  # whether the model passed every check
print(results.to_markdown())  # the same report the CLI writes to disk

results maps each mock task name to its status:

Value Meaning
TaskResult the task ran successfully
TaskError the task failed, the exception attribute contains the reason (tasks failing due to a missing optional dependency are reported as skipped)
None the task is not compatible with the model's modalities or protocols 
results["MockRetrievalTask"]  # the status of a single mock task
results.task_results  # the full {task name: status} mapping

mteb mock-run -m your_model_name

This prints a results summary table to the terminal and saves the markdown file mteb_mock_run_results.md in the directory you ran the command from. Please commit the resulting file with your PR.

Submitting your model as a PR

When submitting you models as a PR, please copy and paste the following checklist into the pull request message:

- [ ] I have filled out the ModelMeta object to the extent possible
- [ ] I have ensured that my model can be loaded using
  - [ ] `mteb.get_model(model_name, revision)` and
  - [ ] `mteb.get_model_meta(model_name, revision)`
- [ ] I have tested the implementation works on a representative set of tasks.
- [ ] The model is public, i.e., is available either as an API or the weights are publicly available to download
- [ ] I reproduced results from the original paper (if applicable) on at least one benchmark, and I am including the results in the PR description.

Matryoshka embeddings

To add support for matryoshka embeddings you can specify embed_dim as a list of dimensions.

import mteb
from mteb.models import ModelMeta

my_model = ModelMeta(
    name="custom/my_model",
    ...,
    embed_dim=[128, 256, 512, 1024],
)