Skip to content

Installation

Installation is as simple as:

pip install mteb

uv add mteb

Model Specific Installations

If you want to run certain models implemented within mteb you will often need some additional dependencies. These can be installed using:

pip install mteb[cohere]

uv add "mteb[cohere]"

If a specific model requires a dependency it will raise an error with the recommended installation. To see full list of available models you can look at the models overview.

Automatically installing model dependencies

Instead of installing extras manually for every model, you can let mteb install the missing ones for you by setting the MTEB_AUTO_INSTALL_EXTRAS environment variable. When it is set to a truthy value (1, true, yes, on), loading a model whose extras are missing will install them automatically (using uv if it is available, otherwise pip) before raising:

export MTEB_AUTO_INSTALL_EXTRAS=1

Audio Tasks

If you want to run audio tasks, install the audio dependencies:

pip install mteb[audio]

uv add "mteb[audio]"

Additional Requirements for datasets>=4

If you are using datasets>=4, you will need to:

  1. Install FFmpeg: The datasets library version 4+ uses torchcodec for audio processing, which requires FFmpeg to be installed on your system.

    brew install ffmpeg

    sudo apt-get install ffmpeg

    Download from ffmpeg.org and add to your PATH.

  2. Use transformers>=4.57.6: Due to compatibility issues with datasets>=4, you need a recent version of transformers: 

    pip install "transformers>=4.57.6"

If you are using datasets<4, no additional requirements are needed beyond the mteb[audio] installation.

Migrating to uv (for Contributors)

If you're a contributor currently using pip, here's how to migrate to uv for faster dependency management:

Why uv?

  • Faster: 10-100x faster dependency resolution
  • Reliable: Deterministic builds with uv.lock
  • Simpler: One tool for virtual environments and packages

Migration Steps

  1. Install uv: 

    curl -LsSf https://astral.sh/uv/install.sh | sh

  2. Replace your workflow:

  3. pip install -e . → uv sync
  4. pip install mteb[extra] → uv sync --extra extra
  5. python -m pytest → uv run pytest

Development Groups

For contributors, uv provides organized dependency groups:

  • uv sync --group test - Install test dependencies
  • uv sync --group docs - Install documentation dependencies
  • uv sync --group typing - Install type checking dependencies
  • uv sync --group lint - Install linting dependencies
  • uv sync --group dev - Install all development dependencies (recommended)