Skip to content

Command Line

This described the is the command line interface for mteb.

mteb is a toolkit for evaluating the quality of embedding models on various benchmarks. It supports the following commands:

In the following we outline some sample use cases, but if you want to learn more about the arguments for each command you can run:

mteb {command} --help

Running Models on Tasks

To run a model on a set of tasks, use the mteb run command. For example:

mteb run -m sentence-transformers/average_word_embeddings_komninos \
         -t Banking77Classification EmotionClassification \
         --output-folder mteb_output

This will create a folder mteb_output/{model_name}/{model_revision} containing the results of the model on the specified tasks supplied as a json file; {task_name}.json.

Listing Available Tasks

To list the available tasks within MTEB, use the mteb available-tasks command. For example:

mteb available-tasks # list _all_ available tasks

You can also use the multiple arguments for filtering: 

mteb available-tasks --task-types Retrieval --languages eng # list all English (eng) retrieval tasks

Listing Available Benchmarks

To list the available benchmarks within MTEB:

mteb available-benchmarks # list all available benchmarks

Creating Model Metadata

Once a model is run you can create the metadata for a model card from a folder of results, use the mteb create-meta command. For example:

mteb create-meta --results-folder mteb_output/sentence-transformers__average_word_embeddings_komninos/{revision} \
                 --output-path model_card.md

This will create a model card at model_card.md containing the metadata for the model on MTEB within the YAML frontmatter. This will make the model discoverable on the MTEB leaderboard.

Running the Leaderboard

To run the MTEB leaderboard locally, use the mteb leaderboard command. For example:

mteb leaderboard

You can specify a custom cache path and other options:

mteb leaderboard --cache-path results --port 8080 --share

Available options: - --cache-path PATH: Custom path for model results cache - --host HOST: Host to run the server on (default: 0.0.0.0) - --port PORT: Port to run the server on (default: 7860) - --share: Create a public URL for the leaderboard - --rebuild: Force rebuild from full results repository, bypassing cached JSON

For more details on running the leaderboard, see the leaderboard documentation.

Checking Model Implementations

To sanity check a model implementation using a set of example test tasks, use the mteb mock-run command. This command evaluates the model on a set of tasks covering various modalities and task types, while not requiring any external datasets, making it a fast way to test that the implementation works as intended before running larger benchmarks or when implementing a model.

For example:

mteb mock-run -m sentence-transformers/average_word_embeddings_komninos

This will run the model on all compatible mock tasks, print a Markdown summary table to stdout, and save the markdown results to mteb_mock_run_results.md.

Available options: - -m, --model MODEL: The model to use. Prioritizes the model implementation from MTEB's model registry, or defaults to loading via sentence-transformers. - --model-revision REVISION: Revision of the model to load. - --device DEVICE: Device to use for computation (e.g. cpu, cuda). - -v, --verbosity VERBOSITY: Verbosity level (0 to 4, default: 2).

The same checks are available from Python using mteb.mock_run, which returns the per-task status instead of writing a file.