ESM2¶

Masked protein language model (BERT-style) from Meta AI/FAIR that produces sequence embeddings, masked-token predictions, and per-sequence log-probabilities for proteins up to 2048 residues.

License: MIT · Molecules: protein · Tasks: embedding, sequence_completion, property_prediction

Actions: encode, predict, log_prob · Variants: 5

At a glance¶

Use it when

You need general-purpose protein embeddings for downstream ML tasks (fitness prediction, subcellular localization, function annotation, clustering) and want the widest ecosystem compatibility
You want fast single-sequence inference without MSA computation — ESM2 produces embeddings in a single forward pass, making it suitable for proteome-scale analyses (millions of sequences)
You are building a multi-model pipeline and need a reliable embedding backbone — ESM2 is the most commonly used feature extractor in published protein ML workflows
You need zero-shot variant effect scoring (log_prob) to prioritize mutations in a directed evolution or clinical genetics study before committing to expensive wet-lab validation
You need to choose between inference cost and quality — the 8M variant on CPU handles rapid screening, while the 3B variant on L40S GPU provides the highest-quality representations
You need contact maps as a quick structural assessment without running a full structure prediction pipeline — ESM2 attention-derived contacts are fast and informative for fold-level analysis
Your sequences include proteins from diverse organisms (bacteria, archaea, eukaryotes, viruses) — UniRef50 training ensures broad taxonomic coverage
You want a permissively licensed (MIT) foundation model with no commercial restrictions for integration into proprietary pipelines or products

Strengths

Five size variants (8M, 35M, 150M, 650M, 3B) allow fine-grained speed/quality tradeoff — the 8M and 35M variants run on CPU-only, making ESM2 accessible without GPU infrastructure
650M single-sequence contact prediction matches MSA Transformer accuracy (long-range P@L ~0.57) while being orders of magnitude faster — no MSA computation required
Most widely benchmarked protein language model: downstream performance validated across ProteinGym DMS (87 datasets), contact prediction, and transfer learning tasks, providing reliable baseline comparisons
MIT license with no usage restrictions — suitable for commercial, academic, and clinical applications without licensing concerns
Rich output repertoire from a single model: per-token embeddings, mean-pooled embeddings, BOS embeddings, predicted contact maps, attention matrices, logits, and per-sequence log-probabilities
Proven as a drop-in replacement for computationally expensive PSSM/MSA features in downstream pipelines (e.g., DeepRank-GNN-esm achieved equal PPI scoring accuracy with ESM2 embeddings vs. PSSM features)
Transfer learning benchmark (Nature Scientific Reports, 2025) demonstrated that the 650M variant matches the 3B variant on most downstream classification tasks, validating the mid-range variant as the practical sweet spot
Deterministic inference (fixed seed) ensures reproducible results across runs on the same hardware, critical for scientific reproducibility and regulatory submissions

Limitations

Encoder-only architecture — cannot generate or design novel protein sequences; for sequence design tasks, use generative models like ProteinMPNN, ProGen2, or ZymCTRL
Maximum input length is 2048 amino-acid residues for all variants (BOS/EOS tokens are added internally); sequences exceeding this limit are truncated, losing C-terminal context. The smaller variants (8M–650M) were trained with a 1024-token crop, so embeddings for very long sequences may be less reliable on those variants.
No structure awareness — trained purely on sequences without 3D structural supervision; structure-aware models such as ProstT5 incorporate structural tokens and outperform ESM2 on structure-dependent tasks when structure information is available
Contact prediction derived from attention weights is approximate — not competitive with dedicated structure prediction models (ESMFold, AlphaFold2) for actual structure determination
Single-chain only — no explicit modeling of protein complexes, multi-chain interactions, or protein-ligand binding; models like Chai-1 or RF3 handle multi-chain inputs natively
Representation quality degrades for under-represented protein types: membrane proteins, intrinsically disordered proteins, fibrous proteins (collagen, keratin), and very short peptides (<10 residues)
No explicit handling of post-translational modifications (phosphorylation, glycosylation, ubiquitination) — modified residues are invisible to the model vocabulary
Variant effect prediction (log-probability scoring, single unmasked forward pass) scores ~0.42 Spearman rho on ProteinGym DMS — competitive but not state-of-the-art compared to ensemble methods like ESM-1v (5-model, ~0.45) or supervised approaches
Significant quality gap across variants: 8M contact prediction P@L ~0.30 vs. 650M ~0.57, yet 3B provides only marginal gains over 650M (ProteinGym DMS: 0.43 vs 0.42 Spearman rho) at much higher compute cost (L40S GPU, reduced tokens_per_batch)

Reach for something else when

You need structure-conditioned protein representations and have access to experimental or predicted structures — use ProstT5 instead, which incorporates Foldseek 3Di structural tokens and captures fold-level information ESM2 cannot access
You need to generate or design novel protein sequences — use ProteinMPNN for structure-conditioned inverse folding, ProGen2 for unconditional generation, or ZymCTRL for enzyme generation with EC number conditioning
You need accurate 3D protein structure prediction — use ESMFold (which uses ESM2 internally but adds a structure module), Chai-1, RF3, or AF2 NIM for higher-accuracy structure prediction
You are working exclusively with antibody sequences and need CDR-specific representations — use AbLang2 or IgBERT, which are trained on antibody-specific corpora and outperform general protein LMs on antibody tasks
You need to analyze protein sequences longer than ~2048 residues without truncation — consider Evo or Evo2 which support much longer context lengths (up to 131K tokens for Evo2)
You need to model protein-protein complexes or multi-chain assemblies — use Chai-1 or RF3 which accept multi-chain inputs with covalent and non-covalent interaction modeling
You need an MSA-aware model that explicitly leverages evolutionary coupling information from alignments — use MSA Transformer or PoET, which process MSA columns jointly rather than single sequences

Alternatives

Model	Better when	Worse when
esmc	Newer, more efficient architecture with competitive embedding quality and updated training.	ESM2 has far more published benchmarks and 5 size variants (8M-3B incl. CPU-only) vs ESMC's single 300M, plus an unmatched downstream ecosystem (e.g. ESMFold).
esm1b	Retained only for backward compatibility with legacy pipelines depending on its specific embedding dimensions.	ESM2-650M matches or exceeds ESM-1b on every evaluated benchmark with the same architecture/input format; ESM2 is strictly preferable.
esm1v	5-model ensemble gives higher Spearman on zero-shot variant-effect prediction.	Single-model ESM2-650M is faster and more versatile; ESM-1v is variant-effect-only (single per sequence) and offers no embeddings.
e1	Retrieval-augmented protein LM: prepending homolog context sequences lifts embeddings and variant-effect scoring above single-sequence ESM2, approaching MSA-level signal without a formal alignment.	ESM2 is far more benchmarked, offers 5 size variants including CPU-only, and needs no context homologs; E1 is newer with a thinner validated track record.
prostt5	Injects Foldseek 3Di structural tokens for fold-aware embeddings that outperform ESM2 on structure-dependent downstream tasks when structure signal helps.	ESM2 has more size variants, a richer LM output repertoire (logits/contacts/attention/log_prob for variant effect), and a much larger validated ecosystem; ProstT5's encode is embedding-focused.

API & schema¶

Variants

Variant	Endpoint slug	GPU	CPU	Memory
8m	`esm2-8m`	CPU	2.0	8 GB
35m	`esm2-35m`	CPU	2.0	8 GB
150m	`esm2-150m`	t4	4.0	16 GB
650m	`esm2-650m`	t4	4.0	16 GB
3b	`esm2-3b`	l40s	4.0	32 GB

Call an action with POST /api/v1/{slug}/{action} — the request envelope is {"items": [...], "params": {...}} and a success returns {"results": [...]}. See the HTTP API page for the base URL, error shape, and full contract.

`encode`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/esm2-8m/encode \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

Request — ESM2EncodeRequest

Field	Type	Required	Constraints	Description
`params`	ESM2EncodeRequestParams	no		Optional parameters controlling this action (defaults are used when omitted).
`items`	list[ESM2EncodeRequestItem]	yes	items 1–8	Batch of inputs to process in a single request. Up to 8 sequences per request.

Nested types

ESM2EncodeIncludeOptions

Allowed values: mean, per_residue, per_residue, bos, contacts, logits, attentions

ESM2EncodeRequestItem

Field	Type	Required	Constraints	Description
`sequence`	string	yes	len 1–2048	A protein sequence in single-letter amino-acid codes.

ESM2EncodeRequestParams

Field	Type	Required	Constraints	Description
`repr_layers`	list[integer]	no	default `[-1]`	Hidden layers whose representations to return (negative indexes count from the last layer).
`include`	list[ESM2EncodeIncludeOptions]	no	default `['mean']`	Optional outputs to compute and include in the response.

Raw JSON Schema

{
  "$defs": {
    "ESM2EncodeIncludeOptions": {
      "enum": [
        "mean",
        "per_residue",
        "per_residue",
        "bos",
        "contacts",
        "logits",
        "attentions"
      ],
      "title": "ESM2EncodeIncludeOptions",
      "type": "string"
    },
    "ESM2EncodeRequestItem": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A protein sequence in single-letter amino-acid codes.",
          "maxLength": 2048,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "ESM2EncodeRequestItem",
      "type": "object"
    },
    "ESM2EncodeRequestParams": {
      "additionalProperties": false,
      "properties": {
        "repr_layers": {
          "description": "Hidden layers whose representations to return (negative indexes count from the last layer).",
          "items": {
            "type": "integer"
          },
          "title": "Repr Layers",
          "type": "array",
          "default": [
            -1
          ]
        },
        "include": {
          "description": "Optional outputs to compute and include in the response.",
          "items": {
            "$ref": "#/$defs/ESM2EncodeIncludeOptions"
          },
          "title": "Include",
          "type": "array",
          "default": [
            "mean"
          ]
        }
      },
      "title": "ESM2EncodeRequestParams",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "params": {
      "$ref": "#/$defs/ESM2EncodeRequestParams",
      "description": "Optional parameters controlling this action (defaults are used when omitted)."
    },
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 8 sequences per request.",
      "items": {
        "$ref": "#/$defs/ESM2EncodeRequestItem"
      },
      "maxItems": 8,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ESM2EncodeRequest",
  "type": "object"
}

Response — ESM2EncodeResponse

Field	Type	Required	Constraints	Description
`results`	list[ESM2EncodeResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

ESM2EncodeResponseResult

Field	Type	Required	Description
`sequence_index`	integer	yes	Index of the corresponding input sequence within the request batch.
`embeddings`	list[LayerEmbedding] \| null	no	Mean-pooled embedding vectors, one entry per requested layer.
`bos_embeddings`	list[LayerEmbedding] \| null	no	Beginning-of-sequence (CLS) token embedding vectors, one per requested layer.
`residue_embeddings`	list[LayerPerTokenEmbeddings] \| null	no	Per-residue embedding vectors.
`contacts`	list[list[number]] \| null	no	Predicted residue–residue contact probability map.
`attentions`	list[list[number]] \| null	no	Per-layer mean attention received per residue position, averaged over all attention heads and query positions. Shape: [num_layers, sequence_length].
`logits`	list[list[number]] \| null	no	Per-position logits over the model vocabulary.
`vocab_tokens`	list[string] \| null	no	Vocabulary token order corresponding to the logits columns.

LayerEmbedding

Field	Type	Required	Constraints	Description
`layer`	integer	yes		Model layer this representation was taken from.
`embedding`	list[number]	yes		Embedding vector for the sequence at this layer.

LayerPerTokenEmbeddings

Field	Type	Required	Constraints	Description
`layer`	integer	yes		Model layer this representation was taken from.
`embeddings`	list[list[number]]	yes		Per-residue embedding matrix for this layer, shape [sequence_length, hidden_dim].

Raw JSON Schema

{
  "$defs": {
    "ESM2EncodeResponseResult": {
      "properties": {
        "sequence_index": {
          "description": "Index of the corresponding input sequence within the request batch.",
          "title": "Sequence Index",
          "type": "integer"
        },
        "embeddings": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/LayerEmbedding"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Mean-pooled embedding vectors, one entry per requested layer.",
          "title": "Embeddings"
        },
        "bos_embeddings": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/LayerEmbedding"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Beginning-of-sequence (CLS) token embedding vectors, one per requested layer.",
          "title": "Bos Embeddings"
        },
        "residue_embeddings": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/LayerPerTokenEmbeddings"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Per-residue embedding vectors.",
          "title": "Residue Embeddings"
        },
        "contacts": {
          "anyOf": [
            {
              "items": {
                "items": {
                  "type": "number"
                },
                "type": "array"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Predicted residue\u2013residue contact probability map.",
          "title": "Contacts"
        },
        "attentions": {
          "anyOf": [
            {
              "items": {
                "items": {
                  "type": "number"
                },
                "type": "array"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Per-layer mean attention received per residue position, averaged over all attention heads and query positions. Shape: [num_layers, sequence_length].",
          "title": "Attentions"
        },
        "logits": {
          "anyOf": [
            {
              "items": {
                "items": {
                  "type": "number"
                },
                "type": "array"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Per-position logits over the model vocabulary.",
          "title": "Logits"
        },
        "vocab_tokens": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Vocabulary token order corresponding to the logits columns.",
          "title": "Vocab Tokens"
        }
      },
      "required": [
        "sequence_index"
      ],
      "title": "ESM2EncodeResponseResult",
      "type": "object"
    },
    "LayerEmbedding": {
      "properties": {
        "layer": {
          "description": "Model layer this representation was taken from.",
          "title": "Layer",
          "type": "integer"
        },
        "embedding": {
          "description": "Embedding vector for the sequence at this layer.",
          "items": {
            "type": "number"
          },
          "title": "Embedding",
          "type": "array"
        }
      },
      "required": [
        "layer",
        "embedding"
      ],
      "title": "LayerEmbedding",
      "type": "object"
    },
    "LayerPerTokenEmbeddings": {
      "properties": {
        "layer": {
          "description": "Model layer this representation was taken from.",
          "title": "Layer",
          "type": "integer"
        },
        "embeddings": {
          "description": "Per-residue embedding matrix for this layer, shape [sequence_length, hidden_dim].",
          "items": {
            "items": {
              "type": "number"
            },
            "type": "array"
          },
          "title": "Embeddings",
          "type": "array"
        }
      },
      "required": [
        "layer",
        "embeddings"
      ],
      "title": "LayerPerTokenEmbeddings",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/ESM2EncodeResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "ESM2EncodeResponse",
  "type": "object"
}

`predict`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/esm2-8m/predict \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

Request — ESM2PredictRequest

Field	Type	Required	Constraints	Description
`items`	list[ESM2PredictRequestItem]	yes	items 1–8	Batch of inputs to process in a single request. Up to 8 sequences per request.

Nested types

ESM2PredictRequestItem

Field	Type	Required	Constraints	Description
`sequence`	string	yes	len 1–2048	A protein sequence in single-letter amino-acid codes containing one or more tokens for masked prediction.

Raw JSON Schema

{
  "$defs": {
    "ESM2PredictRequestItem": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A protein sequence in single-letter amino-acid codes containing one or more <mask> tokens for masked prediction.",
          "maxLength": 2048,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "ESM2PredictRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 8 sequences per request.",
      "items": {
        "$ref": "#/$defs/ESM2PredictRequestItem"
      },
      "maxItems": 8,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ESM2PredictRequest",
  "type": "object"
}

Response — ESM2PredictResponse

Field	Type	Required	Constraints	Description
`results`	list[ESM2PredictResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

ESM2PredictResponseResult

Field	Type	Required	Description
`logits`	list[list[number]]	yes	Per-position logits over the model vocabulary.
`sequence_tokens`	list[string]	yes	Per-position input tokens, aligned with the logits.
`vocab_tokens`	list[string]	yes	Vocabulary token order corresponding to the logits columns.

Raw JSON Schema

{
  "$defs": {
    "ESM2PredictResponseResult": {
      "properties": {
        "logits": {
          "description": "Per-position logits over the model vocabulary.",
          "items": {
            "items": {
              "type": "number"
            },
            "type": "array"
          },
          "title": "Logits",
          "type": "array"
        },
        "sequence_tokens": {
          "description": "Per-position input tokens, aligned with the logits.",
          "items": {
            "type": "string"
          },
          "title": "Sequence Tokens",
          "type": "array"
        },
        "vocab_tokens": {
          "description": "Vocabulary token order corresponding to the logits columns.",
          "items": {
            "type": "string"
          },
          "title": "Vocab Tokens",
          "type": "array"
        }
      },
      "required": [
        "logits",
        "sequence_tokens",
        "vocab_tokens"
      ],
      "title": "ESM2PredictResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/ESM2PredictResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "ESM2PredictResponse",
  "type": "object"
}

`log_prob`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/esm2-8m/log_prob \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

Request — ESM2LogProbRequest

Field	Type	Required	Constraints	Description
`items`	list[ESM2EncodeRequestItem]	yes	items 1–8	Batch of inputs to process in a single request. Up to 8 sequences per request.

Nested types

ESM2EncodeRequestItem

Field	Type	Required	Constraints	Description
`sequence`	string	yes	len 1–2048	A protein sequence in single-letter amino-acid codes.

Raw JSON Schema

{
  "$defs": {
    "ESM2EncodeRequestItem": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A protein sequence in single-letter amino-acid codes.",
          "maxLength": 2048,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "ESM2EncodeRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 8 sequences per request.",
      "items": {
        "$ref": "#/$defs/ESM2EncodeRequestItem"
      },
      "maxItems": 8,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ESM2LogProbRequest",
  "type": "object"
}

Response — ESM2LogProbResponse

Field	Type	Required	Constraints	Description
`results`	list[ESM2LogProbResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

ESM2LogProbResponseResult

Field	Type	Required	Constraints	Description
`log_prob`	number	yes		Log-likelihood of the sequence under the model.

Raw JSON Schema

{
  "$defs": {
    "ESM2LogProbResponseResult": {
      "properties": {
        "log_prob": {
          "description": "Log-likelihood of the sequence under the model.",
          "title": "Log Prob",
          "type": "number"
        }
      },
      "required": [
        "log_prob"
      ],
      "title": "ESM2LogProbResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/ESM2LogProbResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "ESM2LogProbResponse",
  "type": "object"
}

Usage¶

One-line summary: Masked protein language model (BERT-style) from Meta AI/FAIR that produces sequence embeddings, masked-token predictions, and per-sequence log-probabilities for proteins up to 2048 residues.

Overview¶

ESM-2 (Evolutionary Scale Modeling 2) is a protein language model developed by Meta AI's Fundamental AI Research (FAIR) team. It is trained with a masked language modeling objective on UniRef50, learning contextual representations of amino acids from evolutionary sequence data alone -- no structural supervision is required during pre-training.

ESM-2 is the most widely-used model in this catalog and serves as the backbone for downstream models (e.g., ESMFold for structure prediction). Its embeddings capture rich evolutionary and biophysical information and have been shown to be competitive with or superior to alignment-based methods on many protein function prediction tasks.

The model is available in five size variants (8M to 3B parameters), allowing users to trade off between speed and representation quality depending on their use case.

Architecture¶

Property	Value
Architecture	Transformer encoder (BERT-style)
Training objective	Masked language modeling (MLM)
Training data	UniRef50 (UR50/D)
Max sequence length	2048 amino-acid residues (BOS/EOS added internally)
Vocabulary	33 tokens (20 standard AA + special tokens)
License	MIT

Capabilities & Limitations¶

CAN be used for: - Generating per-residue and mean-pooled sequence embeddings for downstream ML tasks - Masked token prediction (fill-in-the-blank for protein sequences) - Zero-shot variant effect prediction via log-probability scoring (log_prob) - Extracting attention maps and contact predictions - Feature extraction for downstream classifiers (stability, function, localization)

CANNOT be used for: - 3D structure prediction (use ESMFold or Chai-1 instead) - Sequence generation or design (this is an encoder-only model) - Non-protein molecules (DNA, RNA, small molecules) - Multi-chain / protein complex modeling

Other considerations: - Sequences longer than 2048 amino-acid residues are truncated (BOS/EOS are added internally) - The model uses GPU memory snapshots for fast cold starts - Batch size is capped at 8 sequences per request - The 3B variant uses a reduced tokens-per-batch (1024 vs 4096) to fit in GPU memory

Usage Examples¶

# Encode  --  get mean embeddings
from models.esm2.schema import (
    ESM2EncodeRequest,
    ESM2EncodeRequestItem,
    ESM2EncodeRequestParams,
)

encode_request = ESM2EncodeRequest(
    params=ESM2EncodeRequestParams(repr_layers=[-1], include=["mean"]),
    items=[
        ESM2EncodeRequestItem(sequence="MKTVRQERLKSIVRILERSKEPVSGAQLAEELSVSRQVIVQDIAYLRSLGYNIVATPRGYVLAGG"),
    ],
)

# Predict  --  masked token prediction
from models.esm2.schema import ESM2PredictRequest, ESM2PredictRequestItem

predict_request = ESM2PredictRequest(
    items=[
        ESM2PredictRequestItem(sequence="MKT<mask>RQERLKSIVRILERSKEPVSGAQ"),
    ],
)

# Predict log probability  --  sequence scoring
from models.esm2.schema import ESM2LogProbRequest, ESM2EncodeRequestItem

log_prob_request = ESM2LogProbRequest(
    items=[
        ESM2EncodeRequestItem(sequence="MKTVRQERLKSIVRILERSKEPVSGAQLAEELSVSRQVIVQDIAYLRSLGYNIVATPRGYVLAGG"),
    ],
)

Architecture & training¶

Architecture¶

Model Type & Innovation¶

ESM-2 is a masked protein language model based on the BERT (bidirectional transformer encoder) architecture. It is trained with a masked language modeling (MLM) objective: 15% of input tokens are randomly masked, and the model learns to predict the original amino acid at each masked position from the surrounding context.

The key innovation of ESM-2 over its predecessor ESM-1b is scaling. The authors systematically trained models from 8M to 15B parameters and demonstrated that representation quality improves log-linearly with model scale -- larger models produce embeddings that better capture protein structure and function. ESM-2 at 650M parameters matches or exceeds the performance of MSA-based methods (e.g., MSA Transformer) using only single sequences as input, eliminating the costly multiple sequence alignment step.

ESM-2 uses a standard transformer encoder with pre-layer normalization, GELU activations, and rotary position embeddings (RoPE). Unlike some newer protein language models, it does not use structural supervision during training. The switch from learned absolute positional embeddings (used in ESM-1b) to RoPE is the key architectural change in ESM-2, and it is what allows a uniform 2048-residue context cap across all model sizes.

Parameters & Layers¶

Variant	Parameters	Layers	Hidden Dim	Attention Heads	Feed-Forward Dim	Embedding Dim
esm2_t6_8M_UR50D	8M	6	320	20	1280	320
esm2_t12_35M_UR50D	35M	12	480	20	1920	480
esm2_t30_150M_UR50D	150M	30	640	20	2560	640
esm2_t33_650M_UR50D	650M	33	1280	20	5120	1280
esm2_t36_3B_UR50D	3B	36	2560	40	10240	2560

Common across all variants:

Property	Value
Vocabulary size	33 tokens (20 standard AA + special tokens)
Positional encoding	Rotary (RoPE)
Normalization	Pre-LayerNorm
Activation	GELU
Max input length	2048 amino-acid residues (BOS/EOS added internally)

The attention head count is 20 for all variants except the 3B model which uses 40 heads. The feed-forward dimension is consistently 4x the hidden dimension across all variants. These values are confirmed from the ESM-2 model architecture (standard transformer encoder with 4x FFN expansion ratio).

Training Data¶

Property	Details
Dataset	UniRef50 (UR50/D)
Clustering	50% sequence identity
Composition	Proteins from all domains of life (bacteria, archaea, eukaryota, viruses)

ESM-2 was trained on approximately 65 million protein sequences from UniRef50 (UR50/D 2021_04 release). UniRef50 clusters UniProtKB sequences at 50% sequence identity and selects the longest sequence as the representative for each cluster. The training data encompasses hundreds of billions of amino acid residues across all domains of life.

Known biases in the training data: - Bacterial proteins are over-represented relative to eukaryotic proteins in UniRef50 - Membrane proteins and intrinsically disordered regions are under-represented - Very long proteins (>1022 residues) were cropped during training, so the model has limited exposure to full-length long proteins

Loss Function & Objective¶

Masked language modeling (MLM) with cross-entropy loss:

L = -Sum_i log P(x_masked_i | x_visible)

During training, 15% of input tokens are randomly selected for masking. Of the selected positions: - 80% are replaced with the <mask> token - 10% are replaced with a random amino acid - 10% are left unchanged

This is the standard BERT masking strategy applied to protein sequences.

Tokenization / Input Processing¶

Property	Details
Tokenizer	Character-level (one token per amino acid)
Special tokens	`<cls>` (BOS), `<eos>` (EOS), `<pad>`, `<mask>`, `<unk>`
BOS prepended	Yes
EOS appended	Yes
Extra tokens per sequence	2 (BOS + EOS)

The maximum input length is 2048 amino-acid residues. BOS and EOS tokens are added internally, so a 2048-residue sequence produces a 2050-token sequence inside the model. max_sequence_len in ESM2Params refers to the residue count, not the token count. Note that during training, smaller variants (8M–650M) used a 1024-token crop, so embeddings for sequences longer than ~1022 residues may be less reliable for those variants.

Input sequences are validated against the extended amino acid alphabet (20 standard + B, J, O, U, X, Z) plus the - gap character for the encode action and the <mask> token for the predict action.

Performance & Benchmarks¶

Published Benchmarks¶

Contact Prediction (Long-Range P@L)¶

Contact prediction accuracy improves log-linearly with model scale. The ESM-2 models achieve the following approximate long-range contact precision (P@L for contacts with sequence separation >= 24):

Variant	Long-Range P@L (approx)
esm2_t6_8M	~0.30
esm2_t12_35M	~0.40
esm2_t30_150M	~0.52
esm2_t33_650M	~0.57
esm2_t36_3B	~0.60

The 650M model matches the performance of MSA Transformer on contact prediction using only single sequences as input.

Structure Prediction (via ESMFold)¶

ESMFold, which uses ESM-2 as its language model backbone, achieves competitive structure prediction accuracy:

Benchmark	ESMFold (ESM-2 15B backbone)	Notes
CAMEO (2022)	Median LDDT ~0.73	Single-sequence, no MSA
CASP14 free modeling	GDT-TS competitive with early AlphaFold2 single-sequence	Without templates or MSAs

ESMFold produces structures in a single forward pass (~1-2 seconds) compared to minutes for AlphaFold2, making it suitable for proteome-scale structure prediction. The accuracy is lower than AlphaFold2 with MSAs but competitive for single-sequence methods.

Variant Effect Prediction¶

ESM-2 log-probability scores (log_prob, a single unmasked forward pass / wt-marginal scoring, not a masked pseudo-log-likelihood) correlate with experimentally measured protein fitness. On the ProteinGym DMS benchmarks, ESM-2 achieves approximately:

Model	Avg Spearman rho (ProteinGym DMS)
ESM-2 650M	~0.42
ESM-2 3B	~0.43
ESM-1v (5-model ensemble)	~0.45

Key findings from the paper: - Representation quality scales log-linearly with model size - ESM-2 650M single-sequence performance matches MSA Transformer on contact prediction - ESM-2 3B is the best single-sequence protein language model at time of publication

BioLM Verification Results¶

The BioLM implementation uses official pre-trained weights loaded via esm.pretrained.load_model_and_alphabet_hub(). Numerical verification is performed against golden reference outputs:

Metric	Threshold	Status
Relative tolerance	1e-4	PASS
Cosine distance	< 0.02	PASS

Tests cover all five variants (8m, 35m, 150m, 650m, 3b) across encode, predict, and log_prob actions.

Comparison to Alternatives¶

Model	Type	Key Advantage	Key Disadvantage
ESM-2 (this)	Protein LM	Fast single-sequence inference, widely adopted	Encoder-only, no generation
ESM-1b	Protein LM	Predecessor, still functional	Strictly worse representations than ESM-2 at same scale
ProtTrans (ProtT5)	Protein LM	Encoder-decoder, generation possible	Larger compute footprint
SaProt	Structure-aware LM	Incorporates 3D structure tokens	Requires structure input (AlphaFold predicted or experimental)
ESM3	Multimodal protein LM	Handles sequence + structure + function	Newer, less widely benchmarked

Error Bars & Confidence¶

ESM-2 is deterministic when seeds are set (as in the BioLM implementation). The same input will produce the same output on the same hardware.

Sources of variability across runs: - Different GPU architectures may produce slightly different floating-point results (within 1e-4 relative tolerance) - The contacts output is derived from attention weights and may show small numerical differences across hardware

Strengths & Limitations¶

Pros¶

Fast inference: single forward pass, no MSA computation needed
Excellent general-purpose protein representations
Wide adoption means extensive downstream benchmarking
MIT licensed with no restrictions
Multiple size variants allow speed/quality tradeoff
Rich output options: embeddings, contacts, logits, attentions

Cons¶

Encoder-only architecture cannot generate or design sequences
Single-chain only -- no explicit multi-chain or complex modeling
Representation quality plateaus relative to newer structure-aware models (e.g., SaProt)
Contact prediction from attention weights is approximate (not a dedicated contact predictor)
No explicit handling of post-translational modifications

Known Failure Modes¶

Very short sequences (< 10 residues): Embeddings may lack meaningful context; consider peptide-specific models
Poly-amino acid repeats (e.g., poly-Q, poly-A): The model has seen these in training but representations are low-quality
Non-standard residues: Characters outside the extended AA alphabet plus - are rejected by validation
Truncation boundary effects: Sequences at exactly the max length may have edge effects at the C-terminal region

Implementation Details¶

Inference Pipeline¶

Request
  |-- 1. Validate sequences (alphabet, length, batch size)
  |-- 2. Tokenize with ESM alphabet (add BOS/EOS)
  |-- 3. Batch with FastaBatchedDataset (toks_per_batch limit)
  |-- 4. Forward pass on GPU (torch.no_grad)
  |     |-- Encoder: N transformer layers
  |     |-- Output: representations, logits, contacts, attentions
  |-- 5. Post-process per include options:
  |     |-- mean: average per-token embeddings (excluding BOS/EOS)
  |     |-- per_token: return all token embeddings
  |     |-- bos: return BOS (CLS) token embedding
  |     |-- contacts: return predicted contact map
  |     |-- logits: slice to 20 AA vocab [4:-9]
  |     |-- attentions: average over heads then queries → (num_layers, seq_len)
  |-- 6. Sort results by sequence_index
  |-- 7. Return ESM2EncodeResponse

For log_prob, the pipeline calls _encode_forward_pass with include=["logits"], then computes log-softmax and sums log P(residue_i) at each position.

Memory & Compute Profile¶

Variant	tokens_per_batch	GPU Memory (approx)
8m	4096	CPU only
35m	4096	CPU only
150m	4096	~4 GB VRAM
650m	4096	~8 GB VRAM
3b	1024	~24 GB VRAM

Attention computation scales as O(n^2) with sequence length. For the 650M model on a T4 (16 GB VRAM), batches of long sequences (>1000 residues) may need smaller batch sizes.

Determinism & Reproducibility¶

Setting	Value
`torch.manual_seed`	42
`torch.cuda.manual_seed_all`	42
`torch.no_grad`	Yes (inference)
cuDNN deterministic	Not explicitly set
cuDNN benchmark	Not explicitly disabled

The model produces reproducible outputs on the same GPU architecture. Small numerical differences (within 1e-4) may occur across different GPU types due to floating-point operation ordering.

Caching Behavior¶

Response caching is handled by the serving infrastructure, not by the model container. The cache key is determined by the request payload (sequences, params, include options, and model variant).

Versions & Changelog¶

Version	Date	Changes
v1	2024-10-23	Initial implementation with encode and predict actions
v1 (updated)	2025-01-20	Added `log_prob` action for zero-shot variant scoring
v1 (updated)	2025-09-14	Added 3B variant with L40S GPU support
v1 (updated)	2026-03-14	Migrated to declarative download system and source layer setup

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

ESM-2 is trained on UniRef50, which covers protein sequences from all domains of life -- bacteria, archaea, eukaryota, and viruses. The model accepts single-chain protein sequences of up to 2048 amino-acid residues (BOS/EOS tokens are added internally).

ESM-2 handles globular proteins best, as these are the most abundant in training data. Performance characteristics vary by protein type:

Globular, soluble proteins: Excellent representation quality. These constitute the bulk of UniRef50.
Membrane proteins: Functional but degraded -- transmembrane helices and membrane-spanning regions are under-represented in UniRef50 relative to their biological importance.
Intrinsically disordered proteins/regions (IDPs/IDRs): The model can embed these, but contact predictions and structural inferences will be unreliable since disordered regions lack stable 3D structure by definition.
Multi-domain proteins: Each domain is well-represented; however, inter-domain contacts may be less accurate for very long sequences that approach the 2048-token limit.
Fibrous proteins (collagen, keratin): Under-represented in training data. Repetitive sequences may produce degenerate embeddings.

Cross-Applicability¶

Molecule Type	Applicability	Evidence	Caveats
Antibodies	Moderate–High	ESM-2 embeddings used for antibody developability prediction in multiple studies	Variable regions (CDRs) are well-represented; constant regions are less informative. Specialized antibody models (e.g., IgBERT, AntiBERTy) may outperform on antibody-specific tasks.
Enzymes	High	Active site and catalytic residue prediction validated across enzyme families	No explicit catalytic mechanism modeling -- embeddings capture evolutionary conservation at active sites
Peptides	Low–Moderate	Short sequences (< 30 residues) provide limited context for the transformer	Consider peptide-specific models for very short sequences; ESM-2 may still be useful as a feature extractor
Antibody-drug conjugates	Low	No training on conjugated or modified proteins	Linker chemistry and payload interactions are invisible to the model

Biological Problems Addressed¶

Protein Representation Learning¶

Problem: Many downstream protein analysis tasks (function prediction, localization, interaction) require numerical representations of protein sequences. Traditional approaches use hand-crafted features (amino acid composition, physicochemical scales) or alignment-based profiles (PSSM, HMM), both of which have limitations in expressiveness or computational cost.

How ESM-2 helps: The encode action produces dense vector embeddings that capture evolutionary, structural, and biophysical properties learned from millions of protein sequences. These embeddings can be used as features for any downstream classifier or regressor.

Biological meaning: A mean-pooled ESM-2 embedding is a fixed-length vector (dimension = hidden_dim of the chosen variant) that encodes the "evolutionary fingerprint" of a protein. Proteins with similar function, structure, or evolutionary origin will have similar embeddings, even if their sequence identity is low. Per-token embeddings additionally capture position-specific information useful for residue-level predictions.

Zero-Shot Variant Effect Prediction¶

Problem: Predicting the functional impact of amino acid substitutions (missense mutations) is critical for clinical genetics, protein engineering, and understanding disease mechanisms. Experimental methods (deep mutational scanning) are expensive and limited to one protein at a time.

How ESM-2 helps: The log_prob action computes the summed per-residue log-probability of a full sequence using a single unmasked forward pass. By comparing log P(wildtype) vs log P(mutant), researchers can estimate variant effects without any task-specific training. The model naturally assigns higher probability to residues that are evolutionarily conserved at a given position. Note: this is a "wt-marginal" score (the model sees the full sequence), not a masked pseudo-log-likelihood (which would mask each position independently).

Biological meaning: A large negative delta-log-probability (mutant much less likely than wildtype) suggests the mutation disrupts an evolutionarily conserved position and is likely deleterious. This correlates with experimental measures of protein fitness (stability, activity, binding affinity) as benchmarked on datasets like ProteinGym.

Masked Token Prediction (Sequence Completion)¶

Problem: In protein engineering and directed evolution, researchers often want to know which amino acids are compatible at specific positions in a protein sequence, accounting for the full sequence context.

How ESM-2 helps: The predict action takes a sequence with one or more <mask> tokens and returns the predicted probability distribution over all 20 standard amino acids at each masked position. This provides a context-aware "consensus" for each position.

Biological meaning: The predicted distribution at a masked position reflects evolutionary constraints: positions critical for structure or function will have narrow distributions (few amino acids tolerated), while surface-exposed loop positions will have broader distributions (many amino acids acceptable). This can guide rational mutagenesis campaigns.

Contact Prediction¶

Problem: Identifying which residue pairs are in physical contact (< 8 Angstroms between C-beta atoms) helps constrain protein structure and reveals functional sites.

How ESM-2 helps: The encode action with include=["contacts"] extracts predicted contact maps from the model's attention weights. This is a byproduct of the transformer's attention mechanism learning structural patterns from evolutionary data.

Biological meaning: Predicted contacts between distant residues in sequence (long-range contacts) are particularly informative for protein fold determination. While dedicated structure prediction models (ESMFold, AlphaFold) are more accurate, ESM-2 contact maps are fast to compute and useful for quick structural assessment.

Applied Use Cases¶

Protein-Protein Interaction Scoring with ESM-2 Embeddings¶

Source: Xu et al. "DeepRank-GNN-esm: a graph neural network for scoring protein-protein models using protein language model." Bioinformatics Advances (2024). DOI: 10.1093/bioadv/vbad191

ESM-2 embeddings were used to replace traditional PSSM features for protein-protein interaction scoring in a graph neural network framework. The approach achieved equal or better performance than alignment-based features while being significantly faster to compute, demonstrating that ESM-2 representations capture sufficient evolutionary information to replace computationally expensive multiple sequence alignments in PPI scoring pipelines.

Transfer Learning Benchmark Across Protein Classification Tasks¶

Source: (2025). "Medium-sized protein language models perform well at transfer learning on realistic datasets." Nature Scientific Reports. DOI: 10.1038/s41598-025-05674-x

A systematic benchmark of ESM-2 650M transfer learning across protein classification tasks showed that medium-sized models match the performance of larger models. This finding is practically important: it validates that the 650M parameter variant (rather than the 3B variant) is sufficient for most downstream tasks, making ESM-2-based pipelines accessible without high-end GPU infrastructure.

Protein Expression, Stability, and Function Evaluation¶

Source: (2024). "Improving Protein Expression, Stability, and Function with ProteinMPNN." Journal of the American Chemical Society. DOI: 10.1021/jacs.3c10941

ESM-2 embeddings were used as part of a protein evaluation pipeline for designed protein variants, complementing ProteinMPNN-based inverse folding. The study demonstrates ESM-2's role in multi-model workflows where embeddings serve as a quality filter for computationally designed proteins.

ESM-2 embeddings have additionally been widely adopted for other downstream applications including:

Subcellular localization: Mean-pooled embeddings as features for multi-class classifiers
Drug target prioritization: Embedding-based clustering and functional annotation of proteomes
Antibody engineering: CDR embedding analysis for humanization and developability scoring

Predecessor Models¶

ESM-1b (Rives et al., 2021): The direct predecessor to ESM-2. A 650M parameter transformer trained on UniRef50. ESM-2 improves on ESM-1b at every model scale by using improved training procedures and scaling. ESM-1b is largely superseded by ESM-2.
ESM-1v (Meier et al., 2021): A variant of ESM-1b specifically evaluated for variant effect prediction. ESM-2's log_prob action provides equivalent functionality with better representations.

Complementary Models¶

ESM-2 is the foundation for downstream models and analysis workflows in this catalog:

ESMFold: Uses ESM-2 embeddings as input for single-sequence protein structure prediction

Typical multi-model workflows: 1. Use ESM-2 encode to generate embeddings, then feed into a custom downstream classifier 2. Use ESM-2 log_prob to score mutant libraries, then use ESMFold to predict structures of top candidates

Alternative Models¶

Alternative	Advantage Over ESM-2	Disadvantage vs ESM-2
ProtTrans (ProtT5-XL)	Encoder-decoder architecture allows sequence generation	Larger model, slower inference
SaProt	Structure-aware tokens improve representation quality when structure is available	Requires predicted or experimental structure as input
ESM3	Multimodal (sequence + structure + function tokens)	Newer, fewer downstream benchmarks available
Ankh	Efficient single-sequence encoder competitive with ESM-2	Less widely adopted and benchmarked

When to choose ESM-2: Use ESM-2 when you need fast, reliable, single-sequence protein embeddings with the widest downstream compatibility. It is the safest default choice for most protein representation tasks.

When to choose alternatives: Consider SaProt when you have structural information available; consider ProtT5 when you need generation capabilities; consider ESM3 when you need multimodal protein understanding.

Biological Background¶

Proteins are linear chains of amino acids (typically 50-2000 residues long) that fold into three-dimensional structures to carry out virtually all cellular functions -- catalysis, signaling, transport, structural support, and immune defense. The sequence of amino acids (the "primary structure") determines the protein's 3D structure and, consequently, its function.

Evolutionary conservation: Over billions of years of evolution, natural selection has preserved amino acids critical for protein function while allowing variation at less constrained positions. By analyzing millions of protein sequences from diverse organisms, patterns of conservation and co-variation encode deep information about protein structure and function. This is the fundamental insight that protein language models like ESM-2 exploit.

Masked language modeling on proteins: Just as BERT learns the rules of grammar and semantics by predicting masked words in sentences, ESM-2 learns the "grammar" of proteins -- which amino acids are compatible at each position given the surrounding context. A position deep in the hydrophobic core "expects" hydrophobic residues (L, I, V, F); a surface-exposed loop tolerates most amino acids; a catalytic residue is highly constrained. The model learns these rules implicitly from evolutionary data.

Key terminology: - Embedding: A dense numerical vector representing a protein (or a position within it) in a high-dimensional space. Similar proteins have similar embeddings. - Masked language modeling (MLM): Training objective where random positions are hidden and the model must predict the original amino acid from context. - Log-probability / log-likelihood: A score measuring how "expected" a sequence is under the model. Higher (less negative) scores indicate the sequence is more consistent with evolutionary patterns. ESM-2's log_prob action uses a single unmasked forward pass (wt-marginal scoring), not masked pseudo-log-likelihood. - Contact map: A symmetric matrix indicating which residue pairs are physically close in 3D space. Long-range contacts (residues far apart in sequence but close in space) are particularly informative for structure. - UniRef50: A clustered version of the UniProt Reference Clusters database where sequences are grouped at 50% sequence identity, reducing redundancy while preserving diversity.

Sources & license¶

License: MIT (text)

Papers

Evolutionary-scale prediction of atomic-level protein structure with a language model — Science, 2023 · DOI
Language models of protein sequences at the scale of evolution enable accurate structure prediction — bioRxiv preprint, 2022 · DOI

Source repositories

github: https://github.com/facebookresearch/esm
huggingface: https://huggingface.co/facebook/esm2_t33_650M_UR50D

Cite

@article{lin2023evolutionary,
  title={Evolutionary-scale prediction of atomic-level protein structure with a language model},
  author={Lin, Zeming and Akin, Halil and Rao, Roshan and Hie, Brian and Zhu, Zhongkai and Lu, Wenting and Smetanin, Nikita and Verkuil, Robert and Kabeli, Ori and Shmueli, Yaniv and dos Santos Costa, Allan and Fazel-Zarandi, Maryam and Sercu, Tom and Candido, Sal and Rives, Alexander},
  journal={Science},
  volume={379},
  number={6637},
  pages={1123--1130},
  year={2023},
  doi={10.1126/science.ade2574}
}

ESM2¶

At a glance¶

API & schema¶

encode¶

predict¶

log_prob¶

Usage¶

Overview¶

Architecture¶

Capabilities & Limitations¶

Usage Examples¶

Architecture & training¶

Architecture¶

Model Type & Innovation¶

Parameters & Layers¶

Training Data¶

Loss Function & Objective¶

Tokenization / Input Processing¶

Performance & Benchmarks¶

Published Benchmarks¶

Contact Prediction (Long-Range P@L)¶

Structure Prediction (via ESMFold)¶

Variant Effect Prediction¶

BioLM Verification Results¶

Comparison to Alternatives¶

Error Bars & Confidence¶

Strengths & Limitations¶

Pros¶

Cons¶

Known Failure Modes¶

Implementation Details¶

Inference Pipeline¶

Memory & Compute Profile¶

Determinism & Reproducibility¶

Caching Behavior¶

Versions & Changelog¶

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

Cross-Applicability¶

Biological Problems Addressed¶

Protein Representation Learning¶

Zero-Shot Variant Effect Prediction¶

Masked Token Prediction (Sequence Completion)¶

Contact Prediction¶

Applied Use Cases¶

Protein-Protein Interaction Scoring with ESM-2 Embeddings¶

Transfer Learning Benchmark Across Protein Classification Tasks¶

Protein Expression, Stability, and Function Evaluation¶

Related Models¶

Predecessor Models¶

Complementary Models¶

Alternative Models¶

Biological Background¶

Sources & license¶

`encode`¶

`predict`¶

`log_prob`¶