ESM1v¶

Zero-shot protein variant effect prediction using a 5-model ensemble of 650M-parameter masked language models, predicting the functional impact of single amino acid mutations from sequence context alone.

License: MIT · Molecules: protein · Tasks: sequence_completion, property_prediction

Actions: predict · Variants: 6

At a glance¶

Use it when

You need dedicated zero-shot variant effect prediction and want an ensemble approach for reduced variance without any training data
You are screening all possible single-site mutations across a protein and need cost-effective CPU-based scoring for individual model runs
You are building a clinical variant interpretation pipeline and want a well-validated, MIT-licensed variant scorer as one component of a multi-evidence framework
You need to prioritize candidate mutations for a directed evolution campaign and want variant fitness predictions to guide experimental screening
You need ensemble predictions for improved reliability — the 5-model average provides more robust scores than any single protein language model
You are benchmarking variant effect prediction methods and need ESM1v as a widely cited reference baseline

Strengths

Purpose-built for variant effect prediction — trained on UniRef90 (larger, less redundancy-reduced than UniRef50) specifically to optimize zero-shot mutation scoring, achieving 0.47 Spearman rho (41-DMS average, Meier et al. 2021)
Five-model ensemble (n1-n5) reduces prediction variance and improves calibration compared to single-model approaches — each model is independently initialized, providing partially orthogonal learned representations
Individual model variants (n1-n5) run on CPU with only 8GB RAM, making ESM1v the most cost-effective option for variant scoring at scale without GPU infrastructure
Validated in genome-wide variant effect prediction — ESM1b (closely related) was used to predict all ~450 million possible missense variants in the human genome, outperforming existing methods on ClinVar/HGMD
Simple, focused API — single predict action that returns sorted amino acid probabilities at masked positions, purpose-optimized for the variant scoring use case
MIT license with no commercial restrictions, suitable for clinical, academic, and commercial variant interpretation applications
Published benchmarks across 41 DMS datasets demonstrate consistent performance gains from the ensemble approach over single-model baselines

Limitations

Single-task model — only variant scoring via masked prediction; no encode action for general-purpose embeddings and no log-probability scoring for full-sequence fitness assessment
Maximum sequence length of 1022 residues — shorter than ESM2 (2048) and ESMC (2048), though matching ESM-1b's architectural limit; large proteins approaching this limit may see degraded predictions
Requires exactly one mask token per sequence — cannot score multiple positions simultaneously or perform multi-site variant assessment in a single call
No structure awareness — purely sequence-based scoring misses structural context that can distinguish surface mutations from buried core disruptions
The 'all' variant loads all 5 models simultaneously, requiring T4 GPU and 28GB RAM — individual models are cheap but the full ensemble has higher resource requirements
Older generation model (2021) — ESM2 and ESMC have improved training procedures, and newer methods such as PoET outperform ESM1v on recent benchmarks
Cannot score insertions or deletions (indels) — only single-site substitution scoring is supported; PoET handles indels natively

Reach for something else when

You need general-purpose protein embeddings for downstream ML tasks — use ESM2 or ESMC, which provide rich embed/encode actions alongside variant scoring
You need to score protein sequences longer than 1022 residues — use ESM2 (up to 2048 residues) or ESMC (up to 2048 residues)
You need to score insertions or deletions (indels) — use PoET, which is the only model in this group that natively handles variable-length variants
You need structure-aware variant scoring — structure-integrated models outperform sequence-only methods; ESM1v has no structural awareness
You need MSA-conditioned variant scoring for maximum accuracy on proteins with deep evolutionary alignments — use MSA Transformer
You want the latest-generation model for new projects — use ESM2 log_prob or ESMC log_prob, which provide comparable or better variant scoring with broader functionality

Alternatives

Model	Better when	Worse when
esm2	ESM2 provides log_prob for variant scoring plus embeddings, masked prediction, and contact maps in a single model; longer context (2048 residues), five size variants, and broader functionality make it more versatile	ESM2 uses a single model for variant scoring, not a 5-model ensemble, so predictions may have higher variance; ESM1v's dedicated optimization on UniRef90 can yield marginally better variant effect correlations
esmc	ESMC provides updated architecture with better parameter efficiency, broader action repertoire (encode, predict, log_prob), and longer context (2048 residues)	ESMC does not offer a multi-model ensemble for variance reduction; ESM1v's focused design and CPU-friendly individual models are more cost-effective for high-throughput variant screening
esm1b	ESM-1b provides general-purpose embeddings (encode action) and longer context (1022 residues) in addition to variant scoring; broader functionality for non-variant tasks	ESM-1b uses a single model rather than ESM1v's 5-model ensemble, and was trained on the smaller UniRef50 dataset; ESM1v has better variant-specific optimization

API & schema¶

Variants

Variant	Endpoint slug	GPU	CPU	Memory
n1	`esm1v-n1`	CPU	2.0	8 GB
n2	`esm1v-n2`	CPU	2.0	8 GB
n3	`esm1v-n3`	CPU	2.0	8 GB
n4	`esm1v-n4`	CPU	2.0	8 GB
n5	`esm1v-n5`	CPU	2.0	8 GB
all	`esm1v-all`	t4	4.0	28 GB

Call an action with POST /api/v1/{slug}/{action} — the request envelope is {"items": [...], "params": {...}} and a success returns {"results": [...]}. See the HTTP API page for the base URL, error shape, and full contract.

`predict`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/esm1v-n1/predict \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

Request — ESM1vPredictRequest

Field	Type	Required	Constraints	Description
`items`	list[ESM1vPredictRequestItem]	yes	items 1–5	Batch of inputs to process in a single request. Up to 5 sequences per request.

Nested types

ESM1vPredictRequestItem

Field	Type	Required	Constraints	Description
`sequence`	string	yes	len 1–1022	A protein sequence in single-letter amino-acid codes with exactly one token marking the position to predict.

Raw JSON Schema

{
  "$defs": {
    "ESM1vPredictRequestItem": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A protein sequence in single-letter amino-acid codes with exactly one <mask> token marking the position to predict.",
          "maxLength": 1022,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "ESM1vPredictRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 5 sequences per request.",
      "items": {
        "$ref": "#/$defs/ESM1vPredictRequestItem"
      },
      "maxItems": 5,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ESM1vPredictRequest",
  "type": "object"
}

Response — ESM1vPredictResponse

Field	Type	Required	Constraints	Description
`results`	list[list[ESM1vPredictResponseLabel]] \| list[object]	yes		Per-input results, returned in the same order as the request items.

Nested types

ESM1vPredictResponseLabel

Field	Type	Required	Description
`token`	integer	yes	Tokenizer vocabulary ID for the predicted amino acid.
`token_str`	string	yes	Single-letter amino-acid code for the predicted residue.
`score`	number	yes	Model probability for this amino acid at the masked position.
`sequence`	string	yes	Full protein sequence with the masked position filled by this amino acid.

Raw JSON Schema

{
  "$defs": {
    "ESM1vPredictResponseLabel": {
      "properties": {
        "token": {
          "description": "Tokenizer vocabulary ID for the predicted amino acid.",
          "title": "Token",
          "type": "integer"
        },
        "token_str": {
          "description": "Single-letter amino-acid code for the predicted residue.",
          "title": "Token Str",
          "type": "string"
        },
        "score": {
          "description": "Model probability for this amino acid at the masked position.",
          "title": "Score",
          "type": "number"
        },
        "sequence": {
          "description": "Full protein sequence with the masked position filled by this amino acid.",
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "token",
        "token_str",
        "score",
        "sequence"
      ],
      "title": "ESM1vPredictResponseLabel",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "anyOf": [
        {
          "items": {
            "items": {
              "$ref": "#/$defs/ESM1vPredictResponseLabel"
            },
            "type": "array"
          },
          "type": "array"
        },
        {
          "items": {
            "patternProperties": {
              "^esm1v-n[1-5]$": {
                "items": {
                  "$ref": "#/$defs/ESM1vPredictResponseLabel"
                },
                "type": "array"
              }
            },
            "type": "object"
          },
          "type": "array"
        }
      ],
      "description": "Per-input results, returned in the same order as the request items.",
      "title": "Results"
    }
  },
  "required": [
    "results"
  ],
  "title": "ESM1vPredictResponse",
  "type": "object"
}

Usage¶

One-line summary: Zero-shot protein variant effect prediction using a 5-model ensemble of 650M-parameter masked language models, predicting the functional impact of single amino acid mutations from sequence context alone.

Overview¶

ESM1v is a protein language model from Meta AI (Meier et al. 2021) designed for zero-shot prediction of the effects of mutations on protein function. It consists of five independently trained 650M-parameter Transformer encoder models (n1--n5). Given a protein sequence with a single masked position, each model predicts the probability distribution over all 20 standard amino acids at that position, enabling variant effect scoring without any task-specific training data.

This catalog deploys all 5 individual models (n1--n5) plus an "all" variant that loads all 5 simultaneously for ensemble predictions.

Architecture¶

Property	Value
Architecture	Transformer encoder (BERT-style, 33 layers)
Parameters	650M per model
Ensemble size	5 models (n1--n5)
Training data	UniRef90 (~98M sequences)
Training objective	Masked Language Modeling (MLM)
Input	Protein sequence with one `<mask>` token
Output	Sorted amino acid probabilities at masked position

Capabilities & Limitations¶

CAN be used for: - Zero-shot prediction of single amino acid mutation effects on protein function - Computing amino acid probability distributions at any position in a protein - Ensemble-averaged predictions for reduced variance (via "all" variant) - Batch processing up to 5 sequences per request

CANNOT be used for: - Multi-site mutation effects (only one <mask> per sequence) - Sequences longer than 1022 residues - Generating protein embeddings (use ESM2 or ESMC for embeddings) - Structure-aware predictions (sequence-only model) - Absolute fitness prediction (outputs are relative probabilities)

Usage Examples¶

Predict variant effects at a single position¶

from models.esm1v.schema import (
    ESM1vPredictRequest,
    ESM1vPredictRequestItem,
)

# Mask position 5 to predict which amino acids are compatible
request = ESM1vPredictRequest(
    items=[
        ESM1vPredictRequestItem(
            sequence="MKTAY<mask>NNKELSKDVR"
        )
    ]
)

Batch prediction for multiple positions¶

from models.esm1v.schema import (
    ESM1vPredictRequest,
    ESM1vPredictRequestItem,
)

# Test multiple positions in the same protein
request = ESM1vPredictRequest(
    items=[
        ESM1vPredictRequestItem(sequence="<mask>KTAYVNNKELSKDVR"),
        ESM1vPredictRequestItem(sequence="M<mask>TAYVNNKELSKDVR"),
        ESM1vPredictRequestItem(sequence="MK<mask>AYVNNKELSKDVR"),
    ]
)

Architecture & training¶

Architecture¶

Model Type & Innovation¶

ESM1v is a protein language model from Meta AI (Meier et al. 2021) specifically designed for zero-shot prediction of the effects of mutations on protein function. The key innovation is training an ensemble of five independently initialized ESM-1b-scale models on UniRef90, then using the masked marginal probability at mutation sites to predict functional effects without any task-specific fine-tuning.

The model uses a standard Transformer encoder (BERT-style) architecture with masked language modeling (MLM). For variant effect prediction, the target residue position is masked, and the model predicts the probability distribution over all 20 amino acids at that position. The log-likelihood ratio between the mutant and wild-type amino acid serves as a zero-shot fitness score.

Parameters & Layers¶

Component	Details
Architecture	Transformer encoder (BERT-style)
Parameters per model	650M
Ensemble size	5 models (n1--n5), each independently initialized
Layers	33 Transformer layers
Training objective	Masked Language Modeling (MLM)
Input	Protein sequence with `<mask>` token at position of interest
Output	Per-position amino acid probabilities over 20 standard amino acids

Training Data¶

Property	Details
Training dataset	UniRef90 (UR90)
Dataset size	~98M unique protein sequences
Training approach	Standard MLM with 15% masking rate
Ensemble strategy	5 independent training runs with different random seeds

Loss Function & Objective¶

ESM1v is trained with standard Masked Language Modeling (MLM) cross-entropy loss:

L = -sum_i log P(aa_i | context_with_mask_at_i)

For variant effect prediction at inference time, the model masks the target position and computes the log-likelihood of each possible amino acid at that position. The score for a mutation X->Y at position i is:

score(X->Y, i) = log P(Y | context_masked_at_i) - log P(X | context_masked_at_i)

Tokenization / Input Processing¶

Input format: Amino acid sequence string with exactly one <mask> token at the position of interest
Validation: Extended amino acid alphabet plus <mask> special token
Single mask requirement: Exactly one <mask> token must be present in each sequence
Maximum length: 1022 residues (1024 tokens - 2 for BOS/EOS; matches ESM-1b's architectural limit)
Tokenization: Uses the ESM tokenizer from HuggingFace Transformers (EsmTokenizer)
Inference: HuggingFace fill-mask pipeline, filtered to 20 standard amino acids only

Performance & Benchmarks¶

Published Benchmarks¶

From Meier et al., NeurIPS (2021):

Model	Spearman rho (DMS average) ↑	Method
ESM1v (5-model avg)	0.47	Zero-shot (no task-specific training)
ESM-1b	0.43	Zero-shot
EVmutation	0.42	MSA-based (requires alignment)
DeepSequence	0.41	VAE, MSA-based
Random	0.00	Baseline

Evaluated across 41 deep mutational scanning (DMS) datasets.

BioLM Verification Results¶

Variant	Action	Tolerance	Status
n1	predict	rel_tol 1e-4	PASS
n2	predict	rel_tol 1e-4	PASS
n3	predict	rel_tol 1e-4	PASS
n4	predict	rel_tol 1e-4	PASS
n5	predict	rel_tol 1e-4	PASS
all	predict	rel_tol 1e-4	PASS

Comparison to Alternatives¶

Model	Strength	When to prefer
ESM1v	Zero-shot variant effect prediction, ensemble	Single-site mutation effect prediction
ESM2	Newer architecture, embeddings, multiple sizes	General protein representation, embeddings
ESMC	Latest generation, better embeddings	General protein tasks, embeddings
GEMME	Evolutionary coupling-based	When MSA is available and relevant
PoET	Generative model, MSA-conditioned	When MSA-conditioned scoring is desired

Strengths & Limitations¶

Pros¶

Zero-shot variant effect prediction without any task-specific training data
Ensemble of 5 models reduces prediction variance
"all" variant loads all 5 models for ensemble predictions in a single request
Individual models (n1--n5) available for faster single-model inference
Well-validated across 41 DMS datasets

Cons¶

Limited to single-site mutations (one <mask> per sequence)
Maximum sequence length of 1022 residues
Does not capture epistatic effects between multiple mutations
CPU-based inference for individual models (GPU only for "all" variant)
Returns probabilities at masked positions, not directly interpretable absolute fitness scores

Known Failure Modes¶

Sequences near or exceeding 1022 residues may be truncated
Multi-site mutations require separate queries per position
Proteins with few homologs in UniRef90 may have less reliable predictions
Active site residues under strong functional selection may not be well-predicted by sequence-only models

Implementation Details¶

Inference Pipeline¶

Request
  |-- 1. Validate sequence (extended AA + <mask>, exactly one mask)
  |-- 2. For each model in pipeline:
  |     |-- 2a. Run HuggingFace fill-mask pipeline
  |     |-- 2b. Filter to 20 standard amino acids
  |     |-- 2c. Sort by score (descending)
  |-- 3a. [individual variant] Return sorted predictions
  |-- 3b. [all variant] Return dict mapping each model to its predictions

Memory & Compute Profile¶

Resource	Individual (n1--n5)	All
GPU	None (CPU-only)	T4
Memory	8 GB RAM	28 GB RAM
CPU	2.0 cores	4.0 cores
Models loaded	1	5

Determinism & Reproducibility¶

Setting	Value
Torch manual seed	Yes (42 at model load)
CUDA manual seed	Yes (42, if available)
Deterministic outputs	Yes (predict is deterministic given same input)

Caching Behavior¶

Model weights are cached in R2 and loaded from the container image snapshot on warm starts. Response-level caching is not performed in the model container; operators may layer a cache in front of the endpoint if desired.

Versions & Changelog¶

Version	Date	Changes
v1	2024	Initial implementation with predict action, all 6 variants

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

ESM1v is designed for proteins broadly. It was trained on the UniRef90 database, which represents the full diversity of known protein sequences. The model is particularly suited for predicting the effects of single amino acid mutations on protein function, using a zero-shot approach that requires no task-specific training data.

Important coverage notes: - Works with any protein sequence up to 1022 residues - Requires exactly one <mask> token per sequence (single-site analysis) - Trained on UniRef90, covering all protein families with known sequences - Does not handle nucleic acid sequences or non-standard amino acids

Cross-Applicability¶

Molecule Type	Applicability	Evidence	Caveats
Globular proteins	High	Primary training target, validated on 41 DMS datasets	Standard application
Enzymes	High	Well-represented in training data and DMS benchmarks	Active site mutations may need structural context
Membrane proteins	Moderate	Present in UniRef90	Fewer DMS benchmarks available
Antibodies	Moderate	Antibody sequences in UniRef90	Use antibody-specific models for CDR engineering
Viral proteins	Moderate	Present in UniRef90	Rapid evolution may limit context
Peptides	Low	Very short sequences lack context for meaningful MLM	Minimum ~20 residues recommended
Disordered proteins	Moderate	Present in training data	Mutations in disordered regions may have different selection pressures

Biological Problems Addressed¶

Zero-Shot Variant Effect Prediction (Published)¶

Biological context: Understanding how single amino acid mutations affect protein function is central to genetics, molecular biology, and drug development. Deep mutational scanning (DMS) experiments systematically measure the functional effects of all possible single mutations at every position, but these experiments are expensive, require functional assays, and have been completed for only a small fraction of the proteome. Computational prediction of variant effects can prioritize mutations for experimental testing and interpret clinical variants of uncertain significance.

How ESM1v helps: For any protein sequence, ESM1v can predict the relative fitness effect of all possible amino acid substitutions at a given position in a zero-shot manner -- without any task-specific training data. The user provides the sequence with a <mask> token at the position of interest, and the model returns the probability of each of the 20 standard amino acids at that position. The log-likelihood difference between the mutant and wild-type amino acid serves as a proxy for the mutation's functional effect.

Output interpretation: The response contains a sorted list of amino acid predictions with scores for each model (or all 5 models for the "all" variant). Higher scores indicate amino acids that are more compatible with the sequence context. To compute a variant effect score, compare the score of the mutant amino acid to the wild-type amino acid: positive differences suggest tolerated or beneficial mutations, while negative differences suggest deleterious mutations.

Clinical Variant Interpretation (Published)¶

Biological context: Human genetic studies increasingly identify variants of uncertain significance (VUS) -- genetic changes whose functional impact is unknown. For protein-coding genes, predicting whether a missense variant is pathogenic or benign is critical for clinical genetics, particularly in cancer genomics and rare disease diagnosis.

How ESM1v helps: By computing the log-likelihood of the wild-type versus mutant amino acid at a given position, ESM1v provides a sequence-based prediction of whether a mutation is likely to be functionally disruptive. The ensemble of 5 models reduces variance in predictions. This information can complement other pathogenicity predictors (SIFT, PolyPhen, CADD) that use different features.

Protein Engineering Guidance (Anticipated)¶

Biological context: When engineering a protein for improved properties (stability, activity, specificity), researchers need to decide which positions to mutate and which amino acid substitutions to try. Exhaustive experimental screening is impractical for proteins with hundreds of residues and 19 possible substitutions per position.

How ESM1v helps: By scoring all possible substitutions at each position, ESM1v can identify positions where the native amino acid is weakly preferred (tolerable to mutate) versus positions where it is strongly preferred (likely essential). This information can prioritize positions for mutagenesis and suggest which substitutions are most likely to be tolerated by the protein fold. However, ESM1v predicts evolutionary fitness, not specific engineered properties, so predictions should be combined with domain knowledge.

Applied Use Cases¶

ESM1v has been used in several published studies since its release in 2021, particularly for benchmarking variant effect prediction methods and interpreting clinical variants.

Genome-wide missense variant effect prediction (Brandes et al., Nature Genetics 2023, DOI: 10.1038/s41588-023-01465-0): Extended ESM-1b (same architecture as ESM-1v) to predict all ~450 million possible missense variant effects in the human genome, outperforming existing methods on ~150,000 ClinVar/HGMD variants and 28 DMS datasets. Demonstrates the scalability of masked language model scoring to clinical genomics.

VariPred — combining embeddings and log-likelihood ratios (DOI: 10.1038/s41598-024-51489-7, 2024): Benchmarks ESM-1v alongside ESM-1b and ESM-2 for missense pathogenicity prediction. Combining masked log-likelihood ratios with residue embeddings achieves MCC of 0.714 without structural features or MSAs, showing ESM-1v embeddings carry complementary signal to its scoring output.

Fine-tuning on DMS data improves variant effect prediction (arXiv: 2405.06729, 2024): Fine-tuning ESM-1v on deep mutational scanning data with a Normalised Log-odds Ratio head consistently improves variant effect prediction across ProteinGym and ClinVar benchmarks, demonstrating that task-specific adaptation can substantially boost the zero-shot baseline.

ESM-Scan — in silico deep mutational scanning tool (DOI: 10.1002/pro.5221, 2024): Builds ESM-Scan on ESM-1v zero-shot predictions for genome-wide in silico deep mutational scanning, enabling prediction of preferential amino acid substitutions for protein engineering at scale.

Rep2Mut-V2 — variant effect prediction from ESM-1v representations (DOI: 10.1016/j.csbj.2023.11.017, 2023): Uses ESM-1v 33rd-layer representations to predict variant effects across 38 proteins (118,933 variants), achieving Spearman 0.7 and surpassing six state-of-the-art methods, illustrating that ESM-1v's learned representations are effective features beyond direct masked scoring.

Predecessor Models¶

ESM-1b (Rives et al., 2021): The base model architecture that ESM1v extends. ESM-1b is a single 650M parameter model trained on UniRef50. ESM1v improves upon ESM-1b by training on UniRef90 (larger, less redundancy-reduced dataset) and using a 5-model ensemble for reduced variance.

Complementary Models¶

ESM1v works well in combination with other models in the catalog:

ESM2 / ESMC: For general protein embeddings and representation. ESM1v focuses on variant effect prediction, while ESM2/ESMC provide richer embeddings for downstream tasks.
ESMFold: For understanding the structural context of mutations. Pipeline: predict structure to visualize mutation location, use ESM1v for functional effect prediction.

Alternative Models¶

Alternative	Advantage over ESM1v	Disadvantage vs ESM1v
ESM2	Newer architecture, multiple sizes, embeddings	Not specifically optimized for variant effects
GEMME	Uses evolutionary conservation from MSA	Requires MSA computation
PoET	Generative model, MSA-conditioned	Requires MSA, more complex setup
EVmutation	Established MSA-based method	Requires MSA, older approach

Biological Background¶

Variant Effect Prediction¶

The effect of a single amino acid mutation on protein function depends on multiple factors:

Structural context: Is the position buried in the protein core or exposed on the surface? Buried positions are more sensitive to mutations due to packing constraints.
Evolutionary conservation: Positions that are highly conserved across homologs are more likely to be functionally important. Mutations at these positions are more likely to be deleterious.
Physicochemical properties: Substitutions between amino acids with similar properties (e.g., Leu -> Ile, both hydrophobic) are generally better tolerated than those between dissimilar amino acids (e.g., Gly -> Trp).
Functional role: Active site residues, ligand-binding residues, and residues involved in protein-protein interactions are often under strong selection and sensitive to mutation.

Deep Mutational Scanning (DMS)¶

Deep mutational scanning is an experimental technique that systematically measures the functional effects of all possible single amino acid mutations across a protein. A library of mutant variants is created, subjected to a functional selection, and the relative fitness of each variant is quantified by sequencing. DMS datasets serve as the primary benchmark for computational variant effect predictors like ESM1v.

Masked Language Modeling for Proteins¶

Protein language models like ESM1v treat amino acid sequences as "sentences" and learn statistical patterns from millions of natural sequences. The masked language modeling objective -- predicting randomly masked amino acids from their sequence context -- implicitly learns the evolutionary constraints on each position. At inference time, masking a specific position and examining the model's predictions reveals which amino acids are compatible with the surrounding sequence context, providing a proxy for evolutionary fitness.

Ensemble Approach¶

ESM1v uses an ensemble of 5 independently trained models (n1--n5) rather than a single model. Each model is trained with a different random initialization, leading to partially independent learned representations. Averaging predictions across the ensemble reduces noise and improves calibration. The "all" variant loads all 5 models simultaneously for ensemble prediction, while individual variants (n1--n5) are available for faster single-model queries.

Sources & license¶

License: MIT (text)

Papers

Language models enable zero-shot prediction of the effects of mutations on protein function — NeurIPS, 2021 · DOI arXiv

Source repositories

github: https://github.com/facebookresearch/esm
huggingface: https://huggingface.co/facebook/esm1v_t33_650M_UR90S_1

Cite

@inproceedings{meier2021language,
  title={Language models enable zero-shot prediction of the effects of mutations on protein function},
  author={Meier, Joshua and Rao, Roshan and Verkuil, Robert and Liu, Jason and Sercu, Tom and Rives, Alexander},
  booktitle={Advances in Neural Information Processing Systems},
  year={2021}
}

ESM1v¶

At a glance¶

API & schema¶

predict¶

Usage¶

Overview¶

Architecture¶

Capabilities & Limitations¶

Usage Examples¶

Predict variant effects at a single position¶

Batch prediction for multiple positions¶

Architecture & training¶

Architecture¶

Model Type & Innovation¶

Parameters & Layers¶

Training Data¶

Loss Function & Objective¶

Tokenization / Input Processing¶

Performance & Benchmarks¶

Published Benchmarks¶

BioLM Verification Results¶

Comparison to Alternatives¶

Strengths & Limitations¶

Pros¶

Cons¶

Known Failure Modes¶

Implementation Details¶

Inference Pipeline¶

Memory & Compute Profile¶

Determinism & Reproducibility¶

Caching Behavior¶

Versions & Changelog¶

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

Cross-Applicability¶

Biological Problems Addressed¶

Zero-Shot Variant Effect Prediction (Published)¶

Clinical Variant Interpretation (Published)¶

Protein Engineering Guidance (Anticipated)¶

Applied Use Cases¶

Related Models¶

Predecessor Models¶

Complementary Models¶

Alternative Models¶

Biological Background¶

Variant Effect Prediction¶

Deep Mutational Scanning (DMS)¶

Masked Language Modeling for Proteins¶

Ensemble Approach¶

Sources & license¶

`predict`¶