ESM-1b¶

Legacy masked protein language model (650M parameters) from Meta AI/FAIR that produces sequence embeddings, masked-token predictions, and per-sequence log-probabilities for proteins up to 1022 residues. Superseded by ESM-2 -- use ESM-2 for new work.**

License: MIT · Molecules: protein · Tasks: embedding, sequence_completion, property_prediction

Actions: encode, predict, log_prob · Variants: 1

At a glance¶

Use it when

You are reproducing published results from studies that specifically used ESM-1b embeddings and need exact embedding compatibility for comparison
You have an existing pipeline built on ESM-1b that would require significant refactoring to switch to ESM2, and the performance difference is not critical for your application
You need backward compatibility with legacy protein ML workflows or databases that store ESM-1b-derived features
You are conducting a comparative study benchmarking multiple protein language model generations and need ESM-1b as a historical reference point
You need domain-adaptive pretraining on a specialized protein class and ESM-1b is the pretrained checkpoint used in the published method you are following

Strengths

Historical significance — the first protein language model to demonstrate that transformer representations spontaneously encode 3D structural information (contact maps from attention heads), a finding that launched the field of protein foundation models
Well-validated in published applied studies: plant PPI prediction (AUPR 0.810), domain-adaptive pretraining for DNA-binding proteins, and few-shot fitness prediction across 87 DMS datasets
Permissive MIT license with no commercial restrictions, suitable for any deployment scenario without licensing concerns
Full action repertoire — encode, predict, log_prob — provides embeddings, masked token prediction, and sequence fitness scoring from a single model
1280-dimensional embeddings are compatible with many published downstream pipelines originally designed for ESM-1b, enabling direct reproduction of literature results
Runs efficiently on T4 GPU (16GB VRAM) at 650M parameters, making it accessible on standard cloud GPU instances

Limitations

Strictly superseded by ESM2-650M on every benchmark — ESM2 provides better representations at the same parameter count with a longer context window (2048 vs 1024 tokens)
Maximum sequence length of only 1022 residues (1024 tokens including BOS/EOS) — shorter than ESM2's 2048 residues, causing more frequent truncation of large proteins
Single size variant only (650M) — no scaling flexibility; ESM2 offers 8M to 3B parameters for different compute budgets
No structure awareness — purely sequence-based, missing structural context that structure-aware models such as ProstT5 provide
Training on UniRef50 with older procedures — ESMC and ESM2 benefit from improved training techniques and potentially broader data coverage
Representation quality degrades for membrane proteins, fibrous proteins, intrinsically disordered proteins, and very short peptides (<30 residues) due to under-representation in training data
No generative capability — cannot design or sample novel protein sequences

Reach for something else when

You are starting a new project with no legacy constraints — use ESM2-650M instead, which is strictly superior on all benchmarks with the same parameter count and longer context
You need the highest-quality protein embeddings — use ESM2-3B or ESMC for improved representation quality
You need structure-aware representations — use ProstT5 when structural information is available
You need to process sequences longer than 1022 residues — use ESM2 (up to 2048 residues) or ESMC (up to 2048 residues)
You need variant effect prediction with competitive accuracy — use ESM1v (5-model ensemble optimized for this task), ESM2 log_prob, or PoET (MSA-conditioned scoring)
You need multiple size variants for cost-performance optimization — use ESM2, which offers 8M to 3B parameters

Alternatives

Model	Better when	Worse when
esm2	ESM2-650M is strictly superior on every evaluated benchmark with the same architecture and parameter count, plus longer context (2048 vs 1022 residues) and five size variants for flexible deployment	Never — ESM2 dominates ESM-1b in all respects; the only reason to use ESM-1b is backward compatibility with existing pipelines
esmc	ESMC-600M provides state-of-the-art parameter efficiency with updated training, achieving quality comparable to ESM2-3B — a two-generation improvement over ESM-1b	ESM-1b has MIT license while ESMC requires EvolutionaryScale Cambrian Open License review; ESM-1b has more published downstream applications due to its longer history
esm1v	ESM-1v uses a 5-model ensemble trained on UniRef90 specifically for variant effect prediction, achieving higher accuracy on DMS benchmarks than ESM-1b's single-model scoring	ESM-1v is specialized for variant prediction only with a single predict action; ESM-1b provides general-purpose embeddings (encode) and broader action coverage
prostt5	ProstT5 provides bidirectional sequence-structure translation and structure-aware embeddings that capture fold-level information ESM-1b cannot access from sequence alone	ProstT5's generation and structure-dependent modes add complexity; ESM-1b is simpler for pure sequence embedding tasks where structural context is not needed

API & schema¶

Call an action with POST /api/v1/{slug}/{action} — the request envelope is {"items": [...], "params": {...}} and a success returns {"results": [...]}. See the HTTP API page for the base URL, error shape, and full contract.

`encode`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/esm1b/encode \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

Request — ESM1bEncodeRequest

Field	Type	Required	Constraints	Description
`params`	ESM1bEncodeRequestParams	no		Optional parameters controlling this action (defaults are used when omitted).
`items`	list[ESM1bEncodeRequestItem]	yes	items 1–8	Batch of inputs to process in a single request. Up to 8 sequences per request.

Nested types

ESM1bEncodeIncludeOptions

Allowed values: mean, per_residue, per_residue, bos, logits, attentions

ESM1bEncodeRequestItem

Field	Type	Required	Constraints	Description
`sequence`	string	yes	len 1–1022	A protein sequence in single-letter amino-acid codes.

ESM1bEncodeRequestParams

Field	Type	Required	Constraints	Description
`repr_layers`	list[integer]	no	default `[-1]`	Hidden layers whose representations to return (negative indexes count from the last layer).
`include`	list[ESM1bEncodeIncludeOptions]	no	default `['mean']`	Output types to include in the encode response; controls which embeddings, logits, or attention outputs are returned.

Raw JSON Schema

{
  "$defs": {
    "ESM1bEncodeIncludeOptions": {
      "enum": [
        "mean",
        "per_residue",
        "per_residue",
        "bos",
        "logits",
        "attentions"
      ],
      "title": "ESM1bEncodeIncludeOptions",
      "type": "string"
    },
    "ESM1bEncodeRequestItem": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A protein sequence in single-letter amino-acid codes.",
          "maxLength": 1022,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "ESM1bEncodeRequestItem",
      "type": "object"
    },
    "ESM1bEncodeRequestParams": {
      "additionalProperties": false,
      "properties": {
        "repr_layers": {
          "description": "Hidden layers whose representations to return (negative indexes count from the last layer).",
          "items": {
            "type": "integer"
          },
          "title": "Repr Layers",
          "type": "array",
          "default": [
            -1
          ]
        },
        "include": {
          "description": "Output types to include in the encode response; controls which embeddings, logits, or attention outputs are returned.",
          "items": {
            "$ref": "#/$defs/ESM1bEncodeIncludeOptions"
          },
          "title": "Include",
          "type": "array",
          "default": [
            "mean"
          ]
        }
      },
      "title": "ESM1bEncodeRequestParams",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "params": {
      "$ref": "#/$defs/ESM1bEncodeRequestParams",
      "description": "Optional parameters controlling this action (defaults are used when omitted)."
    },
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 8 sequences per request.",
      "items": {
        "$ref": "#/$defs/ESM1bEncodeRequestItem"
      },
      "maxItems": 8,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ESM1bEncodeRequest",
  "type": "object"
}

Response — ESM1bEncodeResponse

Field	Type	Required	Constraints	Description
`results`	list[ESM1bEncodeResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

ESM1bEncodeResponseResult

Field	Type	Required	Description
`sequence_index`	integer	yes	Index of the corresponding input sequence within the request batch.
`embeddings`	list[LayerEmbedding] \| null	no	Per-layer mean-pooled embedding vectors; present only when 'mean' is requested.
`bos_embeddings`	list[LayerEmbedding] \| null	no	Per-layer beginning-of-sequence (BOS/CLS) token embedding vectors; present only when 'bos' is requested.
`residue_embeddings`	list[LayerPerTokenEmbeddings] \| null	no	Per-residue embedding vectors.
`attentions`	list[list[number]] \| null	no	Averaged attention weights across all layers and heads; present only when 'attentions' is requested.
`logits`	list[list[number]] \| null	no	Per-position logits over the model vocabulary.
`vocab_tokens`	list[string] \| null	no	Vocabulary token order corresponding to the logits columns.

LayerEmbedding

Field	Type	Required	Constraints	Description
`layer`	integer	yes		Model layer this representation was taken from.
`embedding`	list[number]	yes		Embedding vector for the sequence at this layer.

LayerPerTokenEmbeddings

Field	Type	Required	Constraints	Description
`layer`	integer	yes		Model layer this representation was taken from.
`embeddings`	list[list[number]]	yes		Per-residue (per-token) embedding vectors for this layer.

Raw JSON Schema

{
  "$defs": {
    "ESM1bEncodeResponseResult": {
      "exclude_none": true,
      "exclude_unset": true,
      "properties": {
        "sequence_index": {
          "description": "Index of the corresponding input sequence within the request batch.",
          "title": "Sequence Index",
          "type": "integer"
        },
        "embeddings": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/LayerEmbedding"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Per-layer mean-pooled embedding vectors; present only when 'mean' is requested.",
          "title": "Embeddings"
        },
        "bos_embeddings": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/LayerEmbedding"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Per-layer beginning-of-sequence (BOS/CLS) token embedding vectors; present only when 'bos' is requested.",
          "title": "Bos Embeddings"
        },
        "residue_embeddings": {
          "anyOf": [
            {
              "items": {
                "$ref": "#/$defs/LayerPerTokenEmbeddings"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Per-residue embedding vectors.",
          "title": "Residue Embeddings"
        },
        "attentions": {
          "anyOf": [
            {
              "items": {
                "items": {
                  "type": "number"
                },
                "type": "array"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Averaged attention weights across all layers and heads; present only when 'attentions' is requested.",
          "title": "Attentions"
        },
        "logits": {
          "anyOf": [
            {
              "items": {
                "items": {
                  "type": "number"
                },
                "type": "array"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Per-position logits over the model vocabulary.",
          "title": "Logits"
        },
        "vocab_tokens": {
          "anyOf": [
            {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "Vocabulary token order corresponding to the logits columns.",
          "title": "Vocab Tokens"
        }
      },
      "required": [
        "sequence_index"
      ],
      "title": "ESM1bEncodeResponseResult",
      "type": "object"
    },
    "LayerEmbedding": {
      "properties": {
        "layer": {
          "description": "Model layer this representation was taken from.",
          "title": "Layer",
          "type": "integer"
        },
        "embedding": {
          "description": "Embedding vector for the sequence at this layer.",
          "items": {
            "type": "number"
          },
          "title": "Embedding",
          "type": "array"
        }
      },
      "required": [
        "layer",
        "embedding"
      ],
      "title": "LayerEmbedding",
      "type": "object"
    },
    "LayerPerTokenEmbeddings": {
      "properties": {
        "layer": {
          "description": "Model layer this representation was taken from.",
          "title": "Layer",
          "type": "integer"
        },
        "embeddings": {
          "description": "Per-residue (per-token) embedding vectors for this layer.",
          "items": {
            "items": {
              "type": "number"
            },
            "type": "array"
          },
          "title": "Embeddings",
          "type": "array"
        }
      },
      "required": [
        "layer",
        "embeddings"
      ],
      "title": "LayerPerTokenEmbeddings",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/ESM1bEncodeResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "ESM1bEncodeResponse",
  "type": "object"
}

`predict`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/esm1b/predict \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

Request — ESM1bPredictRequest

Field	Type	Required	Constraints	Description
`items`	list[ESM1bPredictRequestItem]	yes	items 1–8	Batch of inputs to process in a single request. Up to 8 sequences per request.

Nested types

ESM1bPredictRequestItem

Field	Type	Required	Constraints	Description
`sequence`	string	yes	len 1–1022	A protein sequence in single-letter amino-acid codes, with one or more tokens for masked prediction.

Raw JSON Schema

{
  "$defs": {
    "ESM1bPredictRequestItem": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A protein sequence in single-letter amino-acid codes, with one or more <mask> tokens for masked prediction.",
          "maxLength": 1022,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "ESM1bPredictRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 8 sequences per request.",
      "items": {
        "$ref": "#/$defs/ESM1bPredictRequestItem"
      },
      "maxItems": 8,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ESM1bPredictRequest",
  "type": "object"
}

Response — ESM1bPredictResponse

Field	Type	Required	Constraints	Description
`results`	list[ESM1bPredictResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

ESM1bPredictResponseResult

Field	Type	Required	Description
`logits`	list[list[number]]	yes	Per-position logits over the model vocabulary.
`sequence_tokens`	list[string]	yes	Per-position input tokens, aligned with the logits.
`vocab_tokens`	list[string]	yes	Vocabulary token order corresponding to the logits columns.

Raw JSON Schema

{
  "$defs": {
    "ESM1bPredictResponseResult": {
      "properties": {
        "logits": {
          "description": "Per-position logits over the model vocabulary.",
          "items": {
            "items": {
              "type": "number"
            },
            "type": "array"
          },
          "title": "Logits",
          "type": "array"
        },
        "sequence_tokens": {
          "description": "Per-position input tokens, aligned with the logits.",
          "items": {
            "type": "string"
          },
          "title": "Sequence Tokens",
          "type": "array"
        },
        "vocab_tokens": {
          "description": "Vocabulary token order corresponding to the logits columns.",
          "items": {
            "type": "string"
          },
          "title": "Vocab Tokens",
          "type": "array"
        }
      },
      "required": [
        "logits",
        "sequence_tokens",
        "vocab_tokens"
      ],
      "title": "ESM1bPredictResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/ESM1bPredictResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "ESM1bPredictResponse",
  "type": "object"
}

`log_prob`¶

Call it

curl -X POST http://127.0.0.1:8000/api/v1/esm1b/log_prob \
  -H "Content-Type: application/json" \
  -d '{
  "items": [
    {
      "sequence": "MKTAYIAKQRQISFVKSHFSRQLEERLGLIE"
    }
  ]
}'

Request — ESM1bLogProbRequest

Field	Type	Required	Constraints	Description
`items`	list[ESM1bEncodeRequestItem]	yes	items 1–8	Batch of inputs to process in a single request. Up to 8 sequences per request.

Nested types

ESM1bEncodeRequestItem

Field	Type	Required	Constraints	Description
`sequence`	string	yes	len 1–1022	A protein sequence in single-letter amino-acid codes.

Raw JSON Schema

{
  "$defs": {
    "ESM1bEncodeRequestItem": {
      "additionalProperties": false,
      "properties": {
        "sequence": {
          "description": "A protein sequence in single-letter amino-acid codes.",
          "maxLength": 1022,
          "minLength": 1,
          "title": "Sequence",
          "type": "string"
        }
      },
      "required": [
        "sequence"
      ],
      "title": "ESM1bEncodeRequestItem",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "properties": {
    "items": {
      "description": "Batch of inputs to process in a single request. Up to 8 sequences per request.",
      "items": {
        "$ref": "#/$defs/ESM1bEncodeRequestItem"
      },
      "maxItems": 8,
      "minItems": 1,
      "title": "Items",
      "type": "array"
    }
  },
  "required": [
    "items"
  ],
  "title": "ESM1bLogProbRequest",
  "type": "object"
}

Response — ESM1bLogProbResponse

Field	Type	Required	Constraints	Description
`results`	list[ESM1bLogProbResponseResult]	yes		Per-input results, returned in the same order as the request items.

Nested types

ESM1bLogProbResponseResult

Field	Type	Required	Constraints	Description
`log_prob`	number	yes		Log-likelihood of the sequence under the model.

Raw JSON Schema

{
  "$defs": {
    "ESM1bLogProbResponseResult": {
      "properties": {
        "log_prob": {
          "description": "Log-likelihood of the sequence under the model.",
          "title": "Log Prob",
          "type": "number"
        }
      },
      "required": [
        "log_prob"
      ],
      "title": "ESM1bLogProbResponseResult",
      "type": "object"
    }
  },
  "properties": {
    "results": {
      "description": "Per-input results, returned in the same order as the request items.",
      "items": {
        "$ref": "#/$defs/ESM1bLogProbResponseResult"
      },
      "title": "Results",
      "type": "array"
    }
  },
  "required": [
    "results"
  ],
  "title": "ESM1bLogProbResponse",
  "type": "object"
}

Usage¶

One-line summary: Legacy masked protein language model (650M parameters) from Meta AI/FAIR that produces sequence embeddings, masked-token predictions, and per-sequence log-probabilities for proteins up to 1022 residues. Superseded by ESM-2 -- use ESM-2 for new work.

Overview¶

ESM-1b (Evolutionary Scale Modeling 1b) is a protein language model developed by Meta AI's Fundamental AI Research (FAIR) team (Rives et al., PNAS 2021). Trained with a masked language modeling objective on approximately 250 million protein sequences from UniRef50, it learns contextual representations of amino acids from evolutionary sequence data alone -- no structural supervision is required.

ESM-1b was the first large-scale protein language model to demonstrate that biological structure and function emerge from unsupervised learning on protein sequences. Its representations encode secondary structure, tertiary contacts, and remote homology at the fold level. This landmark finding directly motivated the development of ESM-2, ESMFold, and subsequent protein foundation models.

Legacy status: ESM-1b has been superseded by ESM-2 (Lin et al., 2023), which provides strictly better representations at the same parameter count through improved training. ESM-1b is retained in the catalog for backward compatibility and for reproducing results from published studies that specifically used this model.

Architecture¶

Property	Value
Architecture	Transformer encoder (BERT-style)
Parameters	650M
Layers	33
Hidden dimensions	1280
Attention heads	20
Training objective	Masked language modeling (MLM)
Training data	UniRef50 (UR50/S) -- ~250M sequences
Max sequence length	1022 residues
Vocabulary	33 tokens (20 standard AA + special tokens)
License	MIT

The model uses pre-activation layer normalization (differs from standard RoBERTa) and was trained with 15% masked language modeling objective.

Capabilities & Limitations¶

CAN be used for: - Generating per-residue and mean-pooled sequence embeddings for downstream ML tasks - Masked token prediction (fill-in-the-blank for protein sequences) - Zero-shot variant effect prediction via pseudo-log-likelihood scoring (log_prob) - Extracting attention maps for structural analysis - Remote homology detection via embedding space similarity

CANNOT be used for: - Sequences longer than 1022 residues (use ESM-2 for up to 2048) - 3D structure prediction (use ESMFold or Chai-1) - Sequence generation or design (encoder-only model) - Non-protein molecules (DNA, RNA, small molecules) - Multi-chain / protein complex modeling

Other considerations: - Superseded by ESM-2 for most applications -- ESM-2-650M is recommended for new work - Batch size is capped at 8 sequences per request - GPU memory snapshots provide fast cold starts - Deterministic outputs with seed=42

Usage Examples¶

# Encode -- get mean embeddings
from models.esm1b.schema import (
    ESM1bEncodeRequest,
    ESM1bEncodeRequestItem,
    ESM1bEncodeRequestParams,
)

encode_request = ESM1bEncodeRequest(
    params=ESM1bEncodeRequestParams(repr_layers=[-1], include=["mean"]),
    items=[
        ESM1bEncodeRequestItem(sequence="MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQAPILSRVGDGTQDNLSGAEKAV"),
    ],
)

# Predict -- masked token prediction
from models.esm1b.schema import ESM1bPredictRequest, ESM1bPredictRequestItem

predict_request = ESM1bPredictRequest(
    items=[
        ESM1bPredictRequestItem(sequence="MKTAYIAK<mask>RQISFVKSHFSRQLEERLGLIEVQAPILSRVGDGTQDNLSGAEKAV"),
    ],
)

# Predict log probability -- sequence scoring
from models.esm1b.schema import ESM1bLogProbRequest, ESM1bEncodeRequestItem

log_prob_request = ESM1bLogProbRequest(
    items=[
        ESM1bEncodeRequestItem(sequence="MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQAPILSRVGDGTQDNLSGAEKAV"),
    ],
)

Architecture & training¶

Architecture¶

Model Type & Innovation¶

ESM-1b is a masked protein language model based on the BERT (bidirectional transformer encoder) architecture. It was trained with a masked language modeling (MLM) objective on approximately 250 million protein sequences from UniRef50. The model learns contextual amino acid representations by predicting randomly masked tokens from surrounding sequence context.

ESM-1b was the first protein language model to demonstrate that unsupervised learning at evolutionary scale could produce representations encoding biological structure and function -- including secondary structure, tertiary contacts, and remote homology at the fold level -- without any structural supervision. This was a landmark finding: the geometry of protein structure emerges from patterns in evolutionary sequence data alone.

Legacy status: ESM-1b has been superseded by ESM-2 (Lin et al., 2023). ESM-2 at 650M parameters matches or exceeds ESM-1b on all evaluated tasks through improved training procedures and scaling. For new projects, ESM-2 is the recommended choice.

ESM-1b uses a transformer encoder with pre-activation layer normalization (differing from the standard RoBERTa post-LayerNorm convention), GELU activations, and learned positional embeddings.

Parameters & Layers¶

Component	Details
Architecture	Transformer encoder, 33 layers
Parameters	650M
Hidden dimensions	1280
Attention heads	20
Feed-forward dimensions	5120
Embedding dimensions	1280
Vocabulary size	33 tokens (20 standard AA + special tokens)
Positional encoding	Learned
Normalization	Pre-LayerNorm
Activation	GELU

Single variant only -- no size options.

Training Data¶

Property	Details
Dataset	UniRef50 (UR50/S)
Size	~250 million sequences
Clustering	50% sequence identity
Composition	Proteins from all domains of life (bacteria, archaea, eukaryota, viruses)

Known biases in the training data: - Bacterial proteins are over-represented relative to eukaryotic proteins in UniRef50 - Membrane proteins and intrinsically disordered regions are under-represented - Very long proteins (>1022 residues) were not seen in full during training due to the sequence length limit

Loss Function & Objective¶

Masked language modeling (MLM) with cross-entropy loss:

L = -Sum_i log P(x_masked_i | x_visible)

During training, 15% of input tokens are randomly masked. The model learns to predict the identity of the masked amino acid from the bidirectional sequence context. This is the standard BERT masking strategy applied to protein sequences.

Tokenization / Input Processing¶

Property	Details
Tokenizer	Character-level (one token per amino acid)
Special tokens	`<cls>` (BOS), `<eos>` (EOS), `<pad>`, `<mask>`, `<unk>`
BOS prepended	Yes
EOS appended	Yes
Extra tokens per sequence	2 (BOS + EOS)
Max sequence length	1022 residues (1024 tokens - 2 for BOS/EOS)

Input sequences are validated against the extended amino acid alphabet (20 standard + B, J, O, U, X, Z) plus the - gap character for the encode action and the <mask> token for the predict action.

Performance & Benchmarks¶

Published Benchmarks¶

Key findings from the paper (Rives et al., PNAS 2021):

Representations encode secondary structure at per-residue accuracy comparable to or better than alignment-based methods
Long-range contact prediction (top-L precision) from attention weights demonstrated structural information is learned
Remote homology detection at the fold level via embedding similarity
Systematic scaling study showed that larger models (650M) significantly outperform smaller ones (from 6-layer to 34-layer models)

The paper demonstrated that the model's internal representations spontaneously organize to reflect tertiary structure: attention heads learn to attend to residues that are physically proximal in 3D space, despite never seeing structural data during training.

BioLM Verification Results¶

The BioLM implementation uses official pre-trained weights from HuggingFace (facebook/esm1b_t33_650M_UR50S) loaded via EsmForMaskedLM.from_pretrained(). Numerical verification is performed against golden reference outputs:

Metric	Threshold	Status
Relative tolerance	1e-4	PASS
Cosine distance	< 0.02	PASS

Additional biological verification:

Test	Description	Expected	Result	Status
Log-prob scoring	Real vs shuffled ubiquitin	Real > shuffled	-0.17 vs -36.74	PASS
Embedding similarity	Human vs horse hemoglobin	>0.8	0.999	PASS
Embedding dissimilarity	Ubiquitin vs hemoglobin	<0.95	0.889	PASS
Masked prediction	Lys48 in ubiquitin	K (Lysine)	K (exact match)	PASS

Comparison to Alternatives¶

Model	Type	Key Advantage	Key Disadvantage
ESM-1b (this)	Protein LM	Historical significance; well-characterized behavior	Superseded by ESM-2 at every scale
ESM-2 (650M)	Protein LM	Better representations at same parameter count; recommended successor	Same architecture class
ProtTrans (ProtT5)	Protein LM	Encoder-decoder, generation possible	Larger compute footprint
SaProt	Structure-aware LM	Incorporates 3D structure tokens	Requires structure input
ESM3	Multimodal protein LM	Handles sequence + structure + function	Newer, less widely benchmarked

Recommendation: Use ESM-2 instead of ESM-1b for all new work. ESM-1b is retained in the catalog for backward compatibility and for reproducing results from papers that specifically used ESM-1b.

Strengths & Limitations¶

Pros¶

Historically important: first large-scale demonstration that protein LMs learn biological structure
Well-characterized in published literature -- hundreds of downstream studies used ESM-1b
Fast single-sequence inference, no MSA required
MIT licensed with no restrictions
Rich output options: embeddings, logits, attentions, log-probabilities

Cons¶

Superseded by ESM-2 for all tasks -- ESM-2 provides strictly better representations
Encoder-only architecture cannot generate sequences
Single-chain only -- no multi-chain or complex modeling
Maximum 1022 residues (shorter than ESM-2's 2048 limit)
No contact map extraction in BioLM implementation (ESM-2 provides this)

Known Failure Modes¶

Very short sequences (< 10 residues): Embeddings lack meaningful context; consider peptide-specific models
Poly-amino acid repeats (e.g., poly-Q, poly-A): Repetitive sequences produce low-quality representations
Sequences near max length (>1000 residues): Edge effects at the C-terminal boundary
Non-standard residues: Characters outside the extended AA alphabet plus - are rejected by validation

Implementation Details¶

Inference Pipeline¶

Request
  |-- 1. Validate sequences (alphabet, length, batch size)
  |-- 2. Tokenize with HuggingFace EsmTokenizer (add BOS/EOS, pad)
  |-- 3. Forward pass on GPU (torch.no_grad)
  |     |-- Encoder: 33 transformer layers
  |     |-- Output: hidden_states, logits, attentions
  |-- 4. Post-process per include options:
  |     |-- mean: average per-token embeddings (excluding BOS/EOS)
  |     |-- per_token: return all token embeddings
  |     |-- bos: return BOS (CLS) token embedding
  |     |-- logits: slice to canonical AA vocab
  |     |-- attentions: average over layers and heads
  |-- 5. Return ESM1bEncodeResponse

For predict, the pipeline tokenizes sequences containing <mask> tokens, runs a forward pass, and returns per-position logits over the canonical amino acid vocabulary.

For log_prob, the pipeline calls _encode_forward_pass with include=["logits"], then computes log-softmax and sums log P(residue_i) at each position for canonical amino acids.

Memory & Compute Profile¶

Property	Value
GPU	T4 (16 GB VRAM)
System memory	16 GB
CPU	4 cores
Batch size	8 sequences

Attention computation scales as O(n^2) with sequence length. For long sequences (>500 residues), GPU memory usage increases significantly.

Determinism & Reproducibility¶

Setting	Value
`torch.manual_seed`	42
`torch.cuda.manual_seed_all`	42
`torch.no_grad`	Yes (inference)
cuDNN deterministic	Not explicitly set
cuDNN benchmark	Not explicitly disabled

The model produces reproducible outputs on the same GPU architecture. Small numerical differences (within 1e-4) may occur across different GPU types due to floating-point operation ordering.

Caching Behavior¶

Response caching is handled by the serving infrastructure upstream of the model container, not by the model itself: - Cache key: Determined by the request payload (sequences, params, include options)

Versions & Changelog¶

Version	Date	Changes
v1	2025-12	Initial implementation with encode, predict, and log_prob actions; biological verification against ubiquitin and hemoglobin test cases

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

ESM-1b is trained on UniRef50, which covers protein sequences from all domains of life -- bacteria, archaea, eukaryota, and viruses. The model accepts single-chain protein sequences of up to 1022 residues (1024 tokens including BOS/EOS).

ESM-1b handles globular, soluble proteins best, as these dominate the training data. Performance characteristics vary by protein type:

Globular, soluble proteins: Good representation quality. These constitute the bulk of UniRef50.
Membrane proteins: Functional but degraded -- transmembrane regions are under-represented in UniRef50.
Intrinsically disordered proteins/regions (IDPs/IDRs): The model can embed these, but structural inferences will be unreliable since disordered regions lack stable 3D structure.
Multi-domain proteins: Each domain is reasonably represented, but the 1022-residue limit means many multi-domain proteins must be truncated.
Fibrous proteins (collagen, keratin): Under-represented in training data. Repetitive sequences may produce degenerate embeddings.

Legacy note: ESM-2 (the successor model) covers the same molecule types with improved representation quality and a longer maximum sequence length (2048 residues). For new work, ESM-2 is preferred.

Cross-Applicability¶

Molecule Type	Applicability	Evidence	Caveats
Antibodies	Moderate	ESM-1b embeddings used in early antibody engineering studies	Variable regions (CDRs) are represented; specialized antibody models (IgBERT, AntiBERTy) may outperform on antibody-specific tasks
Enzymes	High	Active site conservation patterns captured by embeddings	No explicit catalytic mechanism modeling -- information is implicit in evolutionary signal
Peptides	Low–Moderate	Short sequences (< 30 residues) provide limited context	Consider peptide-specific models for very short sequences

Biological Problems Addressed¶

Protein Representation Learning¶

Problem: Downstream protein analysis tasks (function prediction, localization, interaction prediction) require numerical representations of protein sequences. Traditional approaches use hand-crafted features (amino acid composition, physicochemical scales) or alignment-based profiles (PSSM, HMM), which are limited in expressiveness or computationally expensive.

How ESM-1b helps: The encode action produces dense vector embeddings that capture evolutionary, structural, and biophysical properties learned from hundreds of millions of protein sequences. These embeddings serve as general-purpose features for any downstream model.

Biological meaning: A mean-pooled ESM-1b embedding is a 1280-dimensional vector encoding the "evolutionary fingerprint" of a protein. Proteins with similar function, structure, or evolutionary origin cluster together in embedding space, even at low sequence identity. Per-token embeddings capture position-specific information useful for residue-level predictions (e.g., binding sites, post-translational modification sites).

Historical significance: ESM-1b was the first model to demonstrate at scale that protein language model representations spontaneously encode information about protein tertiary structure -- an emergent property of training on evolutionary data alone. This finding motivated the development of ESM-2, ESMFold, and many subsequent protein foundation models.

Zero-Shot Variant Effect Prediction¶

Problem: Predicting the functional impact of amino acid substitutions (missense mutations) is critical for clinical genetics, protein engineering, and understanding disease. Experimental methods (deep mutational scanning) are expensive and limited to one protein at a time.

How ESM-1b helps: The log_prob action computes the summed log-likelihood of a protein sequence in a single unmasked forward pass (wt-marginal scoring). By comparing log P(wildtype) vs log P(mutant), researchers can estimate variant effects without task-specific training. Evolutionarily conserved positions will have high log-probability for the wildtype amino acid and low probability for substitutions.

Biological meaning: A large negative change in log-probability (mutant much less likely than wildtype) suggests the mutation disrupts an evolutionarily conserved position and is likely deleterious. The BioLM verification confirms this: real ubiquitin scores dramatically higher (-0.17 per-residue log-prob) than a shuffled version with identical composition (-36.74), demonstrating the model captures evolutionary constraints.

Note: ESM-2's log_prob provides equivalent functionality with superior representations. For variant effect prediction, ESM-2 is recommended.

Masked Token Prediction (Sequence Completion)¶

Problem: In protein engineering and directed evolution, researchers want to know which amino acids are compatible at specific positions in a protein, accounting for full sequence context.

How ESM-1b helps: The predict action takes a sequence with <mask> tokens and returns the predicted probability distribution over all 20 standard amino acids at each masked position. This provides a context-aware "consensus" reflecting evolutionary constraints.

Biological meaning: The predicted distribution at a masked position reflects the structural and functional constraints on that residue. Positions critical for the protein's fold will have narrow distributions (few amino acids tolerated); surface-exposed loops will have broader distributions (many amino acids acceptable). The BioLM verification confirms that ESM-1b correctly predicts Lysine at position 48 of ubiquitin -- a biologically critical residue required for polyubiquitin chain formation.

Applied Use Cases¶

Plant Protein-Protein Interaction Prediction¶

Source: (2023). "Pre-trained protein language model sheds new light on the prediction of Arabidopsis protein-protein interactions." Plant Methods. DOI: 10.1186/s13007-023-01119-6

ESM-1b embeddings were combined with an MLP classifier for Arabidopsis protein-protein interaction prediction, achieving AUPR of 0.810 on unseen protein pairs. This demonstrates ESM-1b's utility as a feature extractor for organism-specific interaction networks, even without fine-tuning on the target organism.

Domain-Adaptive Pretraining for DNA-Binding Proteins¶

Source: (2024). "Improving prediction performance of general protein language model by domain-adaptive pretraining on DNA-binding protein." Nature Communications. DOI: 10.1038/s41467-024-52293-7

ESM-1b was domain-adapted for DNA-binding protein prediction through continued pretraining on DBP sequences, outperforming the base model on DBP classification. This study establishes a pattern for specializing general-purpose protein language models to specific functional classes via domain-adaptive pretraining.

Few-Shot Fitness Prediction¶

Source: (2024). "Enhancing efficiency of protein language models with minimal wet-lab data through few-shot learning." Nature Communications. DOI: 10.1038/s41467-024-49798-6

ESM-1b and ESM-1v were benchmarked with few-shot fine-tuning across 87 deep mutational scanning datasets for protein fitness prediction. The study demonstrates that even a small number of experimental measurements (few-shot) can substantially improve PLM-based fitness predictions over zero-shot baselines, a practically important finding for protein engineering campaigns with limited experimental budgets.

Many of these applications have since migrated to ESM-2, which provides improved performance with the same API patterns.

Predecessor Models¶

ESM-1 (Rives et al., earlier versions): Smaller-scale precursors that established the protein language modeling approach. ESM-1b was the largest and best-performing model in the original ESM series.

Successor Models¶

ESM-2 (Lin et al., 2023): The direct successor to ESM-1b. Available in five size variants (8M to 3B parameters), ESM-2 provides strictly better representations than ESM-1b through improved training procedures and systematic scaling. ESM-2 is the recommended choice for all new work.
ESMFold (Lin et al., 2023): Uses ESM-2 embeddings for single-sequence protein structure prediction.
ESM3 (EvolutionaryScale, 2024): A multimodal protein model handling sequence, structure, and function simultaneously.

Complementary Models¶

ESM-1b embeddings can be used as input features for downstream models:

ESMFold / Chai-1: Structure prediction for top-scoring variants from ESM-1b log-likelihood screening

Alternative Models¶

Alternative	Advantage Over ESM-1b	Disadvantage vs ESM-1b
ESM-2 (650M)	Better representations at same parameter count; longer max sequence	None -- strictly superior for all tasks
ProtTrans (ProtT5)	Encoder-decoder enables generation	Larger compute footprint
SaProt	Structure-aware tokens when structure is available	Requires predicted or experimental structure input
ESM3	Multimodal (sequence + structure + function)	Newer, fewer downstream benchmarks

When to use ESM-1b: Only when reproducing results from published studies that specifically used ESM-1b, or when backward compatibility with existing ESM-1b-based pipelines is required.

When to use alternatives: For all new work, use ESM-2. Consider SaProt when structural information is available; consider ESM3 for multimodal protein understanding.

Biological Background¶

Proteins are linear chains of amino acids (typically 50-2000 residues long) that fold into three-dimensional structures to carry out virtually all cellular functions -- catalysis, signaling, transport, structural support, and immune defense. The sequence of amino acids (the "primary structure") determines the protein's 3D structure and, consequently, its function.

Evolutionary conservation: Over billions of years of evolution, natural selection has preserved amino acids critical for protein function while allowing variation at less constrained positions. By analyzing millions of protein sequences from diverse organisms, patterns of conservation and co-variation encode deep information about protein structure and function. This is the fundamental insight that protein language models like ESM-1b exploit.

Masked language modeling on proteins: Just as BERT learns grammar and semantics by predicting masked words in sentences, ESM-1b learns the "grammar" of proteins -- which amino acids are compatible at each position given the surrounding context. A position deep in the hydrophobic core "expects" hydrophobic residues (L, I, V, F); a surface-exposed loop tolerates most amino acids; a catalytic residue is highly constrained. The model learns these rules implicitly from evolutionary data.

Historical context: ESM-1b (2021) was the first model to show convincingly that these learned representations encode three-dimensional structural information as an emergent property. Attention heads in the model learn to attend to residue pairs that are physically close in 3D space, despite never seeing protein structures during training. This discovery was a major milestone in computational biology and directly motivated the development of ESMFold and subsequent protein foundation models.

Key terminology: - Embedding: A dense numerical vector representing a protein (or a position within it) in a high-dimensional space. Similar proteins have similar embeddings. - Masked language modeling (MLM): Training objective where random positions are hidden and the model must predict the original amino acid from context. - Log-likelihood / pseudo-log-likelihood: A score measuring how "expected" a sequence is under the model. Higher (less negative) scores indicate the sequence is more consistent with evolutionary patterns. - UniRef50: A clustered version of the UniProt Reference Clusters database where sequences are grouped at 50% sequence identity, reducing redundancy while preserving diversity. - Remote homology: Evolutionary relationship between proteins that share a common ancestor but have diverged to the point where sequence similarity is undetectable by standard alignment methods. ESM-1b's embeddings can detect such relationships.

Sources & license¶

License: MIT (text)

Papers

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences — Proceedings of the National Academy of Sciences, 2021 · DOI

Source repositories

github: https://github.com/facebookresearch/esm
huggingface: https://huggingface.co/facebook/esm1b_t33_650M_UR50S

Cite

@article{rives2021biological,
  title={Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences},
  author={Rives, Alexander and Meier, Joshua and Sercu, Tom and Goyal, Siddharth and Lin, Zeming and Liu, Jason and Guo, Demi and Ott, Myle and Zitnick, C Lawrence and Ma, Jerry and Fergus, Rob},
  journal={Proceedings of the National Academy of Sciences},
  volume={118},
  number={15},
  pages={e2016239118},
  year={2021},
  doi={10.1073/pnas.2016239118}
}

ESM-1b¶

At a glance¶

API & schema¶

encode¶

predict¶

log_prob¶

Usage¶

Overview¶

Architecture¶

Capabilities & Limitations¶

Usage Examples¶

Architecture & training¶

Architecture¶

Model Type & Innovation¶

Parameters & Layers¶

Training Data¶

Loss Function & Objective¶

Tokenization / Input Processing¶

Performance & Benchmarks¶

Published Benchmarks¶

BioLM Verification Results¶

Comparison to Alternatives¶

Strengths & Limitations¶

Pros¶

Cons¶

Known Failure Modes¶

Implementation Details¶

Inference Pipeline¶

Memory & Compute Profile¶

Determinism & Reproducibility¶

Caching Behavior¶

Versions & Changelog¶

Biology¶

Molecule Coverage¶

Primary Molecule Type(s)¶

Cross-Applicability¶

Biological Problems Addressed¶

Protein Representation Learning¶

Zero-Shot Variant Effect Prediction¶

Masked Token Prediction (Sequence Completion)¶

Applied Use Cases¶

Plant Protein-Protein Interaction Prediction¶

Domain-Adaptive Pretraining for DNA-Binding Proteins¶

Few-Shot Fitness Prediction¶

Related Models¶

Predecessor Models¶

Successor Models¶

Complementary Models¶

Alternative Models¶

Biological Background¶

Sources & license¶

`encode`¶

`predict`¶

`log_prob`¶